1

1 Reassessment of the Performance Standard for the AOAC Use-dilution 

2 Methods (955.15 and 964.02) Revision Date: October 19, 2012

3

4 Albert E. Parker

5 Center for Biofilm Engineering, Montana State University, Bozeman, MT 59715

6 Martin A. Hamilton

7 Big Sky Statistical Analysts LLC, Bozeman, MT 59715

8 Stephen F. Tomasino

9 U.S. Environmental Protection Agency – Office of Pesticide Programs, Biological and 

10 Economic Analysis Division, Microbiology Laboratory Branch, Environmental Science 

11 Center, Fort Meade, MD 20755-5350

12

13

14 The AOAC Use-dilution methods (UDM), 955.14 (Salmonella enterica), 955.15 

15 (Staphylococcus aureus) and 964.02 (Pseudomonas aeruginosa), are laboratory 

16 methods used to substantiate antimicrobial efficacy claims for liquid disinfectants 

17 on inanimate surfaces. The UDM is accepted by the United States Environmental 

18 Protection Agency (EPA) for the purpose of product registration. Testing of S. 

19 aureus and P. aeruginosa is required to support a hospital disinfectant claim. To 

20 meet the current performance standard for the UDM, a product must exhibit no 

21 more than 1 carrier exhibiting growth (“positive”) out of 60 carriers tested. 

22 Employing statistical modeling techniques, the method’s performance standard has 

23 been reassessed based on data collected from a 5 laboratory collaborative study

24 conducted in 2009. Of concern are the pass-error (a Type I error) rate for ineffective 

25 products and the fail-error (a Type II error) rate for effective products. A 

26 performance standard should keep both of these error rates as low as possible. 

27 Based on the model’s output, new performance standards for two UDM methods, 

28 955.15 (S. aureus) and 964.02 (P. aeruginosa) are considered. The performance 

29 standards considered would require three tests (e.g., testing of three representative 

30 production lots per microbe or testing one representative production lot three times

31 per microbe) to be performed against P. aeruginosa and S. aureus; the disinfectant 

32 would need to pass all tests for both microorganisms. To be considered a pass, no 

33 more than 6 positive carriers for a P. aeruginosa test and no more than 3 positive 

34 carriers for a S. aureus test would be allowed. When calculated using the 

35 reproducibility standard deviations observed in the collaborative study, the pass36 error rate is 1-3% for ineffective products and 5% fail-error rate for highly effective 

37 products. The EPA is seeking AOAC’s technical review of the model and the 

38 associated calculations. Proposed revisions to Methods 955.15 and 964.02 are 

39 provided. This paper does not address the performance standard for S. enterica as 

40 no change is currently being explored.
2

41 Background on the Use-dilution Method

42

43 The AOAC Use-dilution methods, 955.14 (Salmonella enterica) (1), 955.15

44 (Staphylococcus aureus) (2), and 964.02 (Pseudomonas aeruginosa) (3), have been used 

45 to assess the efficacy of liquid disinfectants for over four decades. With the exception of 

46 the test microbes and subtle differences in generating the test cultures, the three Use47 dilution methods (UDM) are essentially the same. The United States Environmental 

48 Protection Agency (EPA) accepts the UDM to support the registration of liquid 

49 antimicrobial products bearing claims to control microorganisms that pose a threat to 

50 human health on inanimate surfaces – these are known as public health products.

51 Specifically, public health disinfectants such as those for use in the healthcare

52 environment (i.e., hospital disinfectants) must be deemed effective against S. aureus and 

53 P. aeruginosa. Testing of S. enterica is used to support limited and broad spectrum 

54 disinfectants, but is not currently used to support hospital disinfectant claims.

55

56 The UDM is a semi-quantitative test method that requires the use of stainless steel 

57 cylinder carriers to simulate an environmental surface. Carriers are submerged in a test 

58 culture and dried. The inoculated carriers are divided into two groups; treated (60 carriers 

59 per test) and untreated (6 control carriers per test). The treated carriers are placed into 10 

60 mL of disinfectant and allowed to stand for a defined contact time and temperature.

61 Following the prescribed contact time, each carrier is removed aseptically and placed in a 

62 tube containing a liquid subculture medium for neutralization and recovery of viable 

63 bacteria and incubated. Product efficacy is determined by the presence (growth) or 

64 absence (no growth) of the test microbe on each of the 60 treated carriers. The untreated 

65 (control) carriers are not exposed to the disinfectant product and are used to assess 

66 microbial loading, otherwise known as the carrier counts. The carrier counts are 

67 expressed as the mean number of colony forming units (CFU) per carrier. Currently, the 

68 UDM allows no more than 1 positive carrier out of 60 (for each microbe) for a product to 

69 be deemed efficacious. Numerous editorial revisions have been made to the UDM to 

70 improve the clarity and specificity of the procedure, and minimum and maximum carrier 

71 count levels have been specified to qualify the test results (4). For a complete description 

72 of the UDM methodology, refer to the official AOAC International (AOAC) methods (1-

73 3).

74

75 In practice, the results from one or more UDM tests are classified as a “pass or fail”

76 outcome where a pass means the product exhibited satisfactory disinfection efficacy and 

77 a fail means it did not. The current testing required by EPA for the registration of liquid 

78 disinfectants specifies the evaluation of three representative production lots (or batches) 

79 of the product tested once against each of the required microbes. One of the three lots 

80 must be at least 60 days old. For product registration, all tests may be conducted by one 

81 laboratory. The disinfectant’s efficacy is deemed acceptable only if each of the tests

82 produces a passing outcome (i.e., no more than 1 positive carrier out of 60 tested). Thus, 

83 if 2 or more carriers are positive per test, the outcome is deemed a fail.

84

85 The statistical justification of the current UDM performance standard was published in 

86 1961 by Ortenzio and Stuart (5). They viewed the UDM as a procedure to confirm the 
3

87 efficacy of diluted liquid germicides based on the “maximum effective dilution” that can 

88 be recommended for disinfecting floors, walls, fixtures, and equipment. They proposed 

89 the current performance standard for the UDM which allows no more than 1 positive 

90 carrier out of a 60 carrier set, and claimed that this performance standard allowed only 

91 5% of ineffective treatments to pass when challenged against S. aureus or S. enterica. 

92 Their work relies on a probit analysis (6) of unpublished data from replicate product 

93 applications and a wide range of use-dilutions. Furthermore, the statistical justification 

94 for the current performance standard did not account for variability among laboratories; 

95 assessing the variability among laboratories is considered a critical aspect of method 

96 performance evaluations. Although liquid disinfectants have become more diverse over 

97 the years, and the method has been the subject of many studies and many revisions, the 

98 performance standard has not been reassessed and remains unchanged.

99

100 Two significant multi-laboratory studies by Cole (7) and Rutala (8) demonstrated that 

101 UDM results may vary considerably among laboratories and they identified major aspects 

102 of the method that were likely sources of the variability, most notably irregular surfaces 

103 of the stainless steel carriers, inconsistent inoculum levels, and inadequate removal of the 

104 pellicle associated with P. aeruginosa test cultures. Unfortunately, control carrier counts 

105 were not evaluated in the studies. An initiative designed to resolve the sources of 

106 variability led to the development and validation of AOAC method 991.47, the Hard 

107 Surface Carrier Test (9). The Hard Surface Carrier Test (HSCT) is very similar to the 

108 UDM and specifies the following: 1) standardization of test cultures used to inoculate 

109 carriers, 2) assessment of control carrier counts and associated minimum and maximum 

110 limits, 3) use of glass penicylinders as carriers; and 4) use of an agar-based procedure for 

111 growing the test microbes. In 1992, Rubino et al. published a statistical analysis of a 10-

112 lab, 6-disinfectant collaborative study of the HSCT (10). In the paper, Rubino et al. 

113 proposed single test HSCT performance standards, which the authors claimed allowed at 

114 least 5%, but less than 50%, of ineffective treatments to pass. To receive a pass under the 

115 performance standard specified for the HSCT, the test must result in no more than 2 

116 positive carriers out of 60 for a S. aureus or a S. enterica test, and no more than 3 positive 

117 carriers out of 60 for a P. aeruginosa test. The HSCT is accepted by EPA to support the 

118 registration of liquid disinfectants; however, the HSCT is rarely used by the regulated 

119 community purportedly due to the lack of hard water and organic soil load in the 

120 validation study.

121

122 Critical to the current initiative, a multi-laboratory collaborative study was conducted in 

123 2009 by the EPA and an industry-based technical work group to assess a set of proposed 

124 modifications to the UDM (11); the effort was not designed to specifically address the 

125 subject of this paper (i.e., reassessment of the UDM performance standard). The study 

126 compared the current UDM to a modified UDM in a side-by-side fashion across five 

127 laboratories to evaluate if the modifications improved the method’s performance. The 

128 modifications were largely based on the culture preparation techniques specified for the 

129 HSCT. In the study, a single lot of an EPA registered quaternary ammonium liquid 

130 disinfection product was tested at two levels of presumed efficacy using both S. aureus

131 and P. aeruginosa methods. EPA’s Standard Operating Procedure (SOP) for conducting 

132 the UDM, SOP MB-05 (12), based on a strict interpretation of the published UDM, was 
4

133 followed by the participating labs. The collaborative study results indicate that the 

134 outcomes of the current UDM for S. aureus are very consistent within and among labs, 

135 but vary significantly for P. aeruginosa among laboratories. The variability exhibited by 

136 the current UDM was no larger than that observed for other accepted disinfectant test 

137 methods (13). In fact, the UDM exhibited quite good repeatability within the same lab 

138 and reproducibility amongst multiple labs. 

139

140 This paper utilizes data from the UDM collaborative to perform a statistical assessment 

141 of the existing UDM performance standard and selected alternative performance 

142 standards. The statistical evaluation focuses on the pass-error and fail-error rates 

143 associated with a performance standard (14). These errors, which are defined below,

144 occur when the UDM test results for a disinfectant product lead to the wrong conclusion. 

145 The error rates were calculated by utilizing a conventional hypothesis testing framework 

146 as described in the Modeling Steps section.

147

148 A pass-error occurs when the disinfectant being tested truly has unacceptably low 

149 efficacy, but is deemed a pass; this is the same type of error considered by Ortenzio & 

150 Stuart (5) and Rubino et al. (10). A fail-error occurs when the disinfectant truly is of 

151 acceptable efficacy, but is deemed a fail; this is the type of error emphasized in Arlea et 

152 al. (15). The calculation of these error rates is based on the method’s variability. In this 

153 paper, we show how to calculate the error rates for the performance standard currently in 

154 use for regulatory purposes, and show how to devise a new performance standard that 

155 adequately maintains smaller, acceptable error rates. The evaluations only pertain to tests 

156 involving S. aureus and P. aeruginosa tests, not to S. enterica. These error rates, easily 

157 calculated using the computer code provided in this paper, offer a quantitative decision158 making tool for revising the performance standard for the UDM. The EPA is interested in 

159 utilizing the UDM collaborative data and statistical modeling described in this report for 

160 the purpose of improving the reliability of the UDM. The EPA seeks a technical review 

161 of the statistical model and associated calculations. Based on this modeling exercise, 

162 proposed revisions to Methods 955.15 and 964.02 are provided.

163

164 Assessing the UDM Performance Standard

165

166 Terminology and Process

167

168 A performance standard for a method defines requirements for an acceptable outcome

169 (i.e., acceptable product performance) for a product being tested by the method. The 

170 acceptable outcome can be specified per test, with the requirement that the product passes 

171 all of multiple tests, or the acceptable outcome can be specified as an average over 

172 multiple tests. The performance standard may also stipulate the maximum number of 

173 positive carriers or a minimum log reduction value allowed per test, the number of tests 

174 that must be performed, and the number of labs required to conduct the tests. For 

175 example, for the UDM, the current performance standard specifies that a passing outcome 

176 is no more than 1 positive carrier out of 60 tested per test. For product registration, the 

177 EPA requires the evaluation of three separate batches of product per microbe, where one 

178 batch must be 60 days old, and testing may be performed at a single laboratory.
5

179

180 For the purpose of this statistical evaluation, the reassessment process assumes that we 

181 are reviewing a proposed UDM performance standard which specifies PS, the allowable 

182 number of positive carriers per test, the number of tests that must be passed, and the

183 number of labs needed to conduct the tests. An alternative performance standard, based 

184 on the typical number of positive carriers over the multiple tests, was also considered.

185

186 Various outcomes relevant to the effectiveness of disinfectant products are utilized in the 

187 modeling process, each is defined below. For the sole purpose of this paper and 

188 explaining the calculations, an ineffective disinfectant product is defined as one which 

189 has a true median number of positive carriers which is at least PS + 1 out of a 60 carrier 

190 set. In other words, an ineffective product fails the performance standard by at least a 

191 single positive carrier in at least 50% of all UDM tests. Further, a marginally ineffective 

192 disinfectant product is one which has a true median number of positive carriers equal to 

193 PS + 1 out of a 60 carrier set. For the current performance standard which allows PS = 1 

194 positive carriers in each UDM test, a marginally ineffective product has a true median 

195 number of positive carriers equal to 2. In contrast, an effective disinfectant product is 

196 defined as one which has a true median number of positive carriers less than PS + 1 out 

197 of a 60 carrier set. A highly effective disinfectant product is one which has a true median 

198 number of positive carriers equal to 0. In other words, a highly effective product will 

199 pass at least 50% of all UDM tests with 0 positive carriers observed.

200

201 A proposed UDM performance standard can be evaluated by determining whether it 

202 attains acceptably small pass-error and fail-error rates. A pass-error occurs when an 

203 ineffective product passes the given performance standard. A pass-error is also known as 

204 a Type I error in statistical hypothesis testing (16, section 8.3.1). The pass-error rate, α, 

205 also known as the significance level, is the probability that a truly marginally ineffective 

206 product passes the performance standard. Error rates for passing other ineffective 

207 treatments could also be calculated, but the pass-error rate for a marginally ineffective 

208 product is largest compared to any other ineffective disinfectant. A fail-error occurs 

209 when an effective product fails the given performance standard (15). A fail-error is also 

210 known as a Type II error in statistical hypothesis testing (16). The fail-error rate, β, is 

211 the probability that a truly highly effective product fails the performance standard. The 

212 error rates for failing other effective treatments could also be calculated, and these would 

213 be larger than the fail-error rate of an effective disinfectant. 

214

215 If an infinite number of UDM tests were applied to a disinfectant product at an infinite 

216 number of labs, then the true median of all observed UDM outcomes for that product

217 could be directly calculated. In this paper, the term truly is used to describe a

218 hypothetical disinfectant product for which the true median number of positive carriers of 

219 UDM outcomes is specified. For example, in the previous paragraph, a truly highly 

220 effective product is defined as a disinfectant product for which the true median of UDM 

221 outcomes is equal to zero.

222

223 In this manuscript, we use a hypothesis testing framework to calculate error rates for 

224 different performance standards (i.e., rejection regions). However, whereas the 
6

225 conventional hypothesis testing problem is to find a rejection region that achieves 

226 specified error rates for one set of hypotheses, our problem was to find the error rates for 

227 many specified performance standards and sets of hypotheses.

228

229 Evaluating a performance standard using error rates

230

231 Setting acceptable bounds for pass-error and fail-error rates is not a fixed aspect in 

232 modeling, but rather is set by experts, policy makers, and/or scientific advisors. The error 

233 rates 1%, 5% and 10% are commonly used for these error types in scientific hypothesis 

234 testing. A level of 5% is used as the baseline error rate for this decision–making exercise.

235 A pass-error rate of 5% means that in a long series of tests of a marginally ineffective

236 product at many labs, 95% of the tests will correctly fail the UDM performance standard. 

237 A fail-error rate of 5% means that in a long series of tests of a highly effective product at 

238 many labs, 95% of the tests will correctly pass the UDM performance standard. 

239 Regulatory authorities charged with ensuring public safety focus on minimizing pass240 errors in order to prevent accepting an ineffective product. Others may focus on fail241 errors to avoid removing effective products from the marketplace. An optimal 

242 performance standard addresses both types of errors effectively. Due to natural variability 

243 among microbial cultures, product formulations, microbiological techniques, etc., it is 

244 impossible to eliminate both of these errors; however, the error rates can be reduced to 

245 acceptable levels, for example, by conducting multiple UDM tests.

246

247 Previous Models Used for Assessing the UDM and HSCT Data

248

249 The current performance standards for the UDM and HSCT were set by explicitly 

250 considering the pass-error rate defined above, but not the fail-error rate. The current 

251 UDM performance standard, set by Ortenzio and Stuart (5), was purported to provide “a 

252 result at the 95% confidence level”. In other words, allowing no more than 1 positive out 

253 of 60 treated carriers allows a 5% pass-error rate. However, this claim is based on 

254 unpublished data. For the HSCT, Rubino et al. (10) advocated that “a performance 

255 standard should be no smaller than the expected number of positive carriers and no larger 

256 than the upper confidence limit”. At 95% confidence, such a scheme would allow a pass257 error rate of between 5% and 50%. The empirically observed pass-error and fail-error

258 rates for this performance standard can be calculated for data generated in the 10-lab 

259 collaborative HSCT study involving 6 test chemicals. The observed proportion of times 

260 that an ineffective disinfectant was passed was α1 = 0/120 = 0.00 (10, Table 1,

261 formulations 2, 4 and 5, pooled across all 3 organisms). The observed proportion of times 

262 that an effective disinfectant was failed was β1 = 12/120 = 0.10 (10, Table 1, formulations 

263 1, 3 and 6, pooled across all 3 organisms). In 2008, based on fail-error rate calculations, 

264 Arlea et al. (15) advocated modifications to the UDM. Their study suggested that a fail265 error rate of 10% was acceptable (15, Table 1).

266

267 One common feature of the different statistical approaches that have been historically 

268 applied to UDM data is that they first transform the number of positive carriers to a 

269 decimal value, although the units associated with this new value are not always 

270 interpretable with respect to the UDM test. The first step of the transformation is to 
7

271 rewrite the number of positive carriers N as the proportion of negative carriers in a 60 

272 carrier UDM test,

273

274

275

276 (see Appendix A). Ortenzio & Stuart (5) performed a probit analysis which applies the 

277 inverse of the standard normal cumulative distribution function, denoted by to the 

278 proportion of negative carriers by the following equation,

279

280 (1) .

281

282 A general linear model (which includes any fixed or random effects of interest) can be

283 fitted to these probit-transformed values by modern statistical software packages via

284 application of iteratively reweighted least squares (17, p 572). Another approach to 

285 analyzing the numbers of positive carriers from UDM tests is to fit a logistic regression 

286 model, as Arlea et al. (15) did in their assessment of the UDM. This model requires a

287 logit transform of the number of positive carriers, defined by

288

289 (2) .

290

291 As with the probit analysis, these logit values are analyzed using iteratively reweighted 

292 least squares (17, section 14.4). Alternatively, Rubino et al. (10) analyzed the 

293 proportions of negative carriers from a multi-lab study of the HSCT directly by 

294 employing a type of weighted ANOVA that is not advocated by modern statistical 

295 thinking, and they performed calculations by hand because computer software was not 

296 readily available in 1992 to perform the calculations. In 1995, Rubino and collaborators 

297 (18) re-analyzed the HSCT collaborative study data, this time using an approach similar 

298 to the most probable number (MPN) transform of the number of positive carriers N,

299

300 (3) .

301

302 The MPN formula in equation (3) provides a maximum likelihood estimate of the typical 

303 number of viable organisms per carrier, averaged across the 60 carrier set (19-21). In the 

304 bioassay literature, the ln(MPN) is known as the complementary log-log transform (22,

305 pp 55, 113).

306

307 One of the most significant changes to the UDM over the years has been the addition of a 

308 process for measuring the number of viable bacteria on dried inoculated carriers, thus 

309 providing for a semi-quantitative test method. In this case, the typical bio-challenge on 

310 each of the inoculated carriers can be estimated by

311

312 ,

313
8

314 where the average is taken over the set of all enumerated untreated control carriers in the 

315 test. Similar to how Hamilton et al. (18) re-analyzed the HSCT collaborative study, the 

316 TestLD and the MPN formula in equation (3) can be subtracted to estimate a log 

317 reduction (LR) value for each test by

318

319 (4)

320

321 (20, 23). More recently, in the analysis of the UDM collaborative study (11), the MPN

322 transform was used to calculate LRs for each UDM test using equation (4). Calculating 

323 LRs using the MPN formulation has been used to analyze another AOAC standard 

324 disinfectant test method against spores (24).

325

326 Each of these approaches used to analyze the UDM or the HSCT transform the numbers 

327 of positive carriers to a continuous scale before performing a statistical analysis. If one 

328 were to plot the probit, logit and -log10(MPN) transforms (see equations (1)-(3)) as a 

329 function of the number of positive carriers N, one would observe that all three produce 

330 similar decreasing “sideways S” curves (e.g., see the comparative plots in Collett (22, p 

331 54)). We prefer to use the MPN transform instead of the other kinds of transforms that 

332 could be applied to modern-day UDM test data. The main reason for this choice is that 

333 the availability of control carrier counts enable a LR to be calculated for each test 

334 providing results comparable to quantitative disinfectant tests. Even with control counts 

335 available, it is not clear how to calculate a LR from the probit and logit transforms in 

336 equations (1)-(2).

337

338 Modeling Steps – Use of the UDM Collaborative Study Data

339

340 The primary inputs to the modeling process are: i) the numbers of positive carriers 

341 observed in the UDM collaborative study (11), in which there were 3 UDM tests 

342 performed at each lab using a presumed “high” efficacy product; and ii) the control 

343 carrier counts for each UDM test for both P. aeruginosa and S. aureus. A schematic of 

344 the complete modeling process is given in Figure 1.

345

346 Step 1: Calculating log reductions from the number of positive carriers

347

348 The first step of the modeling process was to MPN-transform (via equation (3)) the 

349 numbers of positive carriers from the UDM collaborative study to provide estimates of 

350 the typical number of viable organisms left on each of the 60 treated carriers. Since 

351 control carriers were also enumerated for each test, then the LR was calculated via 

352 equation (4). 

353

354 Step 2: Estimating the variability of the UDM

355

356 In the second step of the modeling process, the LRs calculated from the UDM 

357 collaborative study were analyzed for each microbe separately using a one-way ANOVA 

358 with a random effect due to lab (11, 25). The ANOVA provides estimates of two 

359 components of variance for the LRs for each microbe separately. One estimated 
9

360 component of variance is due to sources among labs (e.g. different equipment, 

technicians and geographic locations), denoted by S

2

361 lab. The second estimated variance 

362 component is due to within-lab sources (e.g. fresh inocula and different environmental 

conditions on the days on which experiments are run), denoted by S

2

363 test. The values for 

the variance components S

2

lab and S

2

364 test found for P. aeruginosa and S. aureus in the 

365 UDM collaborative study are provided in Table 1. The estimated reproducibility standard 

366 deviation (SD) is calculated from the variance components by

367

(5) SR = [S

2

lab + S

2

test]

1/2 368

369

370 (25). Note that Stest is the repeatability SD of the LRs and is often denoted by Sr. The 

variance components S

2

lab and S

2

371 test were used in all subsequent calculations of the 

372 expected pass-error and fail-error rates. Thus, the variability observed in the UDM 

373 collaborative study is assumed to be representative of any UDM test.

374

375

376 Step 3: Calculating the error rates for a single UDM test

377

378 The third modeling step relies on a specific performance standard for a single UDM test; 

379 namely, a disinfectant passes if no more than PS1 positive carriers out of the 60 carrier set 

380 are observed. The subscript of “1” emphasizes that this is a performance standard for a 

381 single UDM test rather than multiple UDM tests. For the current UDM performance 

382 standard, PS1 = 1. We assume that ineffective treatments will not meet this performance 

383 standard most of the time. In other words, as previously defined, the true median number 

384 of positive carriers for an ineffective treatment is at least PS1 + 1. Thus, for a given 

385 performance standard PS1, in order to calculate the expected pass-error and fail-error

386 rates, the following competing null and alternative hypotheses were considered

387

388 (6) H0: (true median number of positive carriers) ≥ PS1 + 1

389 Ha: (true median number of positive carriers) < PS1 + 1.

390

391 The null hypothesis H0 presumes that the disinfectant being tested is not efficacious. The 

392 goal of the product’s manufacturer is that the outcome of the UDM test will provide 

393 evidence in favor of Ha, that the disinfectant is truly efficacious. Consistent with the 

394 current implementation of the UDM performance standard based on a single UDM test 

395 outcome, the null H0 is rejected in favor of the alternative Ha if the observed number of 

396 positive carriers is less than or equal to PS1.

397

398 The statistic T which tests the hypotheses in equation (6) follows Student’s t-distribution

399 with degrees of freedom df1, and non-centrality parameter λ1 = 0. (see Appendix A for 

400 details). The Type I pass-error and Type II fail-error rates for the t-test can be found by 

401 conventional statistical techniques, and are equal respectively to

402

403 (7) α1 = Pr(Rejecting H0 | H0: true median N = PS1 + 1)

404 = Pr(T ≥ t1 ; df1 , λ1 = 0).

405 (8) β1 = Pr(Failing to reject H0 |Ha: true median N = 0)
10

406 = Pr(T < t1 ; df1, λ1)

407

408 (16). The probability function Pr( ) is straightforward to evaluate using statistical 

409 software (see Appendix B). Explicit formulas for these are provided in Appendix A, as 

410 well as other details showing how this hypothesis testing framework leads to the 

411 calculation of α1 and β1. The calculations are completely determined by the available 

S

2

lab and S2

412 test values and the difference between two known LR values. The same 

413 degrees of freedom df1 applies to both the Type I and Type II error rate calculations. Also 

414 λ1 = 0 for all Type I error rate calculations and therefore for all α1 calculations. A non415 zero numerical value of λ1 had to be calculated only for Type II fail-error rates β1, but the 

416 calculations were uncomplicated.

417

418 Since the calculations of the error rates depend on differences in LRs, it should be 

419 stressed that the error rates do not depend on the magnitude of the control counts (cf.

420 equation (4)), or on the magnitude of the LR values. The error rates depend only on the 

421 LR differences just described, and on the presumed variability of these LR values.

422

423 Step 4: Calculating the error rates for multiple UDM tests for a single organism using a 

424 “pass all” approach

425

426 Currently, single UDM tests per organism are commonly used to verify efficacy claims in 

427 EPA’s post-registration stewardship program; however, for registration purposes, K = 3 

428 tests per organism are required and a product must all pass K UDM tests in each of L = 1 

429 labs. In the context of testing the hypotheses in equation (6) simultaneously for all KL

430 tests, the pass-error rate for a performance standard that requires such a “pass all” test 

431 scheme is

432

433 αKL = Pr(Rejecting H0 in all KL tests | H0: true median N = PS1 + 1)

434 = Pr(Passing all KL tests | H0: true median N = PS1 + 1)

435 (9) = Pr(T ≥ t1 in all KL tests | H0: true median N = PS1 + 1).

436

437 When the KL tests are performed independently of each other, then 

438

(10) αKL = (α1)

KL 439 .

440

441 In the preceding equation, the pass-error rate for a single test, α1, was provided in 

442 equation (7).

443

444 The probability that a highly effective product fails to pass all KL tests is

445

446 βKL = Pr(Failing to reject H0 in at least one test | Ha: true median N = 0)

447 = 1 - Pr(Rejecting H0 in all KL tests | Ha: true median N = 0)

448 (11) = 1 - Pr(T ≥ t1 in all KL tests | Ha: true median N = 0).

449

450 When the KL tests are performed independently of each other, then this equation can be 

451 simplified to
11

452

(12) βKL = 1 - (1 - β1)

KL 453 .

454

455 In the preceding equation, the fail-error rate for a single test, β1, was provided in equation 

456 (8).

457

458 In practice, it is known that UDM tests conducted in the same lab are not statistically 

459 independent of each other since they may be performed, for example, by the same 

460 technician using similar equipment in the same facility and geographic location. The 

461 estimated correlation amongst repeated tests in the same lab is

462

(13) r = S

2

lab / (S

2

lab + S

2

463 test)

464

465 (17, section 24.1), which is readily calculated for each organism using the values in Table 

466 1. The pass-error and fail-error rates using equations (10) and (12) assume that the KL

467 tests are independent (and uncorrelated). If the correlation in equation (13) is 

468 significantly positive, as is the case when there is a significant lab-to-lab variance 

component S

2

469 lab, then equations (10) and (12) provide incorrect error rates for the 

470 performance standard. To properly account for these correlations among tests of the 

471 same microbe in the same lab, we used a multivariate t-distribution to calculate the pass472 error rate αKL and the fail-error rate βKL using equations (9) and (11) (26). The 

473 multivariate t-distribution requires the same inputs t1, df1, and λ1 as were required for 

474 equations (7) and (8). An additional input is the matrix of correlations among the KL 

475 tests. Statistical software for these calculations is readily available (see Appendix B).

476

477 Alternate Step 4: Calculating the error rates for multiple UDM tests for a single 

478 organism using averaging

479

480 A possible alternative to requiring that a disinfectant pass all of multiple UDM tests is to 

481 impose a performance standard PSmean on the mean disinfectant response across the K 

482 multiple tests performed at each of L labs. Since it is assumed that ineffective treatments 

483 will not meet this performance standard most of the time, then the median number of 

484 positive carriers for an ineffective treatment is at least PSmean + 1. Thus, the hypotheses 

485 being tested in this scenario are

486

487 (14) H0: (true median number of positive carriers) ≥ PSmean + 1

488 Ha: (true median number of positive carriers) < PSmean + 1.

489

490 In contrast to the pass-all approach, these hypotheses are tested by calculating the mean 

491 response across all KL tests. The chance of a pass-error when averaging over all KL tests 

492 is

493

494 αmean = Pr(Rejecting H0| H0: true median N = PSmean+1)

495 = Pr(Passing all KL tests on average | H0: true median N = PSmean+1)

496 (15) = Pr(T ≥ tmean ; dfmean , λmean = 0).

497
12

498 The expected fail-error rate for a highly effective product (for which the true median 

499 number of positive carriers is 0) is calculated by

500

501 βmean = Pr(Failing to reject H0 | Ha: true median N = 0)

502 (16) = Pr(T < tmean ; dfmean, λmean).

503

504 Just as was described for the single test scenario, the above probability calculations 

505 correspond to the Type I and Type II error rates for a t-test of the hypotheses in (14). 

506 These calculations depend on Student’s t-distribution with critical value tmean, degrees of 

507 freedom dfmean, and a non-centrality parameter λmean (see Appendix A).

508

509 Step 5: Calculating the error rates for multiple UDM tests for both P. aeruginosa and S. 

510 aureus using a “pass all” approach

511

512 Step 4 of the modeling process considered error rates for each microbe separately;

513 however, the current UDM performance standard, whether at registration or post514 registration, requires that a disinfectant pass all tests for all organisms under 

515 consideration for a hospital disinfectant claim. To calculate the pass-error and fail-error

516 rates for this scenario, appropriate notation must first be developed. Consider a 

517 performance standard that requires KPa UDM tests against P. aeruginosa and KSa UDM 

518 tests against S. aureus at each of L labs, for a total of K = KPa + KSa tests at each of L labs. 

519 The hypotheses in equation (6) must now be extended to account for different target 

520 numbers of positive carriers for each microbe,

521

522 H0: (true median number of positive carriers for a P. aeruginosa test) ≥ PSPa + 1

523 or

524 (true median number of positive carriers for an S. aureus test) ≥ PSSa + 1

525

526 Ha: (true median number of positive carriers for a P. aeruginosa test) < PSPa + 1

527 and

528 (true median number of positive carriers for an S. aureus test) < PSSa + 1.

529

530 Consistent with the current implementation of the UDM performance standard, H0 is 

531 rejected in favor of the alternative Ha if the observed number of positive carriers NPa in 

532 each of the KPa tests against P. aeruginosa is less than or equal to PSPa and the observed 

533 number of positive carriers NSa in each of the KSa tests against S. aureus is less than or 

534 equal to PSSa.

535

536 The Type I pass-error rate αKL for a performance standard that requires that a disinfectant 

537 pass all K = KPa + KSa tests for both organisms at each of L labs is calculated by

538

539 αKL = Pr(Rejecting H0 in all KL tests | H0: true median NPa = PSPa + 1

540 true median NSa = PSSa + 1)

541 = Pr(Passing all KL tests | H0: true median NPa = PSPa + 1

542 true median NSa = PSSa + 1)

543 (17) = Pr(T ≥ t1 in all KL tests | H0: true median NPa = PSPa + 1)
13

544 true median NSa = PSSa + 1).

545

546 If it is assumed that, within each lab, the P. aeruginosa test results are independent of the

547 S. aureus test results, then the pass-error rate in equation (17) can be calculated by

548

549 .

550

551 In the previous equation, is the Type I pass-error rate for KPa P. aeruginosa tests 

552 and is the pass-error rate for KSa S. aureus tests, with all K = KPa + KSa tests 

553 performed in each of L labs. The rates and can each be calculated using 

554 equation (9).

555 The probability that a highly effective product fails to pass all KL tests is

556

557 βKL = Pr(Failing to reject H0 in at least one test | Ha: true median NPa = 0

558 true median NSa = 0)

559 = 1 - Pr(Rejecting H0 in all KL tests | Ha: true median NPa = 0

560 true median NSa = 0)

561 (18) = 1 - Pr(T ≥ t1 in all KL tests | Ha: true median NPa = 0

562 true median NSa = 0).

563

564 If it is assumed that, within each lab, the P. aeruginosa test results are independent of the

565 S. aureus test results, then this equation can be simplified to

566

567

568

569 where and are the Type II fail-error rates for the KPa P. aeruginosa tests and 

570 the KSa S. aureus tests respectively, with all K = KPa + KSa tests performed in each of L 

571 labs. The fail-error rate for each organism can be calculated by equation (11).

572

573 We have already explained that tests using the same microbe conducted in the same 

574 laboratory may not be uncorrelated. For similar reasons, P. aeruginosa test results may

575 be correlated with S. aureus test results when the tests are conducted in the same 

576 laboratory. Although the UDM collaborative study was explicitly designed to reliably 

577 estimate the correlation among LRs of tests of the same microbe conducted in the same 

578 laboratory, the data from the UDM collaborative did not provide a reliable estimate of the 

579 correlation among tests of different microbes conducted in the same laboratory. Thus, we 

580 calculated the error rates for a range of hypothetical correlation values: 0.00, 0.25, 0.5, 

581 0.95, and then reported the range of error rates found. 

582

583 To account for the between microbe correlation, a multivariate t-distribution (26) was 

584 used to calculate the error rates for a performance standard which requires that all tests 

585 from both organisms must be passed. The necessary inputs for the calculations using the 

586 multivariate t are: t1 and λ1 calculated for each organism separately. Unlike the values 

587 used in equations (7) and (8) for each microbe separately, the degrees of freedom df1 = 

588 6.9 for P. aeruginosa was used since it was smaller and thus more conservative. The 
14

589 calculations were also performed for df1 = 4, the number of labs minus one, which 

590 resulted in negligibly different error rates than those reported. Given these inputs, as well 

591 as the matrix of correlations amongst the tests, statistical software is readily available to 

592 perform the calculations (see Appendix B). In some instances, for the between microbe 

593 correlation values of either 0.5 or 0.95, the correlation matrix was no longer positive 

594 definite, in which case the error rates were not calculable. An intuitive explanation of

595 this failure of the model to generate the error probabilities of interest is that the observed 

596 variability of the two organisms is inconsistent with these high hypothetical correlations.

597

598 Modeling Assumptions

599

600 Figure 1 provides an overview of the assumptions used at each step of the modeling 

601 process. The first critical assumption of the model is that the expected variability (i.e. the 

values of the variance components S

2

lab and S

2

602 test) of any product subjected to the UDM is 

603 assumed to be the same as the variability observed in the UDM collaborative study. A 

604 possible critique of this assumption is that only a single high efficacy product was used to 

605 calculate these variances. Although conventional guidelines for collaborative studies 

606 recommend the inclusion of multiple disinfectant formulations (25), a single disinfectant 

607 study can yield results of practical value. If available resources are limited, as was the 

608 case for the UDM collaborative study, then it is advisable to use a single disinfectant in 

609 all tests; otherwise, to reduce cost, one would be forced to use less laboratories in which 

case the reproducibility estimates SR would be less reliable (i.e., S

2

610 R would have lower 

611 degrees of freedom). For similarly efficacious products having similar LR values within a 

612 narrow range, experience shows that the reproducibility SDs, SR, among the products are 

613 also similar. Highly efficacious treatments produce small SR values; completely 

614 ineffective treatments produce small SR values; and partially efficacious treatments 

615 produce medium or high SR values (27-29). Therefore, we believe that the 

616 reproducibility SD, SR, produced by the high efficacy treatment in the single disinfectant 

617 study of the UDM is representative of the reproducibility for most other high efficacy 

618 disinfectants and that the observed SR can be used when crafting performance standards. 

619 Support for this claim is found in the multi-lab collaborative study results for the HSCT.

620 The reproducibility SD, SR, for the single disinfectant UDM study are consistent with the 

621 SR values for two earlier multi-disinfectant HSCT collaborative studies for P. aeruginosa 

622 (18, 30). For all the different high efficacy disinfectant formulations tested in the UDM 

623 and HSCT collaborative studies, the range of SR values was 0.09 to 0.66 and the median 

624 SR was slightly less than 0.50. Thus, if an additional single disinfectant collaborative 

625 study of the UDM was conducted with P. aeruginosa with a different high efficacy 

626 disinfectant, one would expect the resulting SR to be about 0.50. Note that the statistical 

627 calculations for the UDM performance standards presently being proposed in this paper 

628 are based on SR = 0.53 for P. aeruginosa from the UDM collaborative study. The pass629 error and fail-error rates will be smaller for more effective disinfectants (since they are 

630 presumed to have smaller variability) and larger for less-effective disinfectants (that 

631 exhibit larger variability).

632 A second assumption of importance to the calculation of the error rates is that the LRs,

633 calculated from the control carrier counts and the MPN-transformed numbers of positive 

carriers, are normally distributed with homogeneous within-lab variance component S

2

634 test
15

635 across all labs for each microbe. By equation (13), the homogeneous variance assumption 

636 is equivalent to assuming a homogeneous correlation amongst tests using the same 

637 microbe in the same lab. These assumptions are easily checked with well established 

638 diagnostic tools. In our experience, the LR values usually satisfy the assumptions of 

639 normality and homogeneity of variance across a variety of different test organisms and 

640 standard methods (11, 18, 24, 31-34).

641

642 The third assumption, when calculating the error rates for a performance standard which 

643 requires that all tests from both P. aeruginosa and S. aureus must be passed, was that the 

644 LRs from tests of different microbes conducted in the same lab have the same correlation

645 in all labs. Although it was not possible to reliably estimate this correlation from the 

646 UDM collaborative study, the inability to find a significant correlation, and the disparate 

647 among-lab variance of each of the two organisms (Table 1), is suggestive that such a 

648 correlation, if it exists, is small. Nonetheless, we calculated the error rates for a wide 

649 range of hypothetical correlation values: 0.00, 0.25, 0.5, and 0.95.

650

651 A fourth critical assumption is that the pass-error rate is calculated with respect to a 

652 “marginally ineffective” product that has a median number of positive carriers that 

653 exceeds the performance standard by one positive carrier. The pass-error rate could have 

654 been calculated for a different ineffective product, for example, of a highly ineffective 

655 product with true median number of positive carriers equal to twenty. Nonetheless, the 

656 pass-error rate for a marginally ineffective product is larger than the error rate for any 

657 other ineffective product.

658

659 The fifth and last fundamental assumption is that the fail-error rates have been calculated 

660 with respect to a highly efficacious disinfectant that has a median number of positive 

661 carriers of zero over many tests. One could instead calculate the fail-error rate for a 

662 “marginally efficacious” product, for example, when the true median number of positive 

663 carriers just meets the allowable number of positive carriers. Such a product would have 

664 a higher failure rate than a highly effective product.

665

666 Estimates of Log Reduction

667

668 One defining feature of the modeling approach presented in this paper is transforming the 

669 number of positive carriers observed in a 60 carrier UDM test to a LR value. The 

670 advantages of using the LR instead of analyzing the numbers of positive carriers some 

671 other way are: 1) the resulting value is directly interpretable as the LR of organisms over 

672 the entire 60 carrier set; 2) the results of the variability of the method can be directly 

673 compared to other studies of disinfectant tests, which are usually analyzed with respect to 

674 the LR; 3) for analyses such as ANOVA and t-tests, appropriate for small samples of 

675 normally distributed LRs, can be directly applied; and 4) the normality assumption is 

676 easily verified with well known diagnostic tools.

677

678 It is known that the actual LRs for a disinfectant are likely larger than what is estimated 

679 by the MPN formula when zero positive carriers are observed. This is the same challenge

680 faced by other possible analyses of UDM data such as probit and logit (see equations (1) 
16

681 and (2)). If it were possible to enumerate organisms on all 60 treated carriers and estimate 

682 a LR that way, then statistical theory maintains that this LR, and the LR calculated from 

683 the MPN formula, would both estimate the same true mean LR of the disinfectant applied 

684 in the UDM. Nonetheless, it is not to be expected that the LR calculated for a disinfectant 

685 based on UDM outcomes will be predictive of the LR value obtained for the same 

686 disinfectant applied in a different method, such as the ASTM Quantitative Carrier Disk 

687 Test (29).

688

689 One alternative to the MPN transform approach is the use of the probit transform (see 

690 equation (1)) employed by Ortenzio and Stuart (5), which provides the justification for 

691 the current performance standard of passing a disinfectant in a single UDM test. 

692 Unfortunately, the probit response variable, which has units of normal percentiles, has no 

693 practical meaning with respect to the UDM or the disinfectant being tested. A second 

694 alternative analysis is the logit analysis (in equation (2)). There are many other possible 

695 transforms of UDM numbers of positive carriers as well (33). Unlike a probit transform, 

696 the logit response is interpretable as the log of the odds of getting a negative carrier out of 

697 60. Nonetheless, using either the probit or logit models, the variance components are not 

698 interpretable with respect to other disinfectant tests which use LRs, and in contrast to the 

699 MPN formula, neither explicitly estimates the number of organisms killed in the test. 

700 Like the ANOVA models which are fit to MPN-transformed UDM data, the iteratively 

701 reweighted least squares techniques used to analyze probit and logit transformed data 

702 also yield maximum likelihood estimates of model parameters (17, section 14.4). 

703 However, although ANOVA models of LRs provide valid statistical inference for 

704 variance components and other questions of interest for small numbers of LRs (since LRs 

705 tend to be normally distributed), attempting the same inference via probit and logit

706 analyses is only valid for large sample sizes (17, section 14.7). A last critique of non707 MPN transform approach is that to account for the control counts, a covariate would need 

708 to be added to the generalized linear model. This imposes the additional assumption that 

709 changes in the control carrier counts affect the number of positive carriers similarly 

710 across all tests. Use of the LR values calculated by using the MPN formula (equation 

711 (4)) has no such additional assumption.

712

713 Results and Discussion

714

715 Error rates of the current performance standard for a single UDM test

716

717 When a single UDM test is performed, the expected pass-error and fail-error rates using 

718 the current UDM performance standard which allows up to PS1 = 1 positive carrier can 

719 be determined. For P. aeruginosa, to calculate the chance of a pass-error of an

720 ineffective treatment (which has median number of positive carriers equal to 2), the test 

721 statistic critical value is

722

t1 = (LR(0) – LR(2))/ [0.175 + 0.111]1/2 723

724 = (8.1038 – 7.8783)/0.5348

725 = 0.4217.

726
17

727 The variance components in the denominator in the preceding calculation were obtained 

728 from the UDM collaborative study data reproduced in Table 1. The degrees of freedom 

729 using equation (19) in the Appendix was found to be df1 = 6.9. Applying equation (7), 

730 the pass-error rate for a single UDM test using P. aeruginosa is α1 = 0.343. To 

731 determine the fail-error rate, equation (20) in Appendix A shows that the non-centrality 

732 parameter is

733

λ1 = (LR(N = 1) – LR(N = 2))/ [0.175 + 0.111]1/2 734

735 = (8.5846 – 7.8783)/0.5348

736 = 1.32.

737

738 Applying equation (8), the fail-error rate is β1 = 0.182. Similar calculations for S. aureus

739 yield α1 = 0.244 and β1 = 0.064 based on t-distributions with df1 = 13.8 and λ1 = 2.23. 

740 The error rates for P. aeruginosa and S. aureus for a single UDM test are presented in the 

741 top two rows of Table 2. The commands which generated these values using the 

742 statistical software R are provided in Appendix B. These pass-error rates are much 

743 higher than the 0.05 value arrived at by the unpublished data of Ortenzio & Stuart (5), 

744 which presumably was based on UDM tests using S. aureus and S. choleraesuis. The 

745 authors hypothesize that this large difference in pass-error rates is due to the fact that 

746 their calculations were based on the observed within-lab variance from a single lab, 

747 whereas our calculations are based on a within-lab variance averaged across multiple 

748 labs.

749

750 Error rates of the current performance standard for multiple UDM tests

751

752 For product registration, a product must pass all of K = 3 tests, all in a single lab, with 

753 PS1 = 1, for each test of P. aeruginosa and S. aureus.

754

755 Based on the UDM collaborative study data set, the lab-to-lab variance component for S. 

756 aureus was not significant and estimated to be zero. Thus, one may assume that S. 

757 aureus tests conducted within the same laboratory are uncorrelated, and so equation (10)

758 yields a good approximation of the pass-error rate of ineffective disinfectants which are 

759 required to pass all of three S. aureus tests,

760

αKL = (0.2438)

3

761 = 0.015.

762

763 The fail-error rate for a highly efficacious product applied to three independent S. aureus

764 UDM tests is predicted to be (via equation (12))

765

βKL = 1 - (1 – 0.0644)

3

766 = 0.181.

767

768 These calculations are very close to the ones presented in Table 2. They are not exactly 

769 the same since Table 2 was calculated via equations (9) and (11) using a multivariate t

770 calculation.

771
18

772 Similar calculations assuming independent tests cannot be performed for P. aeruginosa

773 since the results from the UDM collaborative study found significant lab-to-lab 

774 variability (see Table 1). Equation (13) shows that there is a correspondingly high 

775 correlation of r = 0.61 amongst P. aeruginosa tests conducted in the same lab. Thus, the 

776 error rates for P. aeruginosa must be found via equations (9) and (11) using a 

777 multivariate t calculation. The results of these calculations are provided in Table 2. The 

778 code which generated these error rates using the statistical software R is provided in 

779 Appendix B.

780

781 Comparing theoretical and empirical UDM error rates

782

783 It is instructive to compare the theoretical pass-error rates just derived for the current 

784 performance standard with the empirical pass-error rate (for an ineffective low efficacy 

785 treatment) and the empirical fail-error rate (for an effective high efficacy treatment) 

786 observed in the UDM collaborative study. These empirical rates over 15 tests are 

787 presented in Table 3. Unlike the assumed median numbers of positive carriers in Table 2, 

788 the observed median values for P. aeruginosa was 1 (instead of 0) for the effective

789 treatment and 15 (instead of 2) for the ineffective treatment. For S. aureus, the observed 

790 median value was 0 for the effective treatment, and 4 for ineffective treatment. Thus, 

791 except for the highly effective treatment applied to S. aureus, the values presented in 

792 Table 3 are estimated lower bounds of the theoretical pass-error rates and upper bounds 

793 of the theoretical fail-error rates provided in Table 2. Even though only 15 tests were 

794 conducted, the resulting bounds agree with the theoretically obtained values, except for 

795 the fail-error rate for the highly effective treatment applied to S. aureus. A larger 

796 collaborative study with many independent tests would need to be run, with a highly 

797 effective disinfectant with median number of positive carriers N = 0, and a marginally 

798 ineffective treatment with median number of positive carriers N = 2, in order to 

799 empirically corroborate the theoretical results in Table 2.

800

801 Proposing a new performance standard for the UDM

802

803 This paper provides the explicit calculations necessary to determine the pass-error and 

804 fail-error rates for performance standards which require passing only a single test,

805 passing all of multiple tests, or passing multiple tests on average. In order to find a 

806 performance standard which keeps both the pass-error and fail-error rates as low as 

807 possible (e.g., <10%) while providing for a minimum amount of testing, a set of possible 

808 testing scenarios and performance standards for the UDM was investigated. The set was 

809 constructed by varying: the allowable numbers of positive carriers for each test; the

810 numbers of tests that must be passed; and different numbers of labs performing the tests. 

811 The results, calculated via equations (9) and (11), are provided in Table 4 for P. 

812 aeruginosa and in Table 5 for S. aureus. Tables 4 and 5 provide results for a 

813 performance standard on a single microbe, and allow a direct comparison to the UDM 

814 and HSCT performance standard error rates suggested by other authors (5, 10, 15). Table 

815 4 suggests that for a performance standard based only on tests of P. aeruginosa, a multi816 lab testing scenario would be critical to maintain a pass-error rate of 10% or less and a 
19

817 fail-error rate of 5% or less (either two tests in each of two labs or one test in each of 

818 three labs). 

819

820 Table 6 provides the pass-error rates for a performance standard that requires that all 

821 tests from both P. aeruginosa and S. aureus pass (calculated via equations (17) and (18). 

822 Since the calculation of the error rates in Table 6 requires an estimate of the correlation 

823 between microbes, and we do not have a reliable estimate of this correlation, these 

824 calculations were performed for a range of hypothetical correlation values. The model’s 

825 output in Table 6 may be used as a tool to develop a practical and informed decision on 

826 the best approach to managing registration testing requirements. Table 6 suggests that in 

827 order to simultaneously maintain pass-error and fail-error rates less than 5%, three tests 

828 (e.g., testing three production lots or testing a single lot three times) must be run for each 

829 organism, all possibly performed in the same lab. Using this proposal, a disinfectant 

830 would pass each P. aeruginosa UDM test if the number of positive carriers N ≤ 6 (i.e., no 

831 more than 6 positive carriers out of 60 tested) and would pass each S. aureus UDM test if 

832 N ≤ 3 (i.e., no more than 3 positive carriers out of 60 tested), in which case the pass-error

833 rate of marginally ineffective products over all six tests is predicted to be between 1.2% 

834 and 2.8% and the fail-error rate of highly effective products is predicted to be between 

835 4.9% and 5.0%.

836

837 The error rates in Table 6 presume that all future UDM tests applied to any disinfectant 

838 exhibit the same variability as observed in the UDM collaborative study, for which the 

839 among-lab variance for S. aureus was estimated to be zero. It is possible that this 

840 variance component of the LRs is under-estimated since 10 of the 15 S. aureus tests 

841 exhibited zero positive carriers. To investigate the influence that a positive among-lab 

842 variance would have on the error rates, the model used to generate Table 6 was rerun, but 

843 this time with an among-lab S. aureus variance equal to the observed among-experiment 

variance (i.e., S

2

lab = S

2

844 test = 0.100). The results are presented in Table 7. Over the range 

845 of hypothetical correlations, the range of pass-error rates for the proposed performance 

846 standard increased from 1.2%-2.8% to 4.2%-10.5%, and the range of fail-error rates 

847 increased from 4.9%-5.0% to 9.2%-10.6%. Thus, in the presence of a moderate level of 

848 among-lab variability for S. aureus, the proposed performance standard maintains a pass849 error rate of at most 10.5% and a fail-error rate of at most 10.6%. In this scenario, to 

850 maintain a pass error rate less than 10% or less, multiple tests, possibly performed at 

851 different labs, are required (Table 7).

852

853 Another approach explored during this exercise was to consider the averaging of UDM 

854 data across multiple tests. However, in order to achieve a pass-error rate at or below 

855 10% for P. aeruginosa, a 10 lab testing scenario was required, with two UDM tests 

856 performed at each lab (these rates were calculated via equations (15) and (16)). This 

857 pass-error rate was attained with only three uncorrelated tests using the “pass-all” 

858 approach. Thus, for the UDM, an averaging performance standard approach appears to 

859 be too resource intensive.

860

861 The Tables in this manuscript provide point estimates for the pass-error and fail-error

862 rates for different performance standards for the UDM. One could also provide, via 
20

863 computer simulations, a margin of error and confidence interval for each error rate. Such 

864 calculations are beyond the scope of this manuscript. The computer code that generated 

865 the error rates in Tables 4-7 using the statistical software R is provided in Appendix B.

866

867 Conclusions

868

869 Concurrent with the development and assessment of new quantitative methodology 

870 designed to replace the UDM, the EPA continues to work with AOAC and the 

871 stakeholder community to facilitate the introduction, review and publication of editorial 

872 and procedural revisions to standard methods. The current investigation of the UDM 

873 performance standard utilizes the best available scientific practices for assessing the

874 available data and is one of many aspects of the method that has been presented for

875 revision. 

876

877 The statistical evaluation of the multi-laboratory UDM collaborative data provides a 

878 unique opportunity for re-evaluating the current UDM performance standard. This paper 

879 provides the quantitative tools, including computer code, to evaluate a number of options 

880 for a new performance standard. As other authors have proposed, we believe that any 

881 proposed performance standard for tests used by regulatory agencies should be based on 

882 a quantitative assessment of the pass-error and fail-error rates. Based on this assessment,

883 the pass-error rate for a single UDM test, for either P. aeruginosa or S. aureus, is much 

884 higher than originally suggested by previous studies of the UDM and HSCT. This 

885 initiative serves to reduce the error rates while maintaining a practical level of testing.

886 Historically, lower pass-error rates have been achieved by EPA by requiring a registrant

887 to submit efficacy data on three batches of the same product for registration.

888 Furthermore, due to the EPA’s registration specifications for a hospital disinfectant which 

889 involve efficacy data for S. aureus and P. aeruginosa, a performance standard should 

890 require that a hospital disinfectant pass all tests (six total) against both P. aeruginosa and 

891 S. aureus (see Tables 6 and 7). In conjunction with the number of tests conducted, use of 

892 N ≤ 6 for P. aeruginosa and N ≤ 3 for S. aureus as the passing requirement for each test 

893 should improve the overall confidence in the outcome of the testing scheme. Thus, two 

894 testing scenarios may be considered: A) testing each of three production lots one time per 

895 microbe, and B) testing a single production lot three times per microbe. Any of the six 

896 tests that fail to meet the performance standard would fail the product. Pass-error rates of 

897 0.3 to 1.0% and fail-error rates of 41.6 to 45.1% are predicted using the current 

898 performance standard (i.e., N ≤ 1 for both microbes and three production lots tested per 

899 microbe). When error rates for the proposed performance standard (N ≤ 6 for P. 

900 aeruginosa and N ≤ 3 for S. aureus) are calculated using the reproducibility standard 

901 deviations observed in the collaborative study, a low pass-error rate between 1.2% and 

902 2.8% is predicted for marginally ineffective products and the fail-error rate for highly 

903 effective products is predicted to be lowered to approximately 5% (Table 6). The 

904 requirement of N ≤ 1 for each of three tests (three batches of product tested for 

905 registration) for S. enterica would remain the same.

906

907 In addition to the proposed new performance standards, meeting the minimum and 

908 maximum carrier count requirements (mean log density of 6-7 per carrier) and the use of 
21

909 the most current and revised version of UDM further strengthens the reliability of the 

910 UDM and the confidence of the test data. Furthermore, if the new performance standard

911 is implemented, it would be possible to reassess the performance standard again 

912 following the generation of additional data.

913

914 Based on the information provided in this report, the EPA is seeking AOAC’s technical 

915 review of the model and the associated calculations. Based on this modeling exercise, 

916 proposed revisions to Methods 955.15 and 964.02 are provided.

917

918 The following revisions are proposed following section C(f) of each method:

919

920 (g) Performance standard.–For method 955.15 (S. aureus), Section C (h): 

921 Performance standard: ≤3 positive (growth) carriers out of 60 tested. Conduct 

922 three separate UDM tests, one test conducted per day. Each test must meet the 

923 performance standard. Testing at a single laboratory is acceptable.

924

925 (g) Performance Standard.–For method 964.02 (P. aeruginosa), Section C (h): 

926 Performance standard: ≤6 positive (growth) carriers out of 60 tested. Conduct 

927 three separate UDM tests, one test conducted per day. Each test must meet the 

928 performance standard. Testing at a single laboratory is acceptable.

929

930 Acknowledgements

931

932 This work was performed under contract EP-W-08-014 between Montana State 

933 University and EPA OPP Microbiology Laboratory Branch (MLB). The authors 

934 gratefully acknowledge substantial contributions from Dr. Darla Goeres at the Center for 

935 Biofilm Engineering at Montana State University in the preparation of this manuscript.

936

937 References

938

(1) Official Methods of Analysis (2012) 18th 939 Ed., AOAC INTERNATIONAL, 

940 Gaithersburg, MD, Method 955.14

(2) Official Methods of Analysis (2012) 18th 941 Ed., AOAC INTERNATIONAL, 

942 Gaithersburg, MD, Method 955.15

(3) Official Methods of Analysis (2012) 18th 943 Ed., AOAC INTERNATIONAL, 

944 Gaithersburg, MD, Method 964.02

945 (4) Tomasino, S.F., Pines, R.M., & Hamilton, G.C. (2012) J. AOAC Int. 95 (in press)

946 (5) Ortenzio, L.F., & Stuart, L.S. (1961) J. AOAC Int. 44, 416-421

947 (6) Morgan, B.J. (1992) Analysis of Quantal Response Data, Chapman & Hall, New 

948 York, NY

949 (7) Cole, E.C., & Rutala, W.A. (1988) J. AOAC Int. 71, 1187-1194

950 (8) Rutala, W.A., & Cole, E.C. (1987) Infect. Control 8, 501-506

(9) Official Methods of Analysis (2009) 18th 951 Ed., AOAC INTERNATIONAL, 

952 Gaithersburg, MD, Method 991.47

953 (10) Rubino, J.R., Bauer, J.M., Clarke, P.H., Woodward, B.B., Porter, F.C., & Hilton, 

954 H.G. (1992) J. AOAC Int. 75, 635-645
22

955 (11) Tomasino, S.F., Parker, A.E., Hamilton, M.A., & Hamilton, G.C. (2012) J. AOAC 

956 Int. 96 (in press)

957 (12) Standard Operating Procedure (MB-05) for conducting the Use-dilution method,

958 U.S. Environmental Protection Agency - Office of Pesticide Programs, Biological 

959 and Economic Analysis Division, Microbiology Laboratory Branch

960 (13) Tilt, N., & Hamilton, M.A. (1999) J. AOAC Int. 82, 384-389

961 (14) Trout, J. (1985) J. AOAC Int. 68,763-765

962 (15) Arlea, C., King, S., Bennie, B., Kemp, K., Mertz, E., & Staub, R. (2008) J. AOAC 

963 Int. 91, 152-158

964 (16) Casella & Berger (1990) Statistical Inference, Duxbury Press, Belmont, CA

965 (17) Neter, J., Kutner, M.H., Wasserman, W., & Nachtsheim, C.J. (1996) Applied 

966 Linear Statistical Models, McGraw-Hill Co., Boston, MA

967 (18) Hamilton, M.A., DeVries, T.A., & Rubino, R.R. (1995) J. AOAC Int. 78, 1102-

968 1109

969 (19) Hamilton, M.A. (2011) The P/N formula for the log reduction when using a semi970 quantitative disinfectant test of type SQ1, KSA-SM-08, Feb 1, 2011, Center for 

971 Biofilm Engineering at Montana State University, Bozeman, MT. 

972 http://www.biofilm.montana.edu/resources/knowledge_sharing_articles

973 (20) Cochran, W.G. (1950) Biometrics 6, 105-116

974 (21) McCrady, M.H. (1915) J Infect. Dis. 17, 183-212

975 (22) Collett, D. (1991) Modelling Binary Data. Chapman & Hall, New York, NY

976 (23) Blodgett, R.J. (2006) Bacteriological Analytical Manual, FDA, 

977 http://www.fda.gov/Food/SceinceResearch/LaboratoryMethods/BacterilogicalAna

978 lytical ManualBAM/default.htm, Appendix 2

979 (24) Tomasino, S.F., & Hamilton, M.A. (2006) J. AOAC Int. 89, 1373-1397

980 (25) Official Methods of Analysis (2005) 18th Ed., AOAC INTERNATIONAL,

981 Gaithersburg, MD, Appendix D

982 (26) Genz, A., & Bretz, F. (2009) Computation of Multivariate Normal and t 

983 Probabilities Springer-Verlage, Heidelberg, Germany. ISBN 978-3-642-01688-2

984 (27) Bloomfield, S.F., & Looney, E. (1992) J. Appl. Bacteriol. 73, 87-93

985 (28) Bloomfield, S.F., Arthur, M., Van Klingeren, B., Pullen, W., Holah, J.T., & Elton, 

986 R. (1994) J. Appl. Bacteriol. 76, 86-94

987 (29) Springthorpe, V.S., & Sattar, S.A. (2005) J. AOAC Int. 88, 182-201

988 (30) Hamilton, M.A. (2001) “Statistical report: preliminary analysis of the second 

989 collaborative study of the hard surface carrier test,” Aug 31, 2001, EPA contract 

990 #68-W-99-015

991 (31) Tomasino, S.F., Pines, R.M., Cottrill, M.P., & Hamilton, M.A. (2008) J. AOAC 

992 Int. 91, 833-852

993 (32) Tomasino, S.F., Pines, R.M., & Hamilton, M.A. (2009) J. AOAC Int. 92, 1531-

994 1540

995 (33) Hamilton, M.A., & DeVries, T.A. (1996) Biometrics 52, 1112-1120

996 (34) Buckingham-Meyer, K., Goeres, D., & Hamilton, M. (2007) J. Microbiol. Meth.

997 70, 236-244

998 (35) McCullagh, P., & Nelder J. (1989) Generalized Linear Models, Chapman & Hall, 

999 New York, NY

1000 (36) Nijhuis, M.B., & Van den Heuvel, E.R. (2007) J Biopharm. Stat. 17, 123–142
23

1001 (37) Mee, R. (1984) Technometrics 26, 251-254

1002 (38) R Development Core Team (2010), R: A language and environment for statistical 

1003 computing, R Foundation for Statistical Computing, Vienna, Austria, ISBN 3-

1004 900051-07-0, http://www.R-project.org

1005 (39) Genz, A., Bretz, F., Miwa, T., Mi, X., Leisch, F., Scheipl, F., & Hothorn, T. 

1006 (2011) mvtnorm: Multivariate Normal and t Distributions, R package version 0.9-

1007 96, http://CRAN.R-project.org/package=mvtnorm

1008

1009

1010

1011 Figure 1. An overview of the modeling process and assumptions used to calculate the 

1012 two error rates. The number of positive carriers out of 60 observed in a UDM test is 

1013 denoted by N. The maximum allowable number of positive carriers allowed by a 

1014 performance standard is denoted by PS. A log reduction is denoted by LR.

1015

1016 Table 1. Variance components and reproducibility SD of LR for each of the two 

1017 microbes in the UDM collaborative study (11, Table 6).

Organism

Among 

lab 

variance 

(S

2

lab)

Within

lab

variance 

(S

2

test)

Ratio

S

2

lab/ S

2

test

Intra-lab 

correlation 

Reproducibility

SD

P. aeruginosa 0.175 0.111 1.5766 0.6119 0.5348

S. aureus 0.000 0.100 0.0000 0.0000 0.3162

1018

1019
24

1020 Table 2. Estimated error rates, expressed as percentages, for the current UDM 

1021 performance standard (which allows up to 1 positive carrier out of a 60 inoculated and 

1022 treated set). “Three tests in one lab” is the current registration performance standard; 

1023 “one test” is the current post-registration performance standard.

Organism PASS ALL of this Performance standard Pass-error Fail-error

number of tests required to PASS rate (α) rate (β)

each test (median N = 2) (median N = 0)

Pa 1 test N ≤ 1 34.30% 18.19%

Sa 1 test N ≤ 1 24.38% 6.44%

Pa 3 tests in 1 lab N ≤ 1 15.50% 33.45%

Sa 3 tests in 1 lab N ≤ 1 1.59% 18.01%

1024

1025

1026 Table 3. Empirical error rates, expressed as percentages, for the current UDM 

1027 performance standard (which allows up to 1 positive carrier out of a 60 inoculated and 

1028 treated set) (11, Tables 8 and 9, Figures 1 and 2). Unlike the assumed median numbers of 

1029 positive carriers in Table 2, the observed median values for the high efficacy treatment 

1030 applied to P. aeruginosa was 1 (instead of 0), and 15 (instead of 2) for the low efficacy 

1031 treatment. For S. aureus, the observed median values were 0 for the high efficacy 

1032 treatment, and 4 for the low efficacy treatment. Thus, the values in this table are bounds 

1033 for the theoretical rates given in Table 2. Rates are reported as percentages.

1034

Organism PASS ALL of this Performance standard Pass-error Fail-error

number of tests required to PASS rate (α) rate (β)

each test

Pa 1 test N ≤ 1 ≥ 0% (0/15) ≤ 26.7% (4/15)

Sa 1 test N ≤ 1 ≥ 20% (3/15) ≤ 33.3% (5/15)

Pa 3 tests in 1 lab N ≤ 1 ≥ 0% (0/15) ≤ 60% (9/15)

Sa 3 tests in 1 lab N ≤ 1 ≥ 0% (0/15) 60% (9/15)

1035

1036
25

1037 Table 4. Estimated error rates, expressed as percentages, for hypothetical 

1038 performance standards for P. aeruginosa that require a disinfectant to pass all of 

1039 multiple tests as the numbers of labs, numbers of tests at each lab, and the 

1040 allowable number of positive carriers (N) out of a 60 carrier set per test is varied. 

1041 The number of labs is critical due to the large lab-to-lab variance for P. 

1042 aeruginosa.

Num. 

labs

Num. tests 

at each lab 

PASS ALL 

of this 

num. tests

Performance 

standard to 

PASS each 

test

Pass-error 

rate (α)

Fail-error

rate (β)

1 1 1 N ≤ 1 34.30% 18.19%

1 1 1 N ≤ 2 39.38% 9.23%

1 1 1 N ≤ 3 41.94% 5.42%

1 1 1 N ≤ 4 43.49% 3.47%

1 1 1 N ≤ 5 44.52% 2.35%

1 1 1 N ≤ 6 45.26% 1.67%

1 3 3 N ≤ 1 15.50% 33.44%

1 3 3 N ≤ 2 19.20% 18.88%

1 3 3 N ≤ 3 21.23% 11.81%

1 3 3 N ≤ 4 22.49% 7.89%

1 3 3 N ≤ 5 23.36% 5.53%

1 3 3 N ≤ 6 23.97% 4.02%

2 1 2 N ≤ 1 12.02% 32.91%

2 1 2 N ≤ 2 15.64% 17.56%

2 1 2 N ≤ 3 17.68% 10.64%

2 1 2 N ≤ 4 18.98% 6.80%

2 1 2 N ≤ 5 19.87% 4.66%

2 1 2 N ≤ 6 20.53% 3.30%

2 2 4 N ≤ 1 4.62% 47.10%

2 2 4 N ≤ 2 6.58% 27.43%

2 2 4 N ≤ 3 7.78% 17.25%

2 2 4 N ≤ 4 8.57% 11.52%

2 2 4 N ≤ 5 9.14% 8.04%

2 2 4 N ≤ 6 9.55%* 5.81%*

3 1 3 N ≤ 1 4.26% 44.92%

3 1 3 N ≤ 2 6.24% 25.12%

3 1 3 N ≤ 3 7.47% 15.35%

3 1 3 N ≤ 4 8.29% 10.03%

3 1 3 N ≤ 5 8.88% 6.89%

3 1 3 N ≤ 6 9.31% 4.91%

1043
26

1044 Table 5. Estimated error rates, expressed as percentages, for hypothetical 

1045 performance standards for S. aureus that require a disinfectant to pass all of 

1046 multiple tests as the numbers of tests and the allowable number of positive 

1047 carriers (N) out of a 60 carrier set per test is varied. The error rates are 

1048 representative of either single or multiple lab scenarios (due to the insignificant 

1049 lab-to-lab variance for S. aureus).

PASS ALL of this 

num. tests

Performance standard 

required to PASS each 

test Pass-error rate (α) Fail-error rate (β)

1 N ≤ 1 24.38% 6.44%

1 N ≤ 2 32.15% 1.28%

1 N ≤ 3 36.32% 0.34%

1 N ≤ 4 38.89% 0.11%

2 N ≤ 1 6.15% 12.43%

2 N ≤ 2 10.46% 2.54%

2 N ≤ 3 13.27% 0.68%

2 N ≤ 4 15.18% 0.22%

3 N ≤ 1 1.59% 18.01%

3 N ≤ 2 3.43% 3.78%

3 N ≤ 3 4.87% 1.01%

3 N ≤ 4 5.94% 0.33%

1050
27

1051 Table 6. Estimated error rates, expressed as percentages, for hypothetical 

1052 performance standards that require a disinfectant to pass all of multiple tests for 

1053 both microbes as the hypothetical correlation between the LRs of tests of the 

1054 different microbes, the numbers of labs, and the numbers of tests for each microbe 

1055 at each lab is varied. For the “current performance standard,” the allowable 

1056 number of positive carriers (N) out of a 60 carrier set per test is 1 for P. 

1057 aeruginosa and 1 for S. aureus; for the “proposed performance standard” the 

1058 allowable number of positive carriers is 6 for P. aeruginosa and 3 for S. aureus. 

1059 “Not calculable” scenarios are denoted by “NC” and indicate high correlation 

1060 values that are inconsistent with the observed variability for each organism

1061 separately.

1062

Current 

Performance 

Standard

Proposed

Performance 

Standard

Correlation 

between

Microbes

Num.

Labs

Num. 

P.a. tests

at ea. lab

Num. 

S.a. tests

at ea. lab

Passerror

rate 

(α)

Failerror 

rate 

(β)

Passerror

rate 

(α)

Failerror 

rate

(β)

0 1 1 1 8.9% 23.3% 16.7% 2.0%

0.25 1 1 1 11.9% 22.4% 20.4% 2.0%

0.5 1 1 1 15.2% 21.1% 24.5% 1.9%

0.95 1 1 1 23.9% 18.2% 34.8% 1.7%

0 1 2 2 1.5% 36.2% 4.2% 3.6%

0.25 1 2 2 3.1% 33.9% 7.1% 3.5%

0.5 1 2 2 5.1% 31.2% 10.5% 3.4%

0.95 1 2 2 NC NC NC NC

0 1 3 3 0.3% 45.1% 1.2% 5.0%

0.25 1 3 3 1.0% 41.6% 2.8% 4.9%

0.5 1 3 3 NC NC NC NC

0.95 1 3 3 NC NC NC NC

0 2 1 1 0.9% 41.0% 2.8% 4.0%

0.25 2 1 1 1.6% 39.5% 4.2% 3.9%

0.5 2 1 1 2.5% 37.6% 6.0% 3.8%

0.95 2 1 1 6.0% 33.0% 12.2% 3.3%

0 2 2 2 0.0% 58.8% 0.2% 7.1%

0.25 2 2 2 0.1% 55.9% 0.5% 6.9%

0.5 2 2 2 0.3% 52.4% 1.1% 6.6%

0.95 2 2 2 NC NC NC NC

0 3 1 1 0.1% 54.4% 0.5% 5.9%

0.25 3 1 1 0.2% 52.7% 0.9% 5.8%

0.5 3 1 1 0.4% 50.5% 1.5% 5.7%

0.95 3 1 1 1.6% 45.1% 4.3% 4.9%

1063
28

1064 Table 7. Assuming an among-lab variance for S. aureus of the same magnitude as 

1065 the observed within-lab variance, these are the estimated error rates, expressed as 

1066 percentages, for hypothetical performance standards that require a disinfectant to 

1067 pass all of multiple tests for both microbes as the hypothetical correlation 

1068 between the LRs of tests of the different microbes, the numbers of labs, and the 

1069 numbers of tests for each microbe at each lab is varied. For the “current 

1070 performance standard,” the allowable number of positive carriers (N) out of a 60 

1071 carrier set per test is 1 for P. aeruginosa and 1 for S. aureus; for the “proposed 

1072 performance standard” the allowable number of positive carriers is 6 for P. 

1073 aeruginosa and 3 for S. aureus. “Not calculable” scenarios are denoted by “NC” 

1074 and indicate high correlation values that are inconsistent with the observed 

1075 variability for each organism separately.

1076

Current 

Performance 

Standard

Proposed

Performance 

Standard

Correlation 

between

Microbes

Num.

Labs

Num. 

P.a. tests

at ea. lab

Num. 

S.a. tests

at ea. lab

Passerror 

rate 

(α)

Failerror

rate 

(β)

Passerror

rate 

(α)

Failerror

rate

(β)

0 1 1 1 11.08% 29.44% 18.4% 4.4%

0.25 1 1 1 14.42% 27.80% 22.2% 4.3%

0.5 1 1 1 18.11% 25.75% 26.4% 4.0%

0.95 1 1 1 28.31% 19.53% 37.5% 3.0%

0 1 2 2 3.76% 43.56% 7.6% 7.8%

0.25 1 2 2 6.17% 40.51% 11.0% 7.5%

0.5 1 2 2 9.22% 36.87% 15.0% 6.9%

0.95 1 2 2 NC NC NC NC

0 1 3 3 1.83% 52.30% 4.1% 10.6%

0.25 1 3 3 3.57% 48.36% 6.9% 10.1%

0.5 1 3 3 6.01% 43.72% 10.4% 9.2%

0.95 1 3 3 NC NC NC NC

0 2 1 1 1.32% 49.92% 3.4% 8.5%

0.25 2 1 1 2.19% 47.62% 5.0% 8.3%

0.5 2 1 1 3.42% 44.66% 7.0% 7.9%

0.95 2 1 1 8.21% 35.12% 14.1% 5.9%

0 2 2 2 0.16% 67.72% 0.6% 14.9%

0.25 2 2 2 0.42% 64.25% 1.2% 14.4%

0.5 2 2 2 0.91% 59.84% 2.3% 13.4%

0.95 2 2 2 NC NC NC NC

0 3 1 1 0.17% 64.27% 0.6% 12.5%

0.25 3 1 1 0.35% 61.84% 1.1% 12.2%

0.5 3 1 1 0.67% 58.61% 1.9% 11.6%

0.95 3 1 1 2.43% 47.60% 5.3% 8.7%

1077
29

1078 Appendix A - Discussion and details for selected modeling steps

1079

1080 Previous Models Used for Assessing the UDM and HSCT Data

1081

1082 In order to evaluate equations (1)-(3) at the extreme cases of N = 0 or N = 60, the 

1083 proportion of negative carriers pN that we consider includes a 1⁄2 in the numerator and a 1 

1084 in the denominator. Morgan (6, p 58), McCullagh & Nelder (35, p 117) and Collett (22, 

1085 p 80) show that the addition of these terms is appropriate.

1086

1087 Step 3: Calculating the error rates for a single UDM test

1088

1089 For the competing null and alternative hypotheses in equation (6), the rejection region 

1090 RR1 is

1091

1092 RR1 = {All N such that N ≤ PS1},

1093

1094 where N is the number of positive carriers observed in a single UDM test of 60 carriers. 

1095 Since the LR transform (via equation (4)) is a monotonically decreasing function of N for 

1096 a fixed TestLD, and because the symmetrically distributed LRs have median equal to the 

1097 mean, then the hypotheses in equation (6) can be rewritten in a mathematically equivalent 

1098 form as

1099

1100 H0: μLR ≤ LR(PS1 + 1)

1101 Ha: μLR > LR(PS1 + 1).

1102

1103 In these new hypotheses, the true mean LR is denoted by μLR. The rejection region with 

1104 respect to the LRs is

1105

1106 RR1 = {All LRs such that LR(N) ≥ LR(PS1)}

1107 = {(LR(N) – LR(PS1 + 1))/SR ≥ (LR(PS1) – LR(PS1 + 1))/SR }

1108 = {T ≥ t1}.

1109

1110 The bold font indicates random variables throughout the sequel. The pass-error rate can 

1111 now be calculated just as a Type I error is calculated in statistical hypothesis testing 

1112 (Casella & Berger, section 8.3.1). The random variable

1113

1114 T = (LR(N) – LR(PS1 + 1))/SR

1115

1116 is only approximately a t(L - 1) random variable, even when the LRs are normal, since 

(L - 1)S

2

R/E(S

2

R) is only approximately a χ

2

1117 (L - 1) random variable (17, p 971). Nijhuis 

1118 & Van den Heuvel (36) showed that this approximation is particularly good and has a 

1119 negligible effect on probability calculations. The critical value, t1, for the test statistic is 

1120 completely determined by the performance standard and the observed reproducibility SD 

1121 observed in the collaborative study,

1122

1123 t1 = (LR(PS1) – LR(PS1 + 1))/SR.
30

1124 Given a performance standard PS1, the LRs in the numerator can be calculated by 

1125 equation (4). The standard deviation SR in the denominator is the reproducibility SD of 

1126 the LRs across multiple labs as in equation (5), and estimated from the UDM

1127 collaborative study (Table 1). The pass-error rate α1 of a single test can now be 

1128 calculated by application of equation (7), which requires probability calculations based 

1129 on a t-distribution with non-centrality parameter λ1 = 0, and degrees of freedom

1130

1131 (19) .

1132

1133 In equation (19), I = 5 is the number of labs in the UDM collaborative study; J = 3 is the 

number of repeated tests in each lab; and R = S

2

lab/S

2

test, where both S

2

lab and S

2

1134 test are the 

1135 variance components from the UDM collaborative study (see Table 1). Not to be 

1136 confused with the well known result that the standard error of the mean LR across the I

labs, [(S

2

test/J + S

2

lab)/I]

1/2 1137 , has degrees of freedom equal to I - 1 (17), equation (19) gives 

1138 the degrees of freedom associated with SR. Equation (19) can be derived directly using 

1139 Satterthwaite’s formula applied to the independent mean squares and their degrees of 

1140 freedom as in Mee (37). Equations (5) and (19) make clear precisely how the variance 

components S

2

lab and S

2

1141 test from the UDM collaborative study are used to calculate the 

1142 pass-error rate α1.

1143

1144 The expected fail-error rate β1 of highly efficacious treatments is calculated just as a 

1145 Type II error is in statistical hypothesis testing when the LRs are normal using equation 

1146 (8), which uses a non-central t-distribution, with degrees of freedom df1 calculated as in 

1147 equation (19), and non-centrality parameter

1148

1149 (20) λ1 = (LR(0) – LR(PS1 + 1))/SR.

1150

1151 In the numerator in equation (20), we utilize the assumption that the median number of 

1152 positive carriers is zero for a highly effective treatment. The preceding equation, as well 

as equations (5) and (19), make clear precisely how the variance components S

2

1153 lab and

S

2

1154 test from the UDM collaborative study are used to calculate the fail-error rate β1.

1155

1156 Alternate Step 4: Calculating the error rates for multiple UDM tests for a single 

1157 organism using averaging

1158

1159 The hypotheses being tested in this scenario are given in equation (14). If the geometric 

1160 mean MPN across the multiple tests attains the performance standard, then the 

1161 disinfectant passes. That is, the rejection region RRmean for this test is

1162

1163 RRmean = {(geometric mean MPN) ≤ PSmean}.

1164

1165 We consider the geometric mean MPN across K tests at each of L labs instead of the 

1166 arithmetic mean for three reasons. First, the geometric mean MPN can be converted to

1167 the typical number of positive carriers over many UDM tests; secondly, the geometric 
31

1168 mean is directly transformed to the mean LR for each lab using equation (4) when the 

1169 LRs are symmetrically distributed; and thirdly, experience shows that the LRs are 

1170 normally distributed. The hypotheses in equation (14) can be written in a mathematically 

1171 equivalent form with respect to the mean LR,

1172

1173 H0: μLR ≤ LR(PSmean + 1)

1174 Ha: μLR > LR(PSmean + 1).

1175

1176 In these hypotheses, the true mean LR is denoted by μLR. The corresponding rejection 

1177 region with respect to the arithmetic mean of the LRs across K tests at each of L labs is

1178

1179 RRmean = {(arithmetic mean LR) ≥ LR(PSmean)}.

1180

1181 The critical value of the test statistic corresponding to this rejection region is

1182

1183 tmean = (LR(PSmean) – LR(PSmean + 1))/SEmean.

1184

1185 Since multiple tests are available, this test statistic differs in a single fundamental respect 

1186 from the critical value t1 for the single test performance standard presented earlier. The 

1187 denominator is now occupied by SEmean, the standard error of the mean LR over K future 

1188 tests performed at each of L future labs. To use all available data, SEmean is found by 

1189 pooling the standard error obtained from the UDM collaborative study with the standard 

1190 error to be observed in the K future tests performed at each of L future labs. Each of the 

1191 variables necessary to evaluate SEmean are defined next. The “future” standard error to be 

1192 observed in the multiple UDM tests of the disinfectant will be calculated by

1193

SEfuture = [S

2

future, lab/L + S

2

future, test/(LK)]

1/2 1194 ,

1195

1196 which has dffuture = L - 1 degrees of freedom (17, pp 958-976), with the variance 

components S

2

future, lab and S

2

1197 future, test calculated from the future UDM test requiring K tests 

1198 performed at each of L future labs. The standard error estimated from the UDM 

1199 collaborative study is

1200

SEcollab = [S

2

lab/L + S

2

test/(LK)]

1/2 1201 .

1202

The variance components S

2

lab and S

2

1203 test in this equation are the same as those calculated 

1204 from the UDM collaborative study (see Table 1). In other words, SEcollab provides our 

1205 best guess for SEfuture, the standard error of the mean LR in a future study of K tests in 

1206 each of L labs. The degrees of freedom associated with SEcollab is given by

1207

1208 (21) .

1209

1210 In equation (21), I = 5 is the number of labs in the UDM collaborative study; J = 3 is the 

number of repeated tests in each lab in the UDM collaborative study; R = S

2

lab/S

2

1211 test, 
32

where both S

2

lab and S

2

1212 test are the variance components from the UDM collaborative 

1213 study; and K is the number of tests in each lab in a future test of the disinfectant (e.g., by 

1214 enforcement officials) of the disinfectant. Equation (21), found via Satterthwaite’s 

1215 approximation, is a modification of Mee’s (37) formula (cf. equation (19)). The resulting 

1216 pooled standard error of the mean is

1217

1218 (22) SEmean = 

1219

1220 The degrees of freedom associated with SEmean is

1221

1222 (23) dfmean = dfcollab+ L - 1.

1223

1224 It is assumed that the K future UDM tests in each of L labs will yield the same variability 

as that observed in the UDM collaborative, S

2

future, lab = S

2

lab and S

2

future, test = S

2

1225 test. In this 

1226 case, equation (22) reduces to

1227

1228 SEmean = SEcollab,

1229

1230 with degrees of freedom calculated as in equations (21) and (23).

1231

1232 Now the expected pass-error and fail-error rate when averaging over all KL tests can be 

1233 formulated as in equations (15) and (16) using a non-central t-distributions, with degrees 

1234 of freedom dfmean calculated as in equations (21) and (23). The non-centrality parameter 

1235 in equation (16) is

1236

1237 λmean = (LR(N=0) – LR(N=PSmean + 1)/SEmean).

1238

1239

1240 Appendix B - Computer code

1241

1242 The error rate calculations presented in this manuscript were generated using the 

1243 statistical software package R (38), package mvtnorm (26, 39). The R code is presented 

1244 in the same order that the modeling steps were presented earlier, by first showing how to 

1245 calculate the error rates for a performance standard on a single UDM test, then for a 

1246 performance standard which requires passing all of multiple tests, and then based on 

1247 passing multiple tests on average. The following R code should be sequentially entered 

1248 directly into R’s command line.

1249

1250 The following R function generates the LR associated with a number of positive carriers 

1251 as in equation (4),

1252

1253 GetMPNLR<-function(N,TestLD=6.5,NumTot=60)

1254 {

1255 LR = TestLD - log10(-log((NumTot-N+.5)/(NumTot + 1)))

1256 return(LR)

1257 }
33

1258

1259 For a single UDM test, the pass-error rate α1 for the current performance standard is 

1260 calculated for each microbe using equation (7) by:

1261

1262 PS1 = 1 # This corresponds to the current UDM performance standard

1263 SR = 0.5348 # For P. aeruginosa, use SR = 0.3162 for S. aureus

1264 df1_Pa = 6.9 # For P. aeruginosa

1265 df1_Sa = 13.8 # , For S. aureus

1266 LR.PS1 = GetMPNLR(PS1)

1267 LR.PS1plus1 = GetMPNLR(PS1+1)

1268 t1_Pa = (LR.PS1 - LR.PS1plus1)/SR

1269 alpha1_Pa = 1-pt(t1_Pa,df1_Pa)

1270 t1_Sa = (LR.PS1 - LR.PS1plus1)/0.3162

1271 alpha1_Sa = 1-pt(t1_Sa,df1_Sa)

1272

1273

1274 For a single UDM test, the fail-error rate β1 is calculated using equations (8) and (20) by:

1275

1276 LR.0 = GetMPNLR(0)

1277 lambda1_Pa = (LR.0-LR.PS1plus1)/SR

1278 beta1_Pa = pt(t1_Pa,df1_Pa,ncp=lambda1_Pa) # for P. aeruginosa

1279

1280 lambda1_Sa = (LR.0-LR.PS1plus1)/0.3162

1281 beta1_Sa = pt(t1_Sa,df1_Sa,ncp=lambda1_Sa) # for S. aureus

1282

1283 When a performance standard requires that a disinfectant passes all of KL uncorrelated

1284 UDM tests (this means that each test is performed in a different lab in the case of P. 

1285 aeruginosa), then the pass-error rate αKL and the fail-error rate βKL are calculated using a 

1286 multivariate t via equations (9) and (11) by:

1287

1288 library(mvtnorm) 

1289 K = 1 # Number of P. aeruginosa tests at each lab

1290 L = 3 # Number of labs

1291 TotTests = K*L # Total number of tests

1292 alphaKL = pmvt(lower=rep(t1_Pa, TotTests ),upper=rep(Inf, TotTests),df=df1_Pa)

1293 betaKL = 1-pmvt(lower=rep(t1_Pa, TotTests ),upper=rep(Inf, TotTests),df=df1_Pa,delta=lambda1_Pa)

1294

1295 The error rate calculations are more complicated when multiple tests of a single microbe 

1296 are performed in the same lab in the presence of a significant among-lab variance 

component S

2

1297 lab, in which case the correlation matrix amongst the tests must be inputted

1298 in order to evaluate equations (9) and (11). For example, the two error rates for two P. 

1299 aeruginosa tests performed in each of two labs are calculated by: 

1300

1301 ##### An R function to generate the correlation matrix for tests of a single microbe

1302 GenMatrix<-function(NumTests,NumLabs=1,VarLab=.175,VarExp=.111)

1303 {

1304 Z = matrix(0,NumTests,(NumTests+1))

1305 Z[,1]=1

1306 for (i in 1:NumTests)

1307 {

1308 Z[i,i+1]=1
34

1309 }

1310 Psi=diag(c(rep(VarLab,1),rep(VarExp,NumTests)))

1311 Vblk=Z%*%Psi%*%t(Z)

1312 N = NumTests*NumLabs # total number of data points across all labs and exps

1313 V=matrix(0,N,N)

1314 for (i in 0:(NumLabs-1))

1315 {

1316 index=(i*NumTests+1):((i+1)*NumTests)

1317 V[index,index]=Vblk # construct the covariance matrix first

1318 }

1319 V = V/V[1,1]

1320 return(V)

1321 }

1322 ##### End of function

1323

1324 L = 2 # Number of labs

1325 K = 2 # Number of tests in each of the labs

1326 TotTests = K*L

1327 R_Pa = GenMatrix(K,L)

1328 alphaKL = pmvt(lower=rep(t1_Pa, TotTests),upper=rep(Inf, TotTests ),df=df1_Pa,corr=R_Pa)

1329 betaKL = 1-pmvt(lower=rep(t1_Pa, TotTests),upper=rep(Inf,TotTests),df=df1_Pa,corr=R_Pa,delta=lambda1_Pa)

1330

1331 When a performance standard requires that all tests from both P. aeruginosa and S. 

1332 aureus must be passed, the error rate calculations via equations (17) and (18) look similar 

1333 as for the single microbe case, but the correlation matrix is more complicated. For 

1334 example, the two error rates for two tests of each microbe performed in each of two labs 

1335 are calculated by:

1336

1337 ##### An R function to generate the correlation matrix for tests of two microbes

1338 GenMatrix_Microbe<-

1339 function(NumTests,NumLabs,CorrMicrobe,VarLab=c(.175,0),VarExp=c(.111,.1))

1340 {

1341 # NumTests is a 2x1 vector, NumTests[1] specifies number of tests for Pa in each lab

1342 # NumTests[2] sepcifies number of tests for Sa in each lab

1343 #

1344 # VarLab is a 2x1 vector, VarLab[1] is for Pa, VarLab[2] is for Sa

1345 # VarExp is a 2x1 vector, VarExp[1] is for Pa, VarExp[2] is for Sa

1346 PaBlock=GenMatrix(NumTests[1],1,VarLab[1],VarExp[1])

1347 SaBlock=GenMatrix(NumTests[2],1,VarLab[2],VarExp[2])

1348 CovMicrobe = CorrMicrobe*sqrt((VarLab[1] + VarExp[1])*(VarLab[2] + VarExp[2]))

1349 Vblk = matrix(0,sum(NumTests),sum(NumTests))

1350 index1=1:NumTests[1]

1351 index2=(NumTests[1]+1):sum(NumTests)

1352 Vblk[index1,index1]=PaBlock

1353 Vblk[index2,index2]=SaBlock

1354 Vblk[index1,index2]=CorrMicrobe

1355 Vblk[index2,index1]=CorrMicrobe

1356 N = sum(NumTests)*NumLabs # total number of data points across all labs and exps

1357 V=matrix(0,N,N)

1358 for (i in 0:(NumLabs-1))

1359 {

1360 index=(i*sum(NumTests)+1):((i+1)*sum(NumTests))
35

1361 V[index,index]=Vblk

1362 }

1363 return(V)

1364 }

1365 ##### End of function

1366

1367 L = 2 # Number of labs

1368 K_Pa = 2 # Number of P. aeruginosa tests in each of the labs

1369 K_Sa = 2 # Number of S. aureus tests in each of the labs

1370 R = GenMatrix_Microbe(c(K_Pa,K_Sa),L,0.25)

1371 vecblk = c(rep(t1_Pa,K_Pa),rep(t1_Sa,K_Sa))

1372 lower.vec = rep(vecblk,L)

1373 deltablk = c(rep(lambda1_Pa,K_Pa),rep(lambda1_Sa,K_Sa))

1374 delta.vec = rep(deltablk,L)

1375 TotTests = (K_Pa + K_Sa)*L

1376 alphaKL = pmvt(lower=lower.vec,upper=rep(Inf,TotTests),df=df1_Pa,corr=R)

1377 betaKL = 1-pmvt(lower=lower.vec,upper=rep(Inf,TotTests),df=df1_Pa,corr=R,delta=delta.vec)

1378

1379 The error rates for a performance standard on the average over K multiple tests performed 

1380 at each of L labs are calculated using equations (15),(16),(21) and (22):

1381

1382 PSmean = 1 # This corresponds to the current UDM performance standard

1383 L = 3 # Number of labs required by the performance standard

1384 K = 10 # Number of future tests in each lab

1385 S2_lab = 0.175 # For P. aeruginosa, use S2_lab = 0.000 for S. aureus

1386 S2_test = 0.111 # For P. aeruginosa, use S2_test = 0.100 for S. aureus

1387 dfcollab = 3.06

1388 SEmean = sqrt(S2_lab /L + S2_test /(L*K))

1389 dfmean = dfcollab + L - 1

1390 LR.PSmean = GetMPNLR(PSmean)

1391 LR.PSmeanplus1 = GetMPNLR(PSmean+1)

1392 tmean = (LR.PSmean - LR.PSmeanplus1)/SEmean

1393 alpha_mean = 1-pt(tmean,dfmean)

1394 lambda_mean = (LR.0-LR.PSmeanplus1)/SEmean

1395 beta_mean = pt(tmean,dfmean,ncp=lambda_mean)

1396