# For Peer Review

Draft Manuscript for Review

Reassessment of the Performance Standard for the AO AC Use-dilution Methods (955.15 and 964.02) Revision D ate: October 19, 2012

Journal: AOAC OMA

Manuscript ID: OMA-2012-Nov-041

Submission Type: Collaborative Study Manuscript

Date Submitted by the Author: 20-Nov-2012

Complete List of Authors: Tomasino, Stephen; US EPA, Parker, Albert; Montana State University, Center fo r Biofilm Engineering Hamilton, Martin; Big Sky Statistical Analysts LLC, ;

Keyword: Disinfectants

Committee Designation: M - Methods Committee on Antimicrobial Efficacy Tes ting

Not For Redistribution AOAC OMA

# For Peer Review

1



### Reassessment of the Performance Standard for the AO AC Use-dilution 1 

### Methods ( 955.15 and 964.02 ) Revision Date: October 19, 2012 2 3 Albert E. Parker 4 Center for Biofilm Engineering, Montana State Unive rsity, Bozeman, MT 59715 5 Martin A. Hamilton 6 Big Sky Statistical Analysts LLC, Bozeman, MT 59715 7 Stephen F. Tomasino 8 U.S. Environmental Protection Agency – Office of Pe sticide Programs, Biological and 9 Economic Analysis Division, Microbiology Laboratory Branch, Environmental Science 10 Center, Fort Meade, MD 20755-5350 11 12

13 The AOAC Use-dilution methods (UDM), 955.14 ( Salmonella enterica ), 955.15 14 ( Staphylococcus aureus ) and 964.02 ( Pseudomonas aeruginosa ), are laboratory 15 methods used to substantiate antimicrobial efficacy claims for liquid disinfectants 16 on inanimate surfaces. The UDM is accepted by the U nited States Environmental 17 Protection Agency (EPA) for the purpose of product registration. Testing of S. 18

aureus and P. aeruginosa is required to support a hospital disinfectant cla im. To 19 meet the current performance standard for the UDM, a product must exhibit no 20 more than 1 carrier exhibiting growth (“positive”) out of 60 carriers tested. 21 Employing statistical modeling techniques, the meth od’s performance standard has 22 been reassessed based on data collected from a 5 la boratory collaborative study 23 conducted in 2009. Of concern are the pass-error (a Type I error) rate for ineffective 24 products and the fail-error (a Type II error) rate for effective products. A 25 performance standard should keep both of these erro r rates as low as possible. 26 Based on the model’s output, new performance standa rds for two UDM methods, 27 955.15 (

S. aureus ) and 964.02 ( P. aeruginosa ) are considered. The performance 28 standards considered would require three tests (e.g ., testing of three representative 29 production lots per microbe or testing one represen tative production lot three times 30 per microbe) to be performed against P. aeruginosa and S. aureus ; the disinfectant 31 would need to pass all tests for both microorganis ms. To be considered a pass, no 32 more than 6 positive carriers for a P. aeruginosa test and no more than 3 positive 33 carriers for a S. aureus test would be allowed. When calculated using the 34 reproducibility standard deviations observed in the collaborative study, the pass35

error

rate is 1-3% for ineffective products and 5% fail-error rate for highly effective 36 products. The EPA is seeking AOAC’s technical revie w of the model and the 37 associated calculations. Proposed revisions to Meth ods 955.15 and 964.02 are 38 provided. This paper does not address the performan ce standard for S. enterica as 39 no change is currently being explored. 40 Page 1 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

2

Background on the Use-dilution Method 41 42 The AOAC Use-dilution methods, 955.14 ( Salmonella enterica ) (1), 955.15 43 ( Staphylococcus aureus ) (2), and 964.02 ( Pseudomonas aeruginosa ) (3), have been used 44 to assess the efficacy of liquid disinfectants for over four decades. With the exception of 45 the test microbes and subtle differences in generat ing the test cultures, the three Use46 dilution methods (UDM) are essentially the same. Th e United States Environmental 47 Protection Agency (EPA) accepts the UDM to support the registration of liquid 48 antimicrobial products bearing claims to control mi croorganisms that pose a threat to 49 human health on inanimate surfaces – these are know n as public health products. 50 Specifically, public health disinfectants such as t hose for use in the healthcare 51 environment (i.e., hospital disinfectants) must be deemed effective against S. aureus and 52 P. aeruginosa . Testing of S. enterica is used to support limited and broad spectrum 53 disinfectants, but is not currently used to support hospital disinfectant claims. 54 55 The UDM is a semi-quantitative test method that requires the use of stainless ste el 56 cylinder carriers to simulate an environmental surf ace. Carriers are submerged in a test 57 culture and dried. The inoculated carriers are divi ded into two groups; treated (60 carriers 58 per test) and untreated (6 control carriers per tes t). The treated carriers are placed into 10 59 mL of disinfectant and allowed to stand for a defin ed contact time and temperature. 60 Following the prescribed contact time, each carrier is removed aseptically and placed in a 61 tube containing a liquid subculture medium for neut ralization and recovery of viable 62 bacteria and incubated. Product efficacy is determi ned by the presence (growth) or 63 absence (no growth) of the test microbe on each of the 60 treated carriers. The untreated 64 (control) carriers are not exposed to the disinfect ant product and are used to assess 65 microbial loading, otherwise known as the carrier c ounts. The carrier counts are 66 expressed as the mean number of colony forming unit s (CFU) per carrier. Currently, the 67 UDM allows no more than 1 positive carrier out of 6 0 (for each microbe) for a product to 68 be deemed efficacious. Numerous editorial revisions have been made to the UDM to 69 improve the clarity and specificity of the procedur e, and minimum and maximum carrier 70 count levels have been specified to qualify the tes t results (4). For a complete description 71 of the UDM methodology, refer to the official AOAC International (AOAC) methods (1- 72 3). 73 74 In practice, the results from one or more UDM tests are classified as a “ pass or fail ” 75 outcome where a pass means the product exhibited sa tisfactory disinfection efficacy and 76 a fail means it did not. The current testing requir ed by EPA for the registration of liquid 77 disinfectants specifies the evaluation of three rep resentative production lots (or batches) 78 of the product tested once against each of the requ ired microbes. One of the three lots 79 must be at least 60 days old. For product registrat ion, all tests may be conducted by one 80 laboratory. The disinfectant’s efficacy is deemed a cceptable only if each of the tests 81 produces a passing outcome (i.e., no more than 1 po sitive carrier out of 60 tested). Thus, 82 if 2 or more carriers are positive per test, the ou tcome is deemed a fail. 83 84 The statistical justification of the current UDM pe rformance standard was published in 85 1961 by Ortenzio and Stuart (5). They viewed the UD M as a procedure to confirm the 86 Page 2 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

3

efficacy of diluted liquid germicides based on the “maximum effective dilution” that can 87 be recommended for disinfecting floors, walls, fixt ures, and equipment. They proposed 88 the current performance standard for the UDM which allows no more than 1 positive 89 carrier out of a 60 carrier set, and claimed that t his performance standard allowed only 90 5% of ineffective treatments to pass when challenge d against S. aureus or S. enterica

. 91 Their work relies on a probit analysis (6) of unpub lished data from replicate product 92 applications and a wide range of use-dilutions. Fur thermore, the statistical justification 93 for the current performance standard did not accoun t for variability among laboratories; 94 assessing the variability among laboratories is con sidered a critical aspect of method 95 performance evaluations. Although liquid disinfecta nts have become more diverse over 96 the years, and the method has been the subject of m any studies and many revisions, the 97 performance standard has not been reassessed and re mains unchanged. 98 99 Two significant multi-laboratory studies by Cole (7 ) and Rutala (8) demonstrated that 100 UDM results may vary considerably among laboratorie s and they identified major aspects 101 of the method that were likely sources of the varia bility, most notably irregular surfaces 102 of the stainless steel carriers, inconsistent inocu lum levels, and inadequate removal of the 103 pellicle associated with P. aeruginosa test cultures. Unfortunately, control carrier coun ts 104 were not evaluated in the studies. An initiative de signed to resolve the sources of 105 variability led to the development and validation o f AOAC method 991.47 , the Hard 106 Surface Carrier Test (9). The Hard Surface Carrier Test (HSCT) is very similar to the 107 UDM and specifies the following: 1) standardization of test cultures used to inoculate 108 carriers, 2) assessment of control carrier counts a nd associated minimum and maximum 109 limits, 3) use of glass penicylinders as carriers; and 4) use of an agar-based procedure for 110 growing the test microbes. In 1992, Rubino et al. p ublished a statistical analysis of a 10- 111 lab, 6-disinfectant collaborative study of the HSCT (10). In the paper, Rubino et al. 112 proposed single test HSCT performance standards, wh ich the authors claimed allowed at 113 least 5%, but less than 50%, of ineffective treatme nts to pass. To receive a pass under the 114 performance standard specified for the HSCT, the te st must result in no more than 2 115 positive carriers out of 60 for a S. aureus or a S. enterica test, and no more than 3 positive 116 carriers out of 60 for a P. aeruginosa test. The HSCT is accepted by EPA to support the 117 registration of liquid disinfectants; however, the HSCT is rarely used by the regulated 118 community purportedly due to the lack of hard water and organic soil load in the 119 validation study. 120 121 Critical to the current initiative, a multi-laborat ory collaborative study was conducted in 122 2009 by the EPA and an industry-based technical wor k group to assess a set of proposed 123 modifications to the UDM (11); the effort was not d esigned to specifically address the 124 subject of this paper (i.e., reassessment of the UD M performance standard). The study 125 compared the current UDM to a modified UDM in a sid e-by-side fashion across five 126 laboratories to evaluate if the modifications impro ved the method’s performance. The 127 modifications were largely based on the culture pre paration techniques specified for the 128 HSCT. In the study, a single lot of an EPA register ed quaternary ammonium liquid 129 disinfection product was tested at two levels of pr esumed efficacy using both S. aureus 130 and P. aeruginosa methods. EPA’s Standard Operating Procedure (SOP) for conducting 131 the UDM, SOP MB-05 (12), based on a strict interpre tation of the published UDM, was 132 Page 3 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

4

followed by the participating labs. The collaborat ive study results indicate that the 133 outcomes of the current UDM for S. aureus are very consistent within and among labs, 134 but vary significantly for P. aeruginosa among laboratories. The variability exhibited by 135 the current UDM was no larger than that observed fo r other accepted disinfectant test 136 methods (13). In fact, the UDM exhibited quite good repeatability within the same lab 137 and reproducibility amongst multiple labs. 138 139 This paper utilizes data from the UDM collaborative to perform a statistical assessment 140 of the existing UDM performance standard and select ed alternative performance 141 standards. The statistical evaluation focuses on th e pass-error and fail-error

rates 142 associated with a performance standard (14). These errors, which are defined below, 143 occur when the UDM test results for a disinfectant product lead to the wrong conclusion. 144 The error rates were calculated by utilizing a conv entional hypothesis testing framework 145 as described in the Modeling Steps section. 146 147 A pass-error

occurs when the disinfectant being tested truly ha s unacceptably low 148 efficacy, but is deemed a pass; this is the same ty pe of error considered by Ortenzio & 149 Stuart (5) and Rubino et al. (10). A fail-error occurs when the disinfectant truly is of 150 acceptable efficacy, but is deemed a fail; this is the type of error emphasized in Arlea et 151 al. (15). The calculation of these error rates is based on the method’s variability. In this 152 paper, we show how to calculate the error rates for the performance standard currently in 153 use for regulatory purposes, and show how to devise a new performance standard that 154 adequately maintains smaller, acceptable error rate s. The evaluations only pertain to tests 155 involving S. aureus and P. aeruginosa tests, not to S. enterica . These error rates, easily 156 calculated using the computer code provided in this paper, offer a quantitative decision157 making tool for revising the performance standard f or the UDM. The EPA is interested in 158 utilizing the UDM collaborative data and statistica l modeling described in this report for 159 the purpose of improving the reliability of the UDM . The EPA seeks a technical review 160 of the statistical model and associated calculation s. Based on this modeling exercise, 161 proposed revisions to Methods 955.15 and 964.02 are provided. 162 163 Assessing the UDM Performance Standard 164 165 Terminology and Process 166 167 A performance standard for a method defines requirements for an acceptabl e outcome 168 (i.e., acceptable product performance) for a produc t being tested by the method. The 169 acceptable outcome can be specified per test, with the requirement that the product passes 170 all of multiple tests, or the acceptable outcome ca n be specified as an average over 171 multiple tests. The performance standard may also s tipulate the maximum number of 172 positive carriers or a minimum log reduction value allowed per test, the number of tests 173 that must be performed, and the number of labs requ ired to conduct the tests. For 174 example, for the UDM, the current performance stand ard specifies that a passing outcome 175 is no more than 1 positive carrier out of 60 tested per test. For product registration, the 176 EPA requires the evaluation of three separate batch es of product per microbe, where one 177 batch must be 60 days old, and testing may be perfo rmed at a single laboratory. 178 Page 4 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

5

179 For the purpose of this statistical evaluation, the reassessment process assumes that we 180 are reviewing a proposed UDM performance standard w hich specifies PS , the allowable 181 number of positive carriers per test, the number of tests that must be passed, and the 182 number of labs needed to conduct the tests. An alte rnative performance standard, based 183 on the typical number of positive carriers over the multiple tests, was also considered. 184 185 Various outcomes relevant to the effectiveness of d isinfectant products are utilized in the 186 modeling process, each is defined below. For the so le purpose of this paper and 187 explaining the calculations, an ineffective disinfectant product is defined as one which 188 has a true median number of positive carriers which is at least PS + 1 out of a 60 carrier 189 set. In other words, an ineffective product fails the performance standard by at least a 190 single positive carrier in at least 50% of all UDM tests. Further, a marginally ineffective 191 disinfectant product is one which has a true median number of positive carriers equal to 192 PS + 1 out of a 60 carrier set. For the current perf ormance standard which allows PS = 1 193 positive carriers in each UDM test, a marginally in effective product has a true median 194 number of positive carriers equal to 2. In contrast , an effective disinfectant product is 195 defined as one which has a true median number of po sitive carriers less than PS + 1 out 196 of a 60 carrier set. A highly effective disinfectant product is one which has a true median 197 number of positive carriers equal to 0. In other w ords, a highly effective product will 198 pass at least 50% of all UDM tests with 0 positive carriers observed. 199 200 A proposed UDM performance standard can be evaluate d by determining whether it 201 attains acceptably small pass-error and fail-error rates. A pass-error

occurs when an 202 ineffective product passes the given performance st andard. A pass-error is also known as 203

a Type I error in statistical hypothesis testing (16, section 8.3. 1). The pass-error rate, α, 204 also known as the significance level , is the probability that a truly marginally ineffe ctive 205 product passes the performance standard. Error rat es for passing other ineffective 206 treatments could also be calculated, but the pass-error rate for a marginally ineffective 207 product is largest compared to any other ineffectiv e disinfectant. A fail-error occurs 208 when an effective product fails the given performan ce standard (15). A fail-error is also 209 known as a

Type II error

in statistical hypothesis testing (16). The fail-error rate, β, is 210 the probability that a truly highly effective produ ct fails the performance standard. The 211 error rates for failing other effective treatments could also be calculated, and these would 212 be larger than the fail-error rate of an effective disinfectant. 213 214 If an infinite number of UDM tests were applied to a disinfectant product at an infinite 215 number of labs, then the true median of all observed UDM outcomes for that product 216 could be directly calculated. In this paper, the t erm truly is used to describe a 217 hypothetical disinfectant product for which the tru e median number of positive carriers of 218 UDM outcomes is specified. For example, in the pre vious paragraph, a truly highly 219 effective product

is defined as a disinfectant product for which the true median of UDM 220 outcomes is equal to zero. 221 222 In this manuscript, we use a hypothesis testing fra mework to calculate error rates for 223 different performance standards (i.e., rejection re gions). However, whereas the 224 Page 5 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

6

conventional hypothesis testing problem is to find a rejection region that achieves 225 specified error rates for one set of hypotheses, ou r problem was to find the error rates for 226 many specified performance standards and sets of hy potheses. 227 228 Evaluating a performance standard using error rates 229 230 Setting acceptable bounds for pass-error and fail-error

rates is not a fixed aspect in 231 modeling, but rather is set by experts, policy make rs, and/or scientific advisors. The error 232 rates 1%, 5% and 10% are commonly used for these er ror types in scientific hypothesis 233 testing. A level of 5% is used as the baseline err or rate for this decision–making exercise. 234 A pass-error

rate of 5% means that in a long series of tests of a marginally ineffective 235 product at many labs, 95% of the tests will correct ly fail the UDM performance standard. 236 A fail-error

rate of 5% means that in a long series of tests of a highly effective product at 237 many labs, 95% of the tests will correctly pass the UDM performance standard. 238 Regulatory authorities charged with ensuring public safety focus on minimizing pass239

errors in order to prevent accepting an ineffective produ ct. Others may focus on fail240

errors to avoid removing effective products from the mark etplace. An optimal 241 performance standard addresses both types of errors effectively. Due to natural variability 242 among microbial cultures, product formulations, mic robiological techniques, etc., it is 243 impossible to eliminate both of these errors; howev er, the error rates can be reduced to 244 acceptable levels, for example, by conducting multi ple UDM tests. 245 246 Previous Models Used for Assessing the UDM and HSCT Data 247 248 The current performance standards for the UDM and H SCT were set by explicitly 249 considering the pass-error rate defined above, but not the fail-error rate. The current 250 UDM performance standard, set by Ortenzio and Stuar t (5), was purported to provide “a 251 result at the 95% confidence level”. In other word s, allowing no more than 1 positive out 252 of 60 treated carriers allows a 5% pass-error rate. However, this claim is based on 253 unpublished data. For the HSCT, Rubino et al. (10) advocated that “a performance 254 standard should be no smaller than the expected num ber of positive carriers and no larger 255 than the upper confidence limit”. At 95% confidenc e, such a scheme would allow a pass256

error

rate of between 5% and 50%. The empirically obser ved pass-error

and fail-error 257 rates for this performance standard can be calculat ed for data generated in the 10-lab 258 collaborative HSCT study involving 6 test chemicals . The observed proportion of times 259 that an ineffective disinfectant was passed was α 1 = 0/120 = 0.00 (10, Table 1, 260 formulations 2, 4 and 5, pooled across all 3 organi sms). The observed proportion of times 261 that an effective disinfectant was failed was β 1 = 12/120 = 0.10 (10, Table 1, formulations 262 1, 3 and 6, pooled across all 3 organisms). In 200 8, based on fail-error rate calculations, 263 Arlea et al. (15) advocated modifications to the UD M. Their study suggested that a fail264

error

rate of 10% was acceptable (15, Table 1). 265 266 One common feature of the different statistical app roaches that have been historically 267 applied to UDM data is that they first transform th e number of positive carriers to a 268 decimal value, although the units associated with t his new value are not always 269 interpretable with respect to the UDM test. The fir st step of the transformation is to 270 Page 6 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

7

rewrite the number of positive carriers N as the proportion of negative carriers in a 60 271 carrier UDM test, 272 273

  .

 274

275 (see Appendix A). Ortenzio & Stuart (5) performed a probit analysis which applies the 276 inverse of the standard normal cumulative distribut ion function, denoted by Φ 

∙ 

, to 277 the proportion of negative carriers by the followin g equation, 278 279 (1)     Φ     

. 280 281 A general linear model (which includes any fixed or random effects of interest) can be 282 fitted to these probit-transformed values by modern statistical software packages via 283 application of iteratively reweighted least squares (17, p 572). Another approach to 284 analyzing the numbers of positive carriers from UDM tests is to fit a logistic regression 285 model, as Arlea et al. (15) did in their assessment of the UDM. This model requires a 286 logit

transform of the number of positive carriers, defi ned by 287 288 (2)     ln       

. 289

290 As with the probit analysis, these logit

values are analyzed using iteratively reweighted 291 least squares (17, section 14.4). Alternatively, R ubino et al. (10) analyzed the 292 proportions of negative carriers from a multi-lab s tudy of the HSCT directly by 293 employing a type of weighted ANOVA that is not advo cated by modern statistical 294 thinking, and they performed calculations by hand b ecause computer software was not 295 readily available in 1992 to perform the calculatio ns. In 1995, Rubino and collaborators 296 (18) re-analyzed the HSCT collaborative study data, this time using an approach similar 297 to the most probable number (MPN) transform of the number of positive carriers N

, 298 299 (3)   	 !ln  

. 300 301 The MPN formula in equation (3) provides a maximum likelihood estimate of the typical 302 number of viable organisms per carrier, averaged ac ross the 60 carrier set (19-21). In the 303 bioassay literature, the ln(MPN) is known as the complementary log-log transform (22, 304 pp 55, 113). 305 306 One of the most significant changes to the UDM over the years has been the addition of a 307 process for measuring the number of viable bacteria on dried inoculated carriers, thus 308 providing for a semi-quantitative test method. In this case, the typical bio-challe nge on 309 each of the inoculated carriers can be estimated by 310 311 "#$%&  '(#'# 

 ) *+,

-./01.2	-311451 6

, 312

313 Page 7 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

8

where the average is taken over the set of all enum erated untreated control carriers in the 314 test. Similar to how Hamilton et al. (18) re-analy zed the HSCT collaborative study, the 315 TestLD and the MPN formula in equation (3) can be subtract ed to estimate a log 316 reduction (LR) value for each test by 317 318 (4) %7  "#$%& !     319 320 (20, 23). More recently, in the analysis of the UD M collaborative study (11), the MPN 321 transform was used to calculate LRs for each UDM te st using equation (4). Calculating 322 LRs using the MPN formulation has been used to anal yze another AOAC standard 323 disinfectant test method against spores (24). 324 325 Each of these approaches used to analyze the UDM or the HSCT transform the numbers 326 of positive carriers to a continuous scale before p erforming a statistical analysis. If one 327 were to plot the probit

, logit and -log 10 (MPN) transforms (see equations (1)-(3)) as a 328 function of the number of positive carriers N , one would observe that all three produce 329 similar decreasing “sideways S” curves (e.g., see t he comparative plots in Collett (22, p 330 54)). We prefer to use the MPN transform instead of the other kinds of transforms that 331 could be applied to modern-day UDM test data. The main reason for this choice is that 332 the availability of control carrier counts enable a LR to be calculated for each test 333 providing results comparable to quantitative disinf ectant tests. Even with control counts 334 available, it is not clear how to calculate a LR fr om the probit and logit transforms in 335 equations (1)-(2). 336 337 Modeling Steps – Use of the UDM Collaborative Study Data 338 339 The primary inputs to the modeling process are: i) the numbers of positive carriers 340 observed in the UDM collaborative study (11), in wh ich there were 3 UDM tests 341 performed at each lab using a presumed “high” effic acy product; and ii) the control 342 carrier counts for each UDM test for both P. aeruginosa and S. aureus . A schematic of 343 the complete modeling process is given in Figure 1. 344 345 Step 1: Calculating log reductions from the number of positive carriers 346 347 The first step of the modeling process was to MPN-t ransform (via equation (3)) the 348 numbers of positive carriers from the UDM collabora tive study to provide estimates of 349 the typical number of viable organisms left on each of the 60 treated carriers. Since 350 control carriers were also enumerated for each test , then the LR was calculated via 351 equation (4). 352 353 Step 2: Estimating the variability of the UDM 354 355 In the second step of the modeling process, the LRs calculated from the UDM 356 collaborative study were analyzed for each microbe separately using a one-way ANOVA 357 with a random effect due to lab (11, 25). The ANOVA provides estimates of two 358 components of variance for the LRs for each microbe separately. One estimated 359 Page 8 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

9

component of variance is due to sources among labs (e.g. different equipment, 360 technicians and geographic locations), denoted by S 2 lab . The second estimated variance 361 component is due to within-lab sources (e.g. fresh inocula and different environmental 362 conditions on the days on which experiments are run ), denoted by S 2 test . The values for 363 the variance components S 2 lab and S

2 test found for P. aeruginosa and S. aureus in the 364 UDM collaborative study are provided in Table 1. Th e estimated reproducibility standard 365 deviation (SD) is calculated from the variance comp onents by 366 367 (5) S R = [ S 2 lab + S

2 test ] 1/2 368 369 (25). Note that S test is the repeatability SD of the LRs and is often de noted by S r . The 370 variance components S 2 lab and S

2 test were used in all subsequent calculations of the 371 expected pass-error and fail-error

rates. Thus, the variability observed in the UDM 372 collaborative study is assumed to be representative of any UDM test. 373 374 375 Step 3: Calculating the error rates for a single UD M test 376 377 The third modeling step relies on a specific perfor mance standard for a single UDM test; 378 namely, a disinfectant passes if no more than PS 1 positive carriers out of the 60 carrier set 379 are observed. The subscript of “1” emphasizes that this is a performance standard for a 380 single UDM test rather than multiple UDM tests. Fo r the current UDM performance 381 standard, PS 1 = 1. We assume that ineffective treatments will n ot meet this performance 382 standard most of the time. In other words, as prev iously defined, the true median number 383 of positive carriers for an ineffective treatment i s at least PS 1 + 1. Thus, for a given 384 performance standard PS 1 , in order to calculate the expected pass-error and fail-error 385 rates, the following competing null and alternative hypotheses were considered 386 387 (6) H 0 : (true median number of positive carriers) ≥ PS 1 + 1 388 H a : (true median number of positive carriers) < PS 1 + 1. 389 390 The null hypothesis H 0 presumes that the disinfectant being tested is not efficacious. The 391 goal of the product’s manufacturer is that the outc ome of the UDM test will provide 392 evidence in favor of H a , that the disinfectant is truly efficacious. Consi stent with the 393 current implementation of the UDM performance stand ard based on a single UDM test 394 outcome, the null H 0 is rejected in favor of the alternative H a if the observed number of 395 positive carriers is less than or equal to PS 1

. 396 397 The statistic T which tests the hypotheses in equation (6) follows Student’s t -distribution 398 with degrees of freedom df 1 , and non-centrality parameter λ 1 = 0. (see Appendix A for 399 details). The Type I pass-error and Type II fail-error rates for the t -test can be found by 400 conventional statistical techniques, and are equal respectively to 401 402 (7) α 1 = Pr (Rejecting H 0 | H 0 : true median N = PS 1 + 1) 403

= Pr ( T ≥ t 1 ; df

1 , λ 1 = 0). 404 (8) β 1 = Pr (Failing to reject H 0 |H a : true median N = 0) 405 Page 9 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

10

= Pr ( T < t 1 ; df

1 , λ 1 ) 406 407 (16). The probability function Pr (

∙ ) is straightforward to evaluate using statistical 408 software (see Appendix B). Explicit formulas for th ese are provided in Appendix A, as 409 well as other details showing how this hypothesis t esting framework leads to the 410 calculation of α 1 and β 1 . The calculations are completely determined by th e available 411 S 2 lab and S 2 test values and the difference between two known LR val ues. The same 412 degrees of freedom df 1 applies to both the Type I and Type II error rate calculations. Also 413 λ 1 = 0 for all Type I error rate calculations and the refore for all α 1 calculations. A non414 zero numerical value of λ 1 had to be calculated only for Type II fail-error rates β 1 , but the 415 calculations were uncomplicated. 416 417 Since the calculations of the error rates depend on differences in LRs, it should be 418 stressed that the error rates do not depend on the magnitude of the control counts (cf. 419 equation (4)), or on the magnitude of the LR values . The error rates depend only on the 420 LR differences just described, and on the presumed variability of these LR values. 421 422 Step 4: Calculating the error rates for multiple UD M tests for a single organism using a 423 “pass all” approach 424 425 Currently, single UDM tests per organism are common ly used to verify efficacy claims in 426 EPA’s post-registration stewardship program; howeve r, for registration purposes, K = 3 427 tests per organism are required and a product must all pass K UDM tests in each of L = 1 428 labs. In the context of testing the hypotheses in equation (6) simultaneously for all KL 429 tests, the pass-error rate for a performance standard that requires such a “pass all” test 430 scheme is 431 432

α KL = Pr (Rejecting H 0 in all KL tests | H 0 : true median N = PS 1 + 1) 433

= Pr (Passing all KL tests | H 0 : true median N = PS 1 + 1) 434 (9) = Pr ( T ≥ t 1 in all KL tests | H 0 : true median N = PS 1 + 1). 435 436 When the KL tests are performed independently of each other, t hen 437 438 (10) α KL = (α 1 ) KL

. 439 440 In the preceding equation, the pass-error rate for a single test, α 1 , was provided in 441 equation (7). 442 443 The probability that a highly effective product fai ls to pass all KL tests is 444 445 β KL = Pr (Failing to reject H 0 in at least one test | H a : true median N = 0) 446 = 1 - Pr (Rejecting H 0 in all KL tests | H a : true median N = 0) 447 (11) = 1 - Pr ( T ≥ t 1 in all KL tests | H a : true median N = 0). 448 449 When the KL tests are performed independently of each other, t hen this equation can be 450 simplified to 451 Page 10 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

11

452 (12) β KL = 1 - (1 - β 1 ) KL

. 453 454 In the preceding equation, the fail-error rate for a single test, β 1 , was provided in equation 455 (8). 456 457 In practice, it is known that UDM tests conducted i n the same lab are not statistically 458 independent of each other since they may be perform ed, for example, by the same 459 technician using similar equipment in the same faci lity and geographic location. The 460 estimated correlation amongst repeated tests in the same lab is 461 462 (13) r = S 2 lab / ( S

2 lab + S

2 test ) 463 464 (17, section 24.1), which is readily calculated for each organism using the values in Table 465 1. The pass-error and fail-error

rates using equations (10) and (12) assume that th e KL 466 tests are independent (and uncorrelated). If the c orrelation in equation (13) is 467 significantly positive, as is the case when there i s a significant lab-to-lab variance 468 component S 2 lab , then equations (10) and (12) provide incorrect er ror rates for the 469 performance standard. To properly account for thes e correlations among tests of the 470 same microbe in the same lab, we used a multivariat e t -distribution to calculate the pass471

error

rate α

KL and the fail-error

rate β KL using equations (9) and (11) (26). The 472 multivariate t -distribution requires the same inputs t 1, df 1 , and λ 1 as were required for 473 equations (7) and (8). An additional input is the matrix of correlations among the KL 474 tests. Statistical software for these calculations is readily available (see Appendix B). 475 476 Alternate Step 4: Calculating the error rates for m ultiple UDM tests for a single 477 organism using averaging 478 479 A possible alternative to requiring that a disinfec tant pass all of multiple UDM tests is to 480 impose a performance standard PS mean on the mean disinfectant response across the K 481 multiple tests performed at each of L labs. Since it is assumed that ineffective treatme nts 482 will not meet this performance standard most of the time, then the median number of 483 positive carriers for an ineffective treatment is a t least PS mean + 1. Thus, the hypotheses 484 being tested in this scenario are 485 486 (14) H 0 : (true median number of positive carriers) ≥ PS mean + 1 487 H a : (true median number of positive carriers) < PS mean + 1. 488 489 In contrast to the pass-all approach, these hypothe ses are tested by calculating the mean 490 response across all KL tests. The chance of a pass-error when averaging over all KL tests 491 is 492 493

α mean = Pr (Rejecting H 0 | H 0 : true median N = PS mean +1) 494

= Pr (Passing all KL tests on average | H 0 : true median N = PS mean +1) 495 (15) = Pr ( T ≥ t mean ; df

mean , λ mean = 0). 496 497 Page 11 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

12

The expected fail-error

rate for a highly effective product (for which the true median 498 number of positive carriers is 0) is calculated by 499 500 β mean = Pr (Failing to reject H 0 | H a : true median N = 0) 501 (16) = Pr ( T < t mean ; df

mean , λ mean ). 502 503 Just as was described for the single test scenario, the above probability calculations 504 correspond to the Type I and Type II error rates fo r a t -test of the hypotheses in (14). 505 These calculations depend on Student’s t -distribution with critical value t mean , degrees of 506 freedom df mean , and a non-centrality parameter λ mean (see Appendix A). 507 508 Step 5: Calculating the error rates for multiple UD M tests for both P. aeruginosa and S. 509 aureus using a “pass all” approach 510 511 Step 4 of the modeling process considered error rat es for each microbe separately; 512 however, the current UDM performance standard, whet her at registration or post513 registration, requires that a disinfectant pass all tests for all organisms under 514 consideration for a hospital disinfectant claim. T o calculate the pass-error and fail-error 515 rates for this scenario, appropriate notation must first be developed. Consider a 516 performance standard that requires K Pa UDM tests against P. aeruginosa and K Sa UDM 517 tests against S. aureus at each of L labs, for a total of K = K Pa + K Sa tests at each of L labs. 518 The hypotheses in equation (6) must now be extended to account for different target 519 numbers of positive carriers for each microbe, 520 521 H 0 : (true median number of positive carriers for a P. aeruginosa test) ≥ PS Pa + 1 522

or 523 (true median number of positive carriers for an S. aureus test) ≥ PS Sa + 1 524 525 H a : (true median number of positive carriers for a P. aeruginosa test) < PS Pa + 1 526 and 527 (true median number of positive carriers for an S. aureus test) < PS Sa + 1. 528 529 Consistent with the current implementation of the U DM performance standard, H 0 is 530 rejected in favor of the alternative H a if the observed number of positive carriers N Pa in 531 each of the K Pa tests against P. aeruginosa is less than or equal to PS Pa and the observed 532 number of positive carriers N Sa in each of the K Sa tests against S. aureus is less than or 533 equal to PS Sa

. 534 535 The Type I pass-error rate α KL for a performance standard that requires that a di sinfectant 536 pass all K = K Pa + K Sa tests for both organisms at each of L labs is calculated by 537 538

α

KL = Pr (Rejecting H 0 in all KL tests | H 0 : true median N Pa = PS

Pa + 1 539 true median N Sa = PS

Sa + 1) 540

= Pr (Passing all KL tests | H 0 : true median N Pa = PS

Pa + 1 541 true median N Sa = PS

Sa + 1) 542 (17) = Pr ( T ≥ t 1 in all KL tests | H 0 : true median N Pa = PS

Pa + 1) 543 Page 12 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

13 true median N Sa = PS

Sa + 1). 544 545 If it is assumed that, within each lab, the P. aeruginosa test results are independent of the 546 S. aureus test results, then the pass-error rate in equation (17) can be calculated by 547 548

8 9:  8 9 ;< : 8 9 =< : 

. 549 550 In the previous equation, 8 9 ;< : is the Type I pass-error rate for K

Pa P. aeruginosa tests 551 and 8 9 =< : is the pass-error rate for K

Sa S. aureus tests, with all K = K Pa + K

Sa tests 552 performed in each of L labs. The rates 8 9 ;< : and 8 9 =< : can each be calculated using 553 equation (9). 554 The probability that a highly effective product fai ls to pass all KL tests is 555 556 β KL = Pr (Failing to reject H 0 in at least one test | H a : true median N Pa = 0 557 true median N Sa = 0) 558 = 1 - Pr (Rejecting H 0 in all KL tests | H a : true median N Pa = 0 559 true median N Sa = 0) 560 (18) = 1 - Pr ( T ≥ t 1 in all KL tests | H a : true median N Pa = 0 561 true median N Sa = 0). 562 563 If it is assumed that, within each lab, the P. aeruginosa test results are independent of the 564 S. aureus test results, then this equation can be simplified to 565 566 > 9:  1 ! 1 ! > 9 ;< : 1 ! > 9 =< :  567 568 where > 9 ;< : and > 9 =< : are the Type II fail-error rates for the K

Pa P. aeruginosa tests and 569 the K

Sa S. aureus tests respectively, with all K = K Pa + K Sa tests performed in each of L 570 labs. The fail-error rate for each organism can be calculated by equati on (11). 571 572 We have already explained that tests using the same microbe conducted in the same 573 laboratory may not be uncorrelated. For similar re asons, P. aeruginosa test results may 574 be correlated with S. aureus test results when the tests are conducted in the s ame 575 laboratory. Although the UDM collaborative study w as explicitly designed to reliably 576 estimate the correlation among LRs of tests of the same microbe conducted in the same 577 laboratory, the data from the UDM collaborative did not provide a reliable estimate of the 578 correlation among tests of different microbes condu cted in the same laboratory. Thus, we 579 calculated the error rates for a range of hypotheti cal correlation values: 0.00, 0.25, 0.5, 580 0.95, and then reported the range of error rates fo und. 581 582 To account for the between microbe correlation, a m ultivariate t -distribution (26) was 583 used to calculate the error rates for a performance standard which requires that all tests 584 from both organisms must be passed. The necessary inputs for the calculations using the 585 multivariate t

are: t 1 and λ 1 calculated for each organism separately. Unlike th e values 586 used in equations (7) and (8) for each microbe sepa rately, the degrees of freedom df 1 = 587 6.9 for P. aeruginosa was used since it was smaller and thus more conser vative. The 588 Page 13 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

14

calculations were also performed for df 1 = 4, the number of labs minus one, which 589 resulted in negligibly different error rates than t hose reported. Given these inputs, as well 590 as the matrix of correlations amongst the tests, st atistical software is readily available to 591 perform the calculations (see Appendix B). In some instances, for the between microbe 592 correlation values of either 0.5 or 0.95, the corre lation matrix was no longer positive 593 definite, in which case the error rates were not ca lculable. An intuitive explanation of 594 this failure of the model to generate the error pro babilities of interest is that the observed 595 variability of the two organisms is inconsistent wi th these high hypothetical correlations. 596 597 Modeling Assumptions 598 599 Figure 1 provides an overview of the assumptions us ed at each step of the modeling 600 process. The first critical assumption of the mode l is that the expected variability (i.e. the 601 values of the variance components S 2 lab and S

2 test ) of any product subjected to the UDM is 602 assumed to be the same as the variability observed in the UDM collaborative study. A 603 possible critique of this assumption is that only a single high efficacy product was used to 604 calculate these variances. Although conventional g uidelines for collaborative studies 605 recommend the inclusion of multiple disinfectant fo rmulations (25), a single disinfectant 606 study can yield results of practical value. If ava ilable resources are limited, as was the 607 case for the UDM collaborative study, then it is ad visable to use a single disinfectant in 608 all tests; otherwise, to reduce cost, one would be forced to use less laboratories in which 609 case the reproducibility estimates S R would be less reliable (i.e., S 2 R would have lower 610 degrees of freedom). For similarly efficacious prod ucts having similar LR values within a 611 narrow range, experience shows that the reproducibi lity SDs, S R , among the products are 612 also similar. Highly efficacious treatments produce small S R values; completely 613 ineffective treatments produce small S R values; and partially efficacious treatments 614 produce medium or high S R values (27-29). Therefore, we believe that the 615 reproducibility SD, S R , produced by the high efficacy treatment in the si ngle disinfectant 616 study of the UDM is representative of the reproduci bility for most other high efficacy 617 disinfectants and that the observed S R can be used when crafting performance standards. 618 Support for this claim is found in the multi-lab co llaborative study results for the HSCT. 619 The reproducibility SD, S R , for the single disinfectant UDM study are consist ent with the 620 S R values for two earlier multi-disinfectant HSCT col laborative studies for P. aeruginosa 621 (18, 30). For all the different high efficacy disin fectant formulations tested in the UDM 622 and HSCT collaborative studies, the range of S R values was 0.09 to 0.66 and the median 623 S R was slightly less than 0.50. Thus, if an additiona l single disinfectant collaborative 624 study of the UDM was conducted with P. aeruginosa with a different high efficacy 625 disinfectant, one would expect the resulting S R to be about 0.50. Note that the statistical 626 calculations for the UDM performance standards pres ently being proposed in this paper 627 are based on S R

= 0.53 for P. aeruginosa from the UDM collaborative study. The pass628

error

and fail-error

rates will be smaller for more effective disinfect ants (since they are 629 presumed to have smaller variability) and larger fo r less-effective disinfectants (that 630 exhibit larger variability). 631 A second assumption of importance to the calculatio n of the error rates is that the LRs, 632 calculated from the control carrier counts and the MPN-transformed numbers of positive 633 carriers, are normally distributed with homogeneous within-lab variance component S 2 test 634 Page 14 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

15

across all labs for each microbe. By equation (13), the homogeneous variance assumption 635 is equivalent to assuming a homogeneous correlation amongst tests using the same 636 microbe in the same lab. These assumptions are easi ly checked with well established 637 diagnostic tools. In our experience, the LR values usually satisfy the assumptions of 638 normality and homogeneity of variance across a vari ety of different test organisms and 639 standard methods (11, 18, 24, 31-34). 640 641 The third assumption, when calculating the error ra tes for a performance standard which 642 requires that all tests from both P. aeruginosa and S. aureus must be passed, was that the 643 LRs from tests of different microbes conducted in t he same lab have the same correlation 644 in all labs. Although it was not possible to relia bly estimate this correlation from the 645 UDM collaborative study, the inability to find a si gnificant correlation, and the disparate 646 among-lab variance of each of the two organisms (Ta ble 1), is suggestive that such a 647 correlation, if it exists, is small. Nonetheless, we calculated the error rates for a wide 648 range of hypothetical correlation values: 0.00, 0.2 5, 0.5, and 0.95. 649 650 A fourth critical assumption is that the pass-error rate is calculated with respect to a 651 “marginally ineffective” product that has a median number of positive carriers that 652 exceeds the performance standard by one positive ca rrier. The pass-error rate could have 653 been calculated for a different ineffective product , for example, of a highly ineffective 654 product with true median number of positive carrier s equal to twenty. Nonetheless, the 655

pass-error

rate for a marginally ineffective product is large r than the error rate for any 656 other ineffective product. 657 658 The fifth and last fundamental assumption is that t he fail-error

rates have been calculated 659 with respect to a highly efficacious disinfectant t hat has a median number of positive 660 carriers of zero over many tests. One could instea d calculate the fail-error rate for a 661 “marginally efficacious” product, for example, when the true median number of positive 662 carriers just meets the allowable number of positiv e carriers. Such a product would have 663 a higher failure rate than a highly effective produ ct. 664 665 Estimates of Log Reduction 666 667 One defining feature of the modeling approach prese nted in this paper is transforming the 668 number of positive carriers observed in a 60 carrie r UDM test to a LR value. The 669 advantages of using the LR instead of analyzing the numbers of positive carriers some 670 other way are: 1) the resulting value is directly i nterpretable as the LR of organisms over 671 the entire 60 carrier set; 2) the results of the va riability of the method can be directly 672 compared to other studies of disinfectant tests, wh ich are usually analyzed with respect to 673 the LR; 3) for analyses such as ANOVA and t -tests, appropriate for small samples of 674 normally distributed LRs, can be directly applied; and 4) the normality assumption is 675 easily verified with well known diagnostic tools. 676 677 It is known that the actual LRs for a disinfectant are likely larger than what is estimated 678 by the MPN formula when zero positive carriers are observed. This is the same challenge 679 faced by other possible analyses of UDM data such a s probit and logit (see equations (1) 680 Page 15 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

16

and (2)). If it were possible to enumerate organism s on all 60 treated carriers and estimate 681 a LR that way, then statistical theory maintains th at this LR, and the LR calculated from 682 the MPN formula, would both estimate the same true mean LR of the disinfectant applied 683 in the UDM. Nonetheless, it is not to be expected t hat the LR calculated for a disinfectant 684 based on UDM outcomes will be predictive of the LR value obtained for the same 685 disinfectant applied in a different method, such as the ASTM Quantitative Carrier Disk 686 Test (29). 687 688 One alternative to the MPN transform approach is th e use of the probit transform (see 689 equation (1)) employed by Ortenzio and Stuart (5), which provides the justification for 690 the current performance standard of passing a disin fectant in a single UDM test. 691 Unfortunately, the probit response variable, which has units of normal perce ntiles, has no 692 practical meaning with respect to the UDM or the di sinfectant being tested. A second 693 alternative analysis is the logit analysis (in equation (2)). There are many other possible 694 transforms of UDM numbers of positive carriers as w ell (33). Unlike a probit transform, 695 the logit

response is interpretable as the log of the odds o f getting a negative carrier out of 696 60. Nonetheless, using either the probit or logit models, the variance components are not 697 interpretable with respect to other disinfectant te sts which use LRs, and in contrast to the 698 MPN formula, neither explicitly estimates the numbe r of organisms killed in the test. 699 Like the ANOVA models which are fit to MPN-transfor med UDM data, the iteratively 700 reweighted least squares techniques used to analyze probit and logit

transformed data 701 also yield maximum likelihood estimates of model pa rameters (17, section 14.4). 702 However, although ANOVA models of LRs provide valid statistical inference for 703 variance components and other questions of interest for small numbers of LRs (since LRs 704 tend to be normally distributed), attempting the sa me inference via probit and logit 705 analyses is only valid for large sample sizes (17, section 14.7). A last critique of non706 MPN transform approach is that to account for the c ontrol counts, a covariate would need 707 to be added to the generalized linear model. This imposes the additional assumption that 708 changes in the control carrier counts affect the nu mber of positive carriers similarly 709 across all tests. Use of the LR values calculated by using the MPN formula (equation 710 (4)) has no such additional assumption. 711 712 Results and Discussion 713 714 Error rates of the current performance standard for a single UDM test 715 716 When a single UDM test is performed, the expected pass-error and fail-error

rates using 717 the current UDM performance standard which allows u p to PS 1 = 1 positive carrier can 718 be determined. For P. aeruginosa , to calculate the chance of a pass-error

of an 719 ineffective treatment (which has median number of p ositive carriers equal to 2), the test 720 statistic critical value is 721 722

t 1 = (LR(0) – LR(2))/ [0.175 + 0.111] 1/2 723 = (8.1038 – 7.8783)/0.5348 724 = 0.4217. 725 726 Page 16 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

17

The variance components in the denominator in the p receding calculation were obtained 727 from the UDM collaborative study data reproduced in Table 1. The degrees of freedom 728 using equation (19) in the Appendix was found to be df 1 = 6.9. Applying equation (7), 729 the pass-error

rate for a single UDM test using P. aeruginosa is α 1 = 0.343. To 730 determine the fail-error rate, equation (20) in Appendix A shows that the n on-centrality 731 parameter is 732 733 λ 1 = (LR( N = 1) – LR(

N = 2))/ [0.175 + 0.111] 1/2 734 = (8.5846 – 7.8783)/0.5348 735 = 1.32. 736 737 Applying equation (8), the fail-error rate is β 1 = 0.182. Similar calculations for S. aureus 738 yield α 1 = 0.244 and β 1 = 0.064 based on t -distributions with df 1 = 13.8 and λ 1 = 2.23. 739 The error rates for P. aeruginosa and S. aureus for a single UDM test are presented in the 740 top two rows of Table 2. The commands which genera ted these values using the 741 statistical software R are provided in Appendix B. These pass-error rates are much 742 higher than the 0.05 value arrived at by the unpubl ished data of Ortenzio & Stuart (5), 743 which presumably was based on UDM tests using S. aureus and S. choleraesuis . The 744 authors hypothesize that this large difference in pass-error rates is due to the fact that 745 their calculations were based on the observed withi n-lab variance from a single lab, 746 whereas our calculations are based on a within-lab variance averaged across multiple 747 labs. 748 749 Error rates of the current performance standard for multiple UDM tests 750 751 For product registration, a product must pass all o f K = 3 tests, all in a single lab, with 752 PS 1 = 1, for each test of P. aeruginosa and S. aureus

. 753 754 Based on the UDM collaborative study data set, the lab-to-lab variance component for S. 755

aureus was not significant and estimated to be zero. Thu s, one may assume that S. 756

aureus tests conducted within the same laboratory are unc orrelated, and so equation (10) 757 yields a good approximation of the pass-error rate of ineffective disinfectants which are 758 required to pass all of three S. aureus tests, 759 760

α KL = (0.2438) 3 = 0.015. 761 762 The fail-error

rate for a highly efficacious product applied to t hree independent S. aureus 763 UDM tests is predicted to be (via equation (12)) 764 765 β KL = 1 - (1 – 0.0644) 3 = 0.181. 766 767 These calculations are very close to the ones prese nted in Table 2. They are not exactly 768 the same since Table 2 was calculated via equations (9) and (11) using a multivariate t 769 calculation. 770 771 Page 17 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

18

Similar calculations assuming independent tests can not be performed for P. aeruginosa 772 since the results from the UDM collaborative study found significant lab-to-lab 773 variability (see Table 1). Equation (13) shows tha t there is a correspondingly high 774 correlation of r

= 0.61 amongst P. aeruginosa tests conducted in the same lab. Thus, the 775 error rates for P. aeruginosa must be found via equations (9) and (11) using a 776 multivariate t

calculation. The results of these calculations ar e provided in Table 2. The 777 code which generated these error rates using the st atistical software R is provided in 778 Appendix B. 779 780 Comparing theoretical and empirical UDM error rates 781 782 It is instructive to compare the theoretical pass-error rates just derived for the current 783 performance standard with the empirical pass-error rate (for an ineffective low efficacy 784 treatment) and the empirical fail-error rate (for an effective high efficacy treatment) 785 observed in the UDM collaborative study. These emp irical rates over 15 tests are 786 presented in Table 3. Unlike the assumed median num bers of positive carriers in Table 2, 787 the observed median values for P. aeruginosa was 1 (instead of 0) for the effective 788 treatment and 15 (instead of 2) for the ineffective treatment. For S. aureus , the observed 789 median value was 0 for the effective treatment, and 4 for ineffective treatment. Thus, 790 except for the highly effective treatment applied t o S. aureus , the values presented in 791 Table 3 are estimated lower bounds of the theoretic al pass-error rates and upper bounds 792 of the theoretical fail-error rates provided in Table 2. Even though only 15 te sts were 793 conducted, the resulting bounds agree with the theo retically obtained values, except for 794 the fail-error

rate for the highly effective treatment applied to S. aureus . A larger 795 collaborative study with many independent tests wou ld need to be run, with a highly 796 effective disinfectant with median number of positi ve carriers N = 0, and a marginally 797 ineffective treatment with median number of positiv e carriers N = 2, in order to 798 empirically corroborate the theoretical results in Table 2. 799 800 Proposing a new performance standard for the UDM 801 802 This paper provides the explicit calculations neces sary to determine the pass-error and 803 fail-error

rates for performance standards which require pass ing only a single test, 804 passing all of multiple tests, or passing multiple tests on average. In order to find a 805 performance standard which keeps both the pass-err or and fail-error rates as low as 806 possible (e.g., <10%) while providing for a minimum amount of testing, a set of possible 807 testing scenarios and performance standards for the UDM was investigated. The set was 808 constructed by varying: the allowable numbers of po sitive carriers for each test; the 809 numbers of tests that must be passed; and different numbers of labs performing the tests. 810 The results, calculated via equations (9) and (11), are provided in Table 4 for P. 811 aeruginosa and in Table 5 for S. aureus . Tables 4 and 5 provide results for a 812 performance standard on a single microbe, and allow a direct comparison to the UDM 813 and HSCT performance standard error rates suggested by other authors (5, 10, 15). Table 814 4 suggests that for a performance standard based on ly on tests of P. aeruginosa , a multi815 lab testing scenario would be critical to maintain a pass-error rate of 10% or less and a 816 Page 18 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

19

fail-error rate of 5% or less (either two tests in each of tw o labs or one test in each of 817 three labs). 818 819 Table 6 provides the pass-error rates for a performance standard that requires tha t all 820 tests from both P. aeruginosa and S. aureus pass (calculated via equations (17) and (18). 821 Since the calculation of the error rates in Table 6 requires an estimate of the correlation 822 between microbes, and we do not have a reliable est imate of this correlation, these 823 calculations were performed for a range of hypothet ical correlation values. The model’s 824 output in Table 6 may be used as a tool to develop a practical and informed decision on 825 the best approach to managing registration testing requirements. Table 6 suggests that in 826 order to simultaneously maintain pass-error and fail-error

rates less than 5%, three tests 827 (e.g., testing three production lots or testing a s ingle lot three times) must be run for each 828 organism, all possibly performed in the same lab. U sing this proposal, a disinfectant 829 would pass each P. aeruginosa UDM test if the number of positive carriers N ≤ 6 (i.e., no 830 more than 6 positive carriers out of 60 tested) and would pass each S. aureus UDM test if 831 N ≤ 3 (i.e., no more than 3 positive carriers out of 60 tested), in which case the pass-error 832 rate of marginally ineffective products over all si x tests is predicted to be between 1.2% 833 and 2.8% and the fail-error rate of highly effective products is predicted to be between 834 4.9% and 5.0%. 835 836 The error rates in Table 6 presume that all future UDM tests applied to any disinfectant 837 exhibit the same variability as observed in the UDM collaborative study, for which the 838 among-lab variance for S. aureus was estimated to be zero. It is possible that this 839 variance component of the LRs is under-estimated si nce 10 of the 15 S. aureus tests 840 exhibited zero positive carriers. To investigate t he influence that a positive among-lab 841 variance would have on the error rates, the model u sed to generate Table 6 was rerun, but 842 this time with an among-lab S. aureus variance equal to the observed among-experiment 843 variance (i.e., S 2 lab = S

2 test = 0.100). The results are presented in Table 7. O ver the range 844 of hypothetical correlations, the range of pass-error rates for the proposed performance 845 standard increased from 1.2%-2.8% to 4.2%-10.5%, an d the range of fail-error rates 846 increased from 4.9%-5.0% to 9.2%-10.6%. Thus, in t he presence of a moderate level of 847 among-lab variability for S. aureus , the proposed performance standard maintains a pass848

error

rate of at most 10.5% and a fail-error rate of at most 10.6%. In this scenario, to 849 maintain a pass error rate less than 10% or less, m ultiple tests, possibly performed at 850 different labs, are required (Table 7). 851 852 Another approach explored during this exercise was to consider the averaging of UDM 853 data across multiple tests. However, in order to a chieve a pass-error rate at or below 854 10% for P. aeruginosa , a 10 lab testing scenario was required, with two UDM tests 855 performed at each lab (these rates were calculated via equations (15) and (16)). This 856

pass-error

rate was attained with only three uncorrelated tes ts using the “pass-all” 857 approach. Thus, for the UDM, an averaging performa nce standard approach appears to 858 be too resource intensive. 859 860 The Tables in this manuscript provide point estimat es for the pass-error and fail-error 861 rates for different performance standards for the U DM. One could also provide, via 862 Page 19 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

20

computer simulations, a margin of error and confide nce interval for each error rate. Such 863 calculations are beyond the scope of this manuscrip t. The computer code that generated 864 the error rates in Tables 4-7 using the statistical software R is provided in Appendix B. 865 866 Conclusions 867 868 Concurrent with the development and assessment of n ew quantitative methodology 869 designed to replace the UDM, the EPA continues to w ork with AOAC and the 870 stakeholder community to facilitate the introductio n, review and publication of editorial 871 and procedural revisions to standard methods. The current investigation of the UDM 872 performance standard utilizes the best available sc ientific practices for assessing the 873 available data and is one of many aspects of the me thod that has been presented for 874 revision. 875 876 The statistical evaluation of the multi-laboratory UDM collaborative data provides a 877 unique opportunity for re-evaluating the current UD M performance standard. This paper 878 provides the quantitative tools, including computer code, to evaluate a number of options 879 for a new performance standard. As other authors ha ve proposed, we believe that any 880 proposed performance standard for tests used by reg ulatory agencies should be based on 881 a quantitative assessment of the pass-error and fail-error

rates. Based on this assessment, 882 the pass-error

rate for a single UDM test, for either P. aeruginosa or S. aureus , is much 883 higher than originally suggested by previous studie s of the UDM and HSCT. This 884 initiative serves to reduce the error rates while m aintaining a practical level of testing

. 885 Historically, lower pass-error rates have been achieved by EPA by requiring a reg istrant 886 to submit efficacy data on three batches of the sam e product for registration. 887 Furthermore, due to the EPA’s registration specific ations for a hospital disinfectant which 888 involve efficacy data for S. aureus and P. aeruginosa , a performance standard should 889 require that a hospital disinfectant pass all tests (six total) against both P. aeruginosa and 890 S. aureus (see Tables 6 and 7). In conjunction with the numb er of tests conducted, use of 891 N ≤ 6 for P. aeruginosa and N ≤ 3 for S. aureus as the passing requirement for each test 892 should improve the overall confidence in the outcom e of the testing scheme. Thus, two 893 testing scenarios may be considered: A) testing eac h of three production lots one time per 894 microbe, and B) testing a single production lot thr ee times per microbe. Any of the six 895 tests that fail to meet the performance standard wo uld fail the product. Pass-error rates of 896 0.3 to 1.0% and fail-error rates of 41.6 to 45.1% are predicted using the cur rent 897 performance standard (i.e., N ≤ 1 for both microbes and three production lots te sted per 898 microbe). When error rates for the proposed perform ance standard ( N ≤ 6 for P. 899 aeruginosa and N ≤ 3 for S. aureus ) are calculated using the reproducibility standard 900 deviations observed in the collaborative study, a l ow pass-error rate between 1.2% and 901 2.8% is predicted for marginally ineffective produc ts and the fail-error rate for highly 902 effective products is predicted to be lowered to ap proximately 5% (Table 6). The 903 requirement of N ≤ 1 for each of three tests (three batches of prod uct tested for 904 registration) for S. enterica would remain the same. 905 906 In addition to the proposed new performance standar ds, meeting the minimum and 907 maximum carrier count requirements (mean log densit y of 6-7 per carrier) and the use of 908 Page 20 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

21

the most current and revised version of UDM further strengthens the reliability of the 909 UDM and the confidence of the test data. Furthermo re, if the new performance standard 910 is implemented, it would be possible to reassess th e performance standard again 911 following the generation of additional data. 912 913 Based on the information provided in this report, t he EPA is seeking AOAC’s technical 914 review of the model and the associated calculations . Based on this modeling exercise, 915 proposed revisions to Methods 955.15 and 964.02 are provided. 916 917 The following revisions are proposed following sect ion C ( f ) of each method: 918 919 ( g ) Performance standard .–For method 955.15 ( S. aureus ), Section C (h): 920 Performance standard: ≤3 positive (growth) carriers out of 60 tested. Conduct 921 three separate UDM tests, one test conducted per da y. Each test must meet the 922 performance standard. Testing at a single laborator y is acceptable. 923 924 ( g ) Performance Standard .–For method 964.02 ( P. aeruginosa ), Section C (h): 925 Performance standard: ≤6 positive (growth) carriers out of 60 tested. Conduct 926 three separate UDM tests, one test conducted per da y. Each test must meet the 927 performance standard. Testing at a single laborato ry is acceptable. 928 929 Acknowledgements 930 931 This work was performed under contract EP-W-08-014 between Montana State 932 University and EPA OPP Microbiology Laboratory Bran ch (MLB). The authors 933 gratefully acknowledge substantial contributions fr om Dr. Darla Goeres at the Center for 934 Biofilm Engineering at Montana State University in the preparation of this manuscript. 935 936 References 937 938 (1) Official Methods of Analysis (2012) 18 th Ed., AOAC INTERNATIONAL, 939 Gaithersburg, MD, Method 955.14 940 (2) Official Methods of Analysis (2012) 18 th Ed., AOAC INTERNATIONAL, 941 Gaithersburg, MD, Method 955.15 942 (3) Official Methods of Analysis (2012) 18 th Ed., AOAC INTERNATIONAL, 943 Gaithersburg, MD, Method 964.02 944 (4) Tomasino, S.F., Pines, R.M., & Hamilton, G.C. ( 2012) J. AOAC Int

. 95 (in press) 945 (5) Ortenzio, L.F., & Stuart, L.S. (1961) J. AOAC Int

. 44 , 416-421 946 (6) Morgan, B.J. (1992) Analysis of Quantal Response Data, Chapman & Hall, New 947 York, NY 948 (7) Cole, E.C., & Rutala, W.A. (1988) J. AOAC Int. 71 , 1187-1194 949 (8) Rutala, W.A., & Cole, E.C. (1987) Infect. Control 8 , 501-506 950 (9) Official Methods of Analysis (2009) 18 th Ed., AOAC INTERNATIONAL, 951 Gaithersburg, MD, Method 991.47 952 (10) Rubino, J.R., Bauer, J.M., Clarke, P.H., Woodw ard, B.B., Porter, F.C., & Hilton, 953 H.G. (1992) J. AOAC Int. 75, 635-645 954 Page 21 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

22

(11) Tomasino, S.F., Parker, A.E., Hamilton, M.A., & Hamilton, G.C. (2012) J. AOAC 955 Int

. 96 ( in press ) 956 (12) Standard Operating Procedure (MB-05) for condu cting the Use-dilution method, 957 U.S. Environmental Protection Agency - Office of Pe sticide Programs, Biological 958 and Economic Analysis Division, Microbiology Labora tory Branch 959 (13) Tilt, N., & Hamilton, M.A. (1999) J. AOAC Int. 82 , 384-389 960 (14) Trout, J. (1985) J. AOAC Int

. 68 ,763-765 961 (15) Arlea, C., King, S., Bennie, B., Kemp, K., Mer tz, E., & Staub, R. (2008) J. AOAC 962 Int. 91 , 152-158 963 (16) Casella & Berger (1990) Statistical Inference, Duxbury Press, Belmont, CA 964 (17) Neter, J., Kutner, M.H., Wasserman, W., & Nach tsheim, C.J. (1996) Applied 965 Linear Statistical Models , McGraw-Hill Co., Boston, MA 966 (18) Hamilton, M.A., DeVries, T.A., & Rubino, R.R. (1995) J. AOAC Int

. 78 , 1102- 967 1109 968 (19) Hamilton, M.A. (2011) The P/N formula for the log reduction when using a semi969 quantitative disinfectant test of type SQ1, KSA-SM-08 , Feb 1, 2011, Center for 970 Biofilm Engineering at Montana State University, Bo zeman, MT. 971 http://www.biofilm.montana.edu/resources/knowledge_ sharing_articles 972 (20) Cochran, W.G. (1950) Biometrics 6 , 105-116 973 (21) McCrady, M.H. (1915) J Infect. Dis

. 17 , 183-212 974 (22) Collett, D. (1991) Modelling Binary Data. Chapman & Hall, New York, NY 975 (23) Blodgett, R.J. (2006) Bacteriological Analytical Manual , FDA, 976 http://www.fda.gov/Food/SceinceResearch/LaboratoryM ethods/BacterilogicalAna 977 lytical ManualBAM/default.htm, Appendix 2 978 (24) Tomasino, S.F., & Hamilton, M.A. (2006) J. AOAC Int. 89 , 1373-1397 979 (25) Official Methods of Analysis (2005) 18th Ed., AOAC INTERNATIONAL, 980 Gaithersburg, MD, Appendix D 981 (26) Genz, A., & Bretz, F. (2009) Computation of Multivariate Normal and t 982 Probabilities Springer-Verlage, Heidelberg, Germany. ISBN 978-3- 642-01688-2 983 (27) Bloomfield, S.F., & Looney, E. (1992) J. Appl. Bacteriol. 73 , 87-93 984 (28) Bloomfield, S.F., Arthur, M., Van Klingeren, B ., Pullen, W., Holah, J.T., & Elton, 985 R. (1994) J. Appl. Bacteriol. 76 , 86-94 986 (29) Springthorpe, V.S., & Sattar, S.A. (2005) J. AOAC Int. 88 , 182-201 987 (30) Hamilton, M.A. (2001) “Statistical report: pre liminary analysis of the second 988 collaborative study of the hard surface carrier tes t,” Aug 31, 2001, EPA contract 989 #68-W-99-015 990 (31) Tomasino, S.F., Pines, R.M., Cottrill, M.P., & Hami lton, M.A. (2008) J. AOAC 991 Int. 91 , 833-852 992 (32) Tomasino, S.F., Pines, R.M., & Hamilton, M.A. (2009 ) J. AOAC Int. 92 , 1531- 993 1540 994 (33) Hamilton, M.A., & DeVries, T.A. (1996) Biometrics 52 , 1112-1120 995 (34) Buckingham-Meyer, K., Goeres, D., & Hamilton, M. (2 007) J. Microbiol. Meth. 996 70 , 236-244 997 (35) McCullagh, P., & Nelder J. (1989) Generalized Linear Models , Chapman & Hall, 998 New York, NY 999 (36) Nijhuis, M.B., & Van den Heuvel, E.R. (2007) J Biopharm. Stat. 17, 123–142 1000 Page 22 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

23

(37) Mee, R. (1984) Technometrics 26 , 251-254 1001 (38) R Development Core Team (2010), R: A language and environment for statistical 1002 computing, R Foundation for Statistical Computing, Vienna, Austria, ISBN 3- 1003 900051-07-0, http://www.R-project.org 1004 (39) Genz, A., Bretz, F., Miwa, T., Mi, X., Leisch, F., Scheipl, F., & Hothorn, T. 1005 (2011) mvtnorm: Multivariate Normal and t Distribut ions, R package version 0.9- 1006 96, http://CRAN.R-project.org/package=mvtnorm 1007 1008 1009

1010 Figure 1. An overview of the modeling process and assumptions used to calculate the 1011 two error rates. The number of positive carriers o ut of 60 observed in a UDM test is 1012 denoted by N

. The maximum allowable number of positive carrier s allowed by a 1013 performance standard is denoted by PS . A log reduction is denoted by LR. 1014 1015 Table 1. Variance components and reproducibility SD of LR fo r each of the two 1016 microbes in the UDM collaborative study (11, Table 6). 1017

Organism Among lab variance ( S 2 lab ) Within

lab variance ( S 2 test ) Ratio

S 2 lab / S 2 test Intra-lab correlation Reproducibility

SD

P. aeruginosa 0.175 0.111 1.5766 0.6119 0.5348 S. aureus 0.000 0.100 0.0000 0.0000 0.3162 1018 1019 Page 23 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

24

Table 2. Estimated error rates, expressed as percentages, fo r the current UDM 1020 performance standard (which allows up to 1 positive carrier out of a 60 inoculated and 1021 treated set). “Three tests in one lab” is the curr ent registration performance standard; 1022 “one test” is the current post-registration perform ance standard. 1023

Organism PASS ALL of this Performance standard Pass-error Fail-error number of tests required to PASS rate (α) rate (β) each test (median N = 2) (median N

= 0)

Pa 1 test N ≤ 1 34.30% 18.19%

Sa 1 test N ≤ 1 24.38% 6.44%

Pa 3 tests in 1 lab N ≤ 1 15.50% 33.45%

Sa 3 tests in 1 lab N ≤ 1 1.59% 18.01% 1024 1025 Table 3. Empirical error rates, expressed as percentages, for the current UDM 1026 performance standard (which allows up to 1 positive carrier out of a 60 inoculated and 1027 treated set) (11, Tables 8 and 9, Figures 1 and 2). Unlike the assumed median numbers of 1028 positive carriers in Table 2, the observed median v alues for the high efficacy treatment 1029 applied to P. aeruginosa was 1 (instead of 0), and 15 (instead of 2) for th e low efficacy 1030 treatment. For S. aureus , the observed median values were 0 for the high ef ficacy 1031 treatment, and 4 for the low efficacy treatment. T hus, the values in this table are bounds 1032 for the theoretical rates given in Table 2. Rates are reported as percentages. 1033 1034

Organism PASS ALL of this Performance standard Pass-error Fail-error number of tests required to PASS rate (α) rate (β)

each test

Pa 1 test N ≤ 1 ≥ 0% (0/15) ≤ 26.7% (4/15)

Sa 1 test N ≤ 1 ≥ 20% (3/15) ≤ 33.3% (5/15)

Pa 3 tests in 1 lab N ≤ 1 ≥ 0% (0/15) ≤ 60% (9/15)

Sa 3 tests in 1 lab N ≤ 1 ≥ 0% (0/15) 60% (9/15) 1035 1036 Page 24 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

25

Table 4. Estimated error rates, expressed as percentages, f or hypothetical 1037 performance standards for P. aeruginosa that require a disinfectant to pass all of 1038 multiple tests as the numbers of labs, numbers of t ests at each lab, and the 1039 allowable number of positive carriers ( N ) out of a 60 carrier set per test is varied. 1040 The number of labs is critical due to the large lab -to-lab variance for P. 1041 aeruginosa

. 1042

Num.

labs Num. tests at each lab PASS ALL

of this

num. tests Performance

standard to PASS each

test Pass-error

rate (α) Fail-error

rate (β)

1 1 1 N ≤ 1 34.30% 18.19%

1 1 1 N ≤ 2 39.38% 9.23%

1 1 1 N ≤ 3 41.94% 5.42%

1 1 1 N ≤ 4 43.49% 3.47%

1 1 1 N ≤ 5 44.52% 2.35%

1 1 1 N ≤ 6 45.26% 1.67%

1 3 3 N ≤ 1 15.50% 33.44%

1 3 3 N ≤ 2 19.20% 18.88%

1 3 3 N ≤ 3 21.23% 11.81%

1 3 3 N ≤ 4 22.49% 7.89%

1 3 3 N ≤ 5 23.36% 5.53%

1 3 3 N ≤ 6 23.97% 4.02%

2 1 2 N ≤ 1 12.02% 32.91%

2 1 2 N ≤ 2 15.64% 17.56%

2 1 2 N ≤ 3 17.68% 10.64%

2 1 2 N ≤ 4 18.98% 6.80%

2 1 2 N ≤ 5 19.87% 4.66%

2 1 2 N ≤ 6 20.53% 3.30%

2 2 4 N ≤ 1 4.62% 47.10%

2 2 4 N ≤ 2 6.58% 27.43%

2 2 4 N ≤ 3 7.78% 17.25%

2 2 4 N ≤ 4 8.57% 11.52%

2 2 4 N ≤ 5 9.14% 8.04%

2 2 4 N ≤ 6 9.55%* 5.81%*

3 1 3 N ≤ 1 4.26% 44.92%

3 1 3 N ≤ 2 6.24% 25.12%

3 1 3 N ≤ 3 7.47% 15.35%

3 1 3 N ≤ 4 8.29% 10.03%

3 1 3 N ≤ 5 8.88% 6.89%

3 1 3 N ≤ 6 9.31% 4.91% 1043 Page 25 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

26

Table 5. Estimated error rates, expressed as percentages, for hypothetical 1044 performance standards for S. aureus that require a disinfectant to pass all of 1045 multiple tests as the numbers of tests and the allo wable number of positive 1046 carriers (

N ) out of a 60 carrier set per test is varied. The error rates are 1047 representative of either single or multiple lab sce narios (due to the insignificant 1048 lab-to-lab variance for S. aureus ). 1049

PASS ALL of this

num. tests Performance standard

required to PASS each

test Pass-error rate (α) Fail-error rate (β)

1 N ≤ 1 24.38% 6.44%

1 N ≤ 2 32.15% 1.28%

1 N ≤ 3 36.32% 0.34%

1 N ≤ 4 38.89% 0.11%

2 N ≤ 1 6.15% 12.43%

2 N ≤ 2 10.46% 2.54%

2 N ≤ 3 13.27% 0.68%

2 N ≤ 4 15.18% 0.22%

3 N ≤ 1 1.59% 18.01%

3 N ≤ 2 3.43% 3.78%

3 N ≤ 3 4.87% 1.01%

3 N ≤ 4 5.94% 0.33% 1050 Page 26 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

27

Table 6. Estimated error rates, expressed as percentages, f or hypothetical 1051 performance standards that require a disinfectant t o pass all of multiple tests for 1052 both microbes as the hypothetical correlation betwe en the LRs of tests of the 1053 different microbes, the numbers of labs, and the nu mbers of tests for each microbe 1054 at each lab is varied. For the “current performanc e standard,” the allowable 1055 number of positive carriers ( N ) out of a 60 carrier set per test is 1 for P. 1056 aeruginosa and 1 for S. aureus; for the “proposed performance standard” the 1057 allowable number of positive carriers is 6 for P. aeruginosa and 3 for S. aureus

. 1058 “Not calculable” scenarios are denoted by “NC” and indicate high correlation 1059 values that are inconsistent with the observed vari ability for each organism 1060 separately. 1061 1062

Current Performance Standard Proposed

Performance Standard

Correlation between Microbes Num.

Labs Num. P.a. tests at ea. lab Num.

S.a. tests at ea. lab Passerror rate (α) Failerror rate (β) Passerror rate (α) Failerror rate (β)

0 1 1 1 8.9% 23.3% 16.7% 2.0%

0.25 1 1 1 11.9% 22.4% 20.4% 2.0%

0.5 1 1 1 15.2% 21.1% 24.5% 1.9%

0.95 1 1 1 23.9% 18.2% 34.8% 1.7%

0 1 2 2 1.5% 36.2% 4.2% 3.6%

0.25 1 2 2 3.1% 33.9% 7.1% 3.5%

0.5 1 2 2 5.1% 31.2% 10.5% 3.4%

0.95 1 2 2 NC NC NC NC

0 1 3 3 0.3% 45.1% 1.2% 5.0%

0.25 1 3 3 1.0% 41.6% 2.8% 4.9%

0.5 1 3 3 NC NC NC NC

0.95 1 3 3 NC NC NC NC

0 2 1 1 0.9% 41.0% 2.8% 4.0%

0.25 2 1 1 1.6% 39.5% 4.2% 3.9%

0.5 2 1 1 2.5% 37.6% 6.0% 3.8%

0.95 2 1 1 6.0% 33.0% 12.2% 3.3%

0 2 2 2 0.0% 58.8% 0.2% 7.1%

0.25 2 2 2 0.1% 55.9% 0.5% 6.9%

0.5 2 2 2 0.3% 52.4% 1.1% 6.6%

0.95 2 2 2 NC NC NC NC

0 3 1 1 0.1% 54.4% 0.5% 5.9%

0.25 3 1 1 0.2% 52.7% 0.9% 5.8%

0.5 3 1 1 0.4% 50.5% 1.5% 5.7%

0.95 3 1 1 1.6% 45.1% 4.3% 4.9% 1063 Page 27 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

28

Table 7. Assuming an among-lab variance for S. aureus of the same magnitude as 1064 the observed within-lab variance, these are the est imated error rates, expressed as 1065 percentages, for hypothetical performance standards that require a disinfectant to 1066 pass all

of multiple tests for both microbes as the hypothe tical correlation 1067 between the LRs of tests of the different microbes, the numbers of labs, and the 1068 numbers of tests for each microbe at each lab is va ried. For the “current 1069 performance standard,” the allowable number of posi tive carriers ( N ) out of a 60 1070 carrier set per test is 1 for P. aeruginosa and 1 for S. aureus; for the “proposed 1071 performance standard” the allowable number of posit ive carriers is 6 for P. 1072 aeruginosa and 3 for S. aureus . “Not calculable” scenarios are denoted by “NC” 1073 and indicate high correlation values that are incon sistent with the observed 1074 variability for each organism separately. 1075 1076

Current Performance Standard Proposed

Performance Standard

Correlation between Microbes Num.

Labs Num. P.a. tests at ea. lab Num.

S.a. tests at ea. lab Passerror rate (α) Failerror rate (β) Passerror rate (α) Failerror rate (β)

0 1 1 1 11.08% 29.44% 18.4% 4.4%

0.25 1 1 1 14.42% 27.80% 22.2% 4.3%

0.5 1 1 1 18.11% 25.75% 26.4% 4.0%

0.95 1 1 1 28.31% 19.53% 37.5% 3.0%

0 1 2 2 3.76% 43.56% 7.6% 7.8%

0.25 1 2 2 6.17% 40.51% 11.0% 7.5%

0.5 1 2 2 9.22% 36.87% 15.0% 6.9%

0.95 1 2 2 NC NC NC NC

0 1 3 3 1.83% 52.30% 4.1% 10.6%

0.25 1 3 3 3.57% 48.36% 6.9% 10.1%

0.5 1 3 3 6.01% 43.72% 10.4% 9.2%

0.95 1 3 3 NC NC NC NC

0 2 1 1 1.32% 49.92% 3.4% 8.5%

0.25 2 1 1 2.19% 47.62% 5.0% 8.3%

0.5 2 1 1 3.42% 44.66% 7.0% 7.9%

0.95 2 1 1 8.21% 35.12% 14.1% 5.9%

0 2 2 2 0.16% 67.72% 0.6% 14.9%

0.25 2 2 2 0.42% 64.25% 1.2% 14.4%

0.5 2 2 2 0.91% 59.84% 2.3% 13.4%

0.95 2 2 2 NC NC NC NC

0 3 1 1 0.17% 64.27% 0.6% 12.5%

0.25 3 1 1 0.35% 61.84% 1.1% 12.2%

0.5 3 1 1 0.67% 58.61% 1.9% 11.6%

0.95 3 1 1 2.43% 47.60% 5.3% 8.7% 1077 Page 28 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

29

Appendix A - Discussion and details for selected mo deling steps 1078 1079 Previous Models Used for Assessing the UDM and HSCT Data 1080 1081 In order to evaluate equations (1)-(3) at the extre me cases of N = 0 or N = 60, the 1082 proportion of negative carriers p N that we consider includes a 1⁄2 in the numerator and a 1 1083 in the denominator. Morgan (6, p 58), McCullagh & Nelder (35, p 117) and Collett (22, 1084 p 80) show that the addition of these terms is appr opriate. 1085 1086 Step 3: Calculating the error rates for a single UD M test 1087 1088 For the competing null and alternative hypotheses i n equation (6), the rejection region 1089 RR 1 is 1090 1091 RR 1 = {All N such that N ≤ PS 1 }, 1092 1093 where N

is the number of positive carriers observed in a s ingle UDM test of 60 carriers. 1094 Since the LR transform (via equation (4)) is a mono tonically decreasing function of N for 1095 a fixed TestLD , and because the symmetrically distributed LRs hav e median equal to the 1096 mean, then the hypotheses in equation (6) can be re written in a mathematically equivalent 1097 form as 1098 1099 H 0 : μ LR ≤ LR( PS 1 + 1) 1100 H a : μ LR > LR( PS 1 + 1). 1101 1102 In these new hypotheses, the true mean LR is denote d by μ LR . The rejection region with 1103 respect to the LRs is 1104 1105 RR 1 = {All LRs such that LR( N ) ≥ LR( PS 1 )} 1106 = {( LR( N ) – LR( PS 1 + 1))/ S R ≥ (LR( PS 1 ) – LR(

PS 1 + 1))/ S

R } 1107 = { T ≥ t 1 }. 1108 1109 The bold font indicates random variables throughout the seq uel. The pass-error

rate can 1110 now be calculated just as a Type I error is calcula ted in statistical hypothesis testing 1111 (Casella & Berger, section 8.3.1). The random vari able 1112 1113 T

= ( LR( N ) – LR( PS 1 + 1))/ S R 1114 1115 is only approximately a t ( L - 1) random variable, even when the LRs are normal , since 1116 ( L - 1) S

2 R /E( S 2 R ) is only approximately a χ 2 ( L - 1) random variable (17, p 971). Nijhuis 1117 & Van den Heuvel (36) showed that this approximatio n is particularly good and has a 1118 negligible effect on probability calculations. The critical value, t 1 , for the test statistic is 1119 completely determined by the performance standard a nd the observed reproducibility SD 1120 observed in the collaborative study, 1121 1122

t 1 = (LR( PS 1 ) – LR( PS 1 + 1))/ S R

. 1123 Page 29 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

30

Given a performance standard PS 1 , the LRs in the numerator can be calculated by 1124 equation (4). The standard deviation S R in the denominator is the reproducibility SD of 1125 the LRs across multiple labs as in equation (5), an d estimated from the UDM 1126 collaborative study (Table 1). The pass-error rate α 1 of a single test can now be 1127 calculated by application of equation (7), which re quires probability calculations based 1128

on a t -distribution with non-centrality parameter λ 1 = 0, and degrees of freedom 1129 1130 (19) @A  B  C )DE F G 6 C

HIF  )FI F

G 6 C HG)FI F G 6 C . 1131

1132 In equation (19), I = 5 is the number of labs in the UDM collaborative st udy; J = 3 is the 1133 number of repeated tests in each lab; and R = S 2 lab / S 2 test , where both S 2 lab and S

2 test are the 1134 variance components from the UDM collaborative stud y (see Table 1). Not to be 1135 confused with the well known result that the standa rd error of the mean LR across the I 1136 labs, [( S 2 test / J + S 2 lab )/ I ] 1/2 , has degrees of freedom equal to I - 1 (17), equation (19) gives 1137 the degrees of freedom associated with S R . Equation (19) can be derived directly using 1138 Satterthwaite’s formula applied to the independent mean squares and their degrees of 1139 freedom as in Mee (37). Equations (5) and (19) mak e clear precisely how the variance 1140 components S 2 lab and S

2 test from the UDM collaborative study are used to calcu late the 1141

pass-error

rate α 1

. 1142 1143 The expected fail-error rate β 1 of highly efficacious treatments is calculated just as a 1144 Type II error is in statistical hypothesis testing when the LRs are normal using equation 1145 (8), which uses a non-central t -distribution, with degrees of freedom df 1 calculated as in 1146 equation (19), and non-centrality parameter 1147 1148 (20) λ 1 = (LR(0) – LR( PS 1 + 1))/ S

R

. 1149 1150 In the numerator in equation (20), we utilize the a ssumption that the median number of 1151 positive carriers is zero for a highly effective tr eatment. The preceding equation, as well 1152 as equations (5) and (19), make clear precisely how the variance components S 2 lab and 1153 S

2 test from the UDM collaborative study are used to calcu late the fail-error rate β 1

. 1154 1155 Alternate Step 4: Calculating the error rates for m ultiple UDM tests for a single 1156 organism using averaging 1157 1158 The hypotheses being tested in this scenario are gi ven in equation (14). If the geometric 1159 mean MPN across the multiple tests attains the perf ormance standard, then the 1160 disinfectant passes. That is, the rejection region RR mean for this test is 1161 1162 RR mean = {(geometric mean MPN) ≤ PS mean }. 1163 1164 We consider the geometric mean MPN across K tests at each of L labs instead of the 1165 arithmetic mean for three reasons. First, the geom etric mean MPN can be converted to 1166 the typical number of positive carriers over many U DM tests; secondly, the geometric 1167 Page 30 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

31

mean is directly transformed to the mean LR for eac h lab using equation (4) when the 1168 LRs are symmetrically distributed; and thirdly, exp erience shows that the LRs are 1169 normally distributed. The hypotheses in equation ( 14) can be written in a mathematically 1170 equivalent form with respect to the mean LR, 1171 1172 H 0 : μ LR ≤ LR( PS mean + 1) 1173 H a : μ LR > LR( PS mean + 1). 1174 1175 In these hypotheses, the true mean LR is denoted by μ LR . The corresponding rejection 1176 region with respect to the arithmetic mean of the L Rs across K tests at each of L labs is 1177 1178 RR mean = {(arithmetic mean LR) ≥ LR( PS mean )}. 1179 1180 The critical value of the test statistic correspond ing to this rejection region is 1181 1182

t mean = (LR(

PS mean ) – LR( PS

mean + 1))/ SE

mean . 1183 1184 Since multiple tests are available, this test stati stic differs in a single fundamental respect 1185 from the critical value t 1 for the single test performance standard presented earlier. The 1186 denominator is now occupied by SE mean , the standard error of the mean LR over K future 1187 tests performed at each of L future labs. To use all available data, SE mean is found by 1188 pooling the standard error obtained from the UDM co llaborative study with the standard 1189 error to be observed in the K future tests performed at each of L future labs. Each of the 1190 variables necessary to evaluate SE mean are defined next. The “future” standard error to be 1191 observed in the multiple UDM tests of the disinfect ant will be calculated by 1192 1193 SE future = [ S

2 future, lab / L + S 2 future, test /( LK )] 1/2

, 1194 1195 which has df future = L - 1 degrees of freedom (17, pp 958-976), with the variance 1196 components S 2 future, lab and S

2 future, test calculated from the future UDM test requiring K tests 1197 performed at each of L future labs. The standard error estimated from th e UDM 1198 collaborative study is 1199 1200 SE collab = [ S

2 lab / L + S 2 test /( LK )] 1/2

. 1201 1202 The variance components S 2 lab and S

2 test in this equation are the same as those calculated 1203 from the UDM collaborative study (see Table 1). In other words, SE collab provides our 1204 best guess for SE future , the standard error of the mean LR in a future stu dy of K tests in 1205 each of L labs. The degrees of freedom associated with SE collab is given by 1206 1207

(21) @A JKLLMN  )B F

O 6 C

)DE F G 6 C

HIF  )FI O

G 6 C

HGO C )FI F

G 6 C  )B F

O 6 C

)DE F P 6 C

Q  )FI O

P 6 C FR) C P O6 C . 1208

1209 In equation (21), I = 5 is the number of labs in the UDM collaborative st udy; J = 3 is the 1210 number of repeated tests in each lab in the UDM col laborative study; R = S 2 lab / S 2 test , 1211 Page 31 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

32

where both S

2 lab and S

2 test are the variance components from the UDM collabora tive 1212 study; and K

is the number of tests in each lab in a future tes t of the disinfectant (e.g., by 1213 enforcement officials) of the disinfectant. Equation (21), found via Satterthwaite’s 1214 approximation, is a modification of Mee’s (37) form ula (cf. equation (19)). The resulting 1215 pooled standard error of the mean is 1216 1217

(22) SE mean = S TU VWXX<Y Z[ C VWXX<Y :
Z[ C \]^]_`

TU VWXX<Y 	: . 1218

1219 The degrees of freedom associated with SE mean is 1220 1221 (23) df mean = df

collab + L - 1. 1222 1223 It is assumed that the K future UDM tests in each of L labs will yield the same variability 1224 as that observed in the UDM collaborative, S 2 future, lab = S

2 lab and S

2 future, test = S

2 test . In this 1225 case, equation (22) reduces to 1226 1227 SE mean = SE

collab

, 1228 1229 with degrees of freedom calculated as in equations (21) and (23). 1230 1231 Now the expected pass-error and fail-error

rate when averaging over all KL tests can be 1232 formulated as in equations (15) and (16) using a no n-central t -distributions, with degrees 1233 of freedom df mean calculated as in equations (21) and (23). The non- centrality parameter 1234 in equation (16) is 1235 1236 λ mean = (LR(

N= 0) – LR( N=PS mean + 1)/ SE

mean ). 1237 1238 1239 Appendix B - Computer code 1240 1241 The error rate calculations presented in this manus cript were generated using the 1242 statistical software package R (38), package mvtnorm (26, 39). The R code is presented 1243 in the same order that the modeling steps were pres ented earlier, by first showing how to 1244 calculate the error rates for a performance standar d on a single UDM test, then for a 1245 performance standard which requires passing all of multiple tests, and then based on 1246 passing multiple tests on average. The following R code should be sequentially entered 1247 directly into R’s command line. 1248 1249 The following R function generates the LR associate d with a number of positive carriers 1250 as in equation (4), 1251 1252 GetMPNLR<-function(N,TestLD=6.5,NumTot=60) 1253 { 1254 LR = TestLD - log10(-log((NumTot-N+.5)/(NumTot + 1) )) 1255 return(LR) 1256 } 1257 Page 32 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

33

1258 For a single UDM test, the pass-error rate α 1 for the current performance standard is 1259 calculated for each microbe using equation (7) by: 1260 1261 PS1 = 1 # This corresponds to the curre nt UDM performance standard 1262

SR = 0.5348 # For P. aeruginosa , use SR = 0.3162 for S. aureus 1263 df1_Pa = 6.9 # For P. aeruginosa 1264 df1_Sa = 13.8 # , For S. aureus 1265 LR.PS1 = GetMPNLR(PS1) 1266 LR.PS1plus1 = GetMPNLR(PS1+1) 1267 t1_Pa = (LR.PS1 - LR.PS1plus1)/SR 1268 alpha1_Pa = 1-pt(t1_Pa,df1_Pa) 1269 t1_Sa = (LR.PS1 - LR.PS1plus1)/0.3162 1270 alpha1_Sa = 1-pt(t1_Sa,df1_Sa) 1271 1272 1273 For a single UDM test, the fail-error rate β 1 is calculated using equations (8) and (20) by: 1274 1275 LR.0 = GetMPNLR(0) 1276 lambda1_Pa = (LR.0-LR.PS1plus1)/SR 1277 beta1_Pa = pt(t1_Pa,df1_Pa,ncp=lambda1_Pa) # for P. aeruginosa 1278 1279 lambda1_Sa = (LR.0-LR.PS1plus1)/0.3162 1280 beta1_Sa = pt(t1_Sa,df1_Sa,ncp=lambda1_Sa) # for S. aureus 1281 1282 When a performance standard requires that a disinfe ctant passes all of KL uncorrelated 1283 UDM tests (this means that each test is performed i n a different lab in the case of P. 1284 aeruginosa ), then the pass-error rate α KL and the fail-error

rate β KL are calculated using a 1285 multivariate t

via equations (9) and (11) by: 1286 1287 library(mvtnorm) 1288

K = 1 # Number of P. aeruginosa tests at each lab 1289

L = 3 # Number of labs 1290

TotTests = K*L # Total number of tests 1291 alphaKL = pmvt(lower=rep(t1_Pa, TotTests ),upper=r ep(Inf, TotTests),df=df1_Pa) 1292 betaKL = 1-pmvt(lower=rep(t1_Pa, TotTests ),upper= rep(Inf, TotTests),df=df1_Pa,delta=lambda1_Pa) 1293 1294 The error rate calculations are more complicated wh en multiple tests of a single microbe 1295 are performed in the same lab in the presence of a significant among-lab variance 1296 component S 2 lab , in which case the correlation matrix amongst the tests must be inputted 1297 in order to evaluate equations (9) and (11). For e xample, the two error rates for two P. 1298 aeruginosa tests performed in each of two labs are calculated by: 1299 1300 ##### An R function to generate the correlation mat rix for tests of a single microbe 1301 GenMatrix<-function(NumTests,NumLabs=1,VarLab=.175,V arExp=.111) 1302 { 1303 Z = matrix(0,NumTests,(NumTests+1)) 1304 Z[,1]=1 1305 for (i in 1:NumTests) 1306 { 1307 Z[i,i+1]=1 1308 Page 33 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

34 } 1309 Psi=diag(c(rep(VarLab,1),rep(VarExp,NumTests))) 1310 Vblk=Z%*%Psi%*%t(Z) 1311 N = NumTests*NumLabs # total number of data point s across all labs and exps 1312 V=matrix(0,N,N) 1313 for (i in 0:(NumLabs-1)) 1314 { 1315 index=(i*NumTests+1):((i+1)*NumTests) 1316 V[index,index]=Vblk # construct the covariance matr ix first 1317 } 1318 V = V/V[1,1] 1319 return(V) 1320 } 1321

##### End of function 1322 1323

L = 2 # Number of labs 1324

K = 2 # Number of tests in each of the labs 1325

TotTests = K*L 1326 R_Pa = GenMatrix(K,L) 1327 alphaKL = pmvt(lower=rep(t1_Pa, TotTests),upper=re p(Inf, TotTests ),df=df1_Pa,corr=R_Pa) 1328 betaKL = 1-pmvt(lower=rep(t1_Pa, TotTests),upper=re p(Inf,TotTests),df=df1_Pa,corr=R_Pa,delta=lambda1_P a) 1329 1330 When a performance standard requires that all tests from both P. aeruginosa and S. 1331

aureus must be passed, the error rate calculations via eq uations (17) and (18) look similar 1332 as for the single microbe case, but the correlation matrix is more complicated. For 1333 example, the two error rates for two tests of each microbe performed in each of two labs 1334 are calculated by: 1335 1336 ##### An R function to generate the correlation ma trix for tests of two microbes 1337 GenMatrix_Microbe<- 1338 function(NumTests,NumLabs,CorrMicrobe,VarLab=c(.175, 0),VarExp=c(.111,.1)) 1339 { 1340 # NumTests is a 2x1 vector, NumTests[1] specifies n umber of tests for Pa in each lab 1341 # NumTests[2] sepcifies n umber of tests for Sa in each lab 1342

# 1343 # VarLab is a 2x1 vector, VarLab[1] is for Pa, VarLab[ 2] is for Sa 1344 # VarExp is a 2x1 vector, VarExp[1] is for Pa, VarExp[ 2] is for Sa 1345 PaBlock=GenMatrix(NumTests[1],1,VarLab[1],VarExp[1]) 1346 SaBlock=GenMatrix(NumTests[2],1,VarLab[2],VarExp[2]) 1347 CovMicrobe = CorrMicrobe*sqrt((VarLab[1] + VarExp[1]) *(VarLab[2] + VarExp[2])) 1348 Vblk = matrix(0,sum(NumTests),sum(NumTests)) 1349 index1=1:NumTests[1] 1350 index2=(NumTests[1]+1):sum(NumTests) 1351 Vblk[index1,index1]=PaBlock 1352 Vblk[index2,index2]=SaBlock 1353 Vblk[index1,index2]=CorrMicrobe 1354 Vblk[index2,index1]=CorrMicrobe 1355 N = sum(NumTests)*NumLabs # total number of data points across all labs and exps 1356 V=matrix(0,N,N) 1357 for (i in 0:(NumLabs-1)) 1358 { 1359 index=(i*sum(NumTests)+1):((i+1)*sum(NumTests)) 1360 Page 34 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

# For Peer Review

35 V[index,index]=Vblk 1361 } 1362 return(V) 1363 } 1364

##### End of function 1365 1366

L = 2 # Number of labs 1367 K_Pa = 2 # Number of P. aeruginosa tests in each of the labs 1368 K_Sa = 2 # Number of S. aureus tests in each of the labs 1369 R = GenMatrix_Microbe(c(K_Pa,K_Sa),L,0.25) 1370 vecblk = c(rep(t1_Pa,K_Pa),rep(t1_Sa,K_Sa)) 1371 lower.vec = rep(vecblk,L) 1372 deltablk = c(rep(lambda1_Pa,K_Pa),rep(lambda1_Sa,K_ Sa)) 1373 delta.vec = rep(deltablk,L) 1374 TotTests = (K_Pa + K_Sa)*L 1375 alphaKL = pmvt(lower=lower.vec,upper=rep(Inf,TotTes ts),df=df1_Pa,corr=R) 1376 betaKL = 1-pmvt(lower=lower.vec,upper=rep(Inf,TotTe sts),df=df1_Pa,corr=R,delta=delta.vec) 1377 1378 The error rates for a performance standard on the a verage over K multiple tests performed 1379 at each of L labs are calculated using equations (15),(16),(21) and (22): 1380 1381 PSmean = 1 # This corresponds to the cu rrent UDM performance standard 1382 L = 3 # Number of labs required by the perfo rmance standard 1383

K = 10 # Number of future tests in each lab 1384 S2_lab = 0.175 # For P. aeruginosa , use S2_lab = 0.000 for S. aureus 1385 S2_test = 0.111 # For P. aeruginosa , use S2_test = 0.100 for S. aureus 1386

dfcollab = 3.06 1387 SEmean = sqrt(S2_lab /L + S2_test /(L*K)) 1388

dfmean = dfcollab + L - 1 1389 LR.PSmean = GetMPNLR(PSmean) 1390 LR.PSmeanplus1 = GetMPNLR(PSmean+1) 1391 tmean = (LR.PSmean - LR.PSmeanplus1)/SEmean 1392 alpha_mean = 1-pt(tmean,dfmean) 1393 lambda_mean = (LR.0-LR.PSmeanplus1)/SEmean 1394 beta_mean = pt(tmean,dfmean,ncp=lambda_mean) 1395 1396 Page 35 of 35

Not For Redistribution AOAC OMA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60