1



### Reassessment of the Performance Standard for the AOAC Use

- dilution 1



### Method s ( 955.15 and 964.02 ) Revision Date: October 1 9 , 2012 2

3

Al bert E. Parker 4

C enter for Biofilm Engineering, Montana Stat e University, Bozeman, MT 59715 5

Martin A. Hamilton 6

Big Sky Statistical Analysts LLC, Bozeman, MT 59715 7

Stephen F. Tomasino 8

U.S. Environmental Protection Agency

– Office of Pesticide Programs, Biological and 9

Economic Analysis Division, Microbiology Laboratory Branch , Environmental Science 10

Center, F or

t Meade, MD 207 55

- 5350 11

12

13

The AOAC Use

- dilution methods (UDM) , 955.14 ( Salmonella enterica ), 955.15 14

( Staphylococcus aureus ) and 964.02 ( Pseudomonas aeruginosa ), are laboratory 15

methods used to substantiate antimicrobial efficacy claims for liquid disinfectants 16

on inanimat e surfaces. The UDM is accepted by the United States Environmental 17

Protection Agency (EPA) for the purpose of product registration

. Testing of S. 18

aureus and P. aeruginosa is required to support a hospital disinfectant claim. To 19

meet the current performance standard for the UDM

, a product must exhibit no 20

more than 1 carrier exhibiting growth (“positive”) out of 60 carriers tested. 21

Employing statistical modeling techniques, the method’s performance standard has 22

been reassessed b ased on data collected from a 5 laboratory collaborative study 23

conducted in 2009

. Of concern are the pass

- error ( a Type I error) rate for ineffective 24

products and the fail

- error ( a T ype II error) rate for effective products

. A 25

performance standard should keep both of these error rates as low as possible

. 26

Based on the model’s output, new performance standard s for two UDM methods, 27

955.15 ( S. aureus ) and 964.02 ( P. aeruginosa ) are considered

. T he performance 28

standard s considered would require three tests (e. g., testing of three representative 29

production lots per microbe or testing one representative production lot three times 30

per microbe ) to be performed against P . aeruginosa and S. aureus ; the disinfectant 31

would need to pass all tests for both microorganism s

. To be considered a pass , no 32

more than 6 positive carriers for a P. aeruginosa test and no more than 3 positive 33

carriers for a S. aureus test would be allowed

. When calculated using the 34

reproducibility standard deviations observed in the collaborative study , the pass

35

error rate is 1

- 3% for ineffective products and 5% fail

- error rate for highly effective 36

products. The EPA is seeking AOAC’s technical review of the mode l and the 37

associated calculations. P roposed revisions to Methods 955.15 and 964.02 are 38

provided

. This paper does not address the performance standard for S

. enterica as 39

no change is currently being explored. 402

Background on the Use

- dilution Method 41

42

The AOAC Use

- dilution methods, 955.14 ( Salmonella enterica ) ( 1 )

, 955.15 43

( Staphylococcus aureus ) ( 2 ) , and 964.02 ( Pseudomonas aeruginosa ) ( 3 ) , have been used 44

to assess the efficacy of liquid disinfectants for over four decades. With the exception of 45

the test mi crobes and subtle differences in generating the test cultures, the three Use

46

dilution methods (UDM) are essentially the same. T he United States Environmental 47

Protection Agency (EPA) accepts the UDM to support the registration of liquid 48

antimicrobial products bearing claims to control microorganisms that pose a threat to 49

human health on inanimate surfaces

– t he se are known as public health products

. 50

Specifically, p ublic health disinfectants such as those for use in the healthcare 51

environment (i.e., hos pital disinfectants ) must be deemed effective against S

. aureus and 52

P

. aeruginosa

. T esting of S. enterica is used to support limited and broad spectrum 53

disinfectants , but is not currently used to support hospital disinfectant claims. 54

55

T he UDM is a semi

- qua ntitative test method that requires the use of stainless steel 56

cylinder carriers t o simulate an environmental surface. Carriers are submerge d in a test 57

culture and d ried. The inoculated carriers are divided into two groups ; treated (60 carriers 58

per test) and untreated (6 control carriers per test). The treated carriers are placed into 10 59

mL of disinfectant and allowed to stand for a defined contact time and temperature

. 60

Following the prescribed contact time, e ach carrier is r emoved aseptically and placed in a 61

tube containing a liquid subculture medi um for neutralization and recovery of viable 62

bacteria and incubated

. Product efficacy is determined by the presence (growth) or 63

absence (no growth) of the test microbe on each of th e 60 treated carriers

. The untreated 64

(control) carriers are not exposed to the disinfectant product and are used to a ss ess 65

microbial loading, otherwise known as the carrier counts. The carrier counts are 66

expressed as the mean number of colony forming units (CFU) per carrier. Currently, the 67

UDM allows no more than 1 positive carrier out of 60 ( for each microbe ) for a product to 68

be deemed efficacious. N umerous editorial revisions have been made to the UDM to 69

improve the clarity and specificity of the procedur e

, and minimum and maximum carrier 70

count levels have been specified to qualify the test results ( 4 ). For a complete description 71

of the UDM methodology, refer to the official AOAC International (AOAC) methods ( 1

- 72

3 ). 73

74

In practice, the results from one or mor e UDM tests are classified as a “ pass or fail ” 75

outcome where a pass means the product exhibited satisfactory disinfection efficacy and 76

a fail means it did not. The current testing required by EPA for the registration of liquid 77

disinfectants specifies the evaluation of three representative production lots (or batches) 78

of the product tested once against each of th e required microbes

. One of the three lots 79

must be at least 60 days old. For product registration, a ll tests may be conducted by one 80

laboratory. Th e disinfectant ’s efficacy is deemed acceptable only if each of the tests 81

produces a pass ing outcome (i.e., no more than 1 positive carrier out of 60 tested). Thus, 82

if 2 or more carriers are positive per test , the outcome is deemed a fail. 83

84

T he statistical justification of the current UDM performance standard was published in 85

1961 by Ortenzio and Stuart ( 5 ). They viewed the UDM as a procedure to confirm the 863

efficacy of diluted liquid germicides based on the “maximum effective dilution” that can 87

be recommended for disinfecting floors, walls, fixtures, and equipment. They proposed 88

the current performance standard for the UDM which allows no more than 1 positive 89

carrier o ut of a 60 carrier set, and claimed that this performance standard allowed only 90

5 % of ineffective treatments to pass when challenged against S. aureus or S. enterica

. 91

Their work relies on a probit analysis ( 6 ) of unpublished data from replicate product 92

applications and a wide range of use

- dilutions. Furthermore, the statistical justifi cation 93

for the current performance standard did not account for variability among laboratories ; 94

assessing the variability among laboratories is considered a critical aspect of method 95

performance evaluations. Although liquid disinfectants have become more d iverse over 96

the years, and the method has been the subject of many studies and many revisions, the 97

performance standard has not been reassessed and remain s unchanged. 98

99

Two significant multi

- laboratory studies by Cole ( 7 ) and Rutala ( 8 ) demonstrated that 100

UD M results may vary considerably among laboratories and they identified major aspects 101

of the method that were likely sources of the variability, most notably irregular surfaces 102

of the stainless steel carriers, inconsistent inoculum levels, and inadequate re moval of the 103

pellicle associated with P. aeruginosa test cultures. Unfortunately, control carrier counts 104

were not evaluated in the studies. An initiative designed to resolve the sources of 105

variability led to the development and validation of AOAC method 99 1.47 , the Hard 106

Surface Carrier Test ( 9 ). The Hard Surface Carrier Test (HSCT) is very similar to the 107

UDM and specifies the following: 1) standardization of test cultures used to inoculate 108

carriers, 2) assessment of control carrier counts and associated minimum and maximum 109

limits, 3) use of glass penicylinders as carriers; and 4) use of an agar

- based procedure for 110

growing the test microbes. In 1992, Rubino et al. published a statistical analysis of a 10

- 111

lab, 6

- disinfectant collaborative study of the HSCT ( 10 ). In the paper, Rubino et al. 112

proposed single test HSCT performance standards, which the authors claimed allowed at 113

least 5%, but less than 50%, of ineffective treatments to pass. To receive a pass under the 114

performance standard specified for the HSCT , the test must result in no more than 2 115

positive carriers out of 60 for a S. aureus or a S. enterica test, and no more than 3 positive 116

carriers out of 60 for a P. aeruginosa test. The HSCT is accepted by EPA to support the 117

registration of liquid disinfect ants; however, the HSCT is rarely used by the regulated 118

community purportedly due to the lack of hard water and organic soil load in the 119

validation study. 120

121

Critical to th e current initiative, a multi

- laboratory collaborative study was conducted i n 122

2009 by the EPA and an industry

- based technical work group to assess a set of proposed 123

modifications to the UDM ( 11) ; the effort was not designed to specifically address the 124

subject of this paper (i.e., reassessment of the UDM performance standard). The study 125

com

pared the current UDM to a modified UDM in a side

- by

- side fashion across five 126

laboratories to evaluate if the modifications improve d the method’s performance. T he 127

modifications were largely based on the culture preparation techniques specified for the 128

HSCT

. In the study, a single lot of an EPA registered quaternary ammonium liquid 129

disinfection product was tested at two levels of presumed efficac y using both S. aureus 130

and P. aeruginosa methods

. EPA’s Standard Operating Procedure (SOP) for conducting 131

the UDM, SOP MB

- 05 (1 2 ), based on a strict interpretation of the published UDM, was 1324

followed by the p articipating labs. The collaborative study results indicate that the 133

outcomes of the current UDM for S. aureus are very consistent within and among labs, 134

but vary significantly for P. aeruginosa among laboratories . T he variability exhibited by 135

the current UDM was no larger than that observed for other accepted disinfectant test 136

methods ( 13 ). In fact, the UDM exhibit ed quite good repeatability within the same lab 137

an d reproducibility amongst multiple labs. 138

139

This paper utilizes data from the UDM collaborative to perform a statistical assessment 140

of the existing UDM performance standard and selected alternative performance 141

standards

. The statistical evaluation focuses on the pass

- error and fail

- error rates 142

associated with a performance standard (14) . These errors , which are defined below, 143

occur when the UDM test results for a disinfectant product lead to the wrong conclusion. 144

The erro r rates were calculated by utilizing a conventional hypothesis testing framework 145

as described in the Modeling Steps section. 146

147

A pass

- error occurs when the disinfectant being tested truly has unacceptably low 148

efficacy, but is deemed a pass; this is the same type of error considered by Ortenzio & 149

Stuart ( 5 ) and Rubino et al. (10) . A fail

- error occurs when the disinfectant truly is of 150

acceptable efficacy, but is deemed a fail; this is the type of error emphasized in Arlea et 151

al. ( 15 )

. The calculation of thes e error rates is based on the method’s variability. In this 152

paper, we show how to calculate the error rate s for the performance standard currently in 153

use for regulatory purposes

, and show how to devise a new performance standard that 154

adequately maintains small er, acceptable error rates. The evaluations only pertain to tests 155

involving S. aureus and P. aeruginosa tests, not to S. enterica

. These error rates , easily 156

calculated using the computer code provided in this paper

, offer a quanti tative decision

157

making tool for r evising the performance standard for the UDM. T he EPA is interested in 158

utilizing th e UDM collaborative data and s tatistical modeling described in this report for 159

the purpose of improving the reliability of the UDM

. The EPA seeks a technical review 160

of the statistical model and associated calculations. Based on this modeling exercise, 161

proposed revisions to Methods 955.15 and 964 .02 are provided. 162

163

A ssessing the UDM Performance Standard 164

165

Terminology and Process 166

167

A performance standard for a method defines requirements for an acceptable outcome 168

(i.e., acceptable product performance) for a product being tested by the method. The 169

acceptable outcome can be specified per test, with the requirement that the product passes 170

all of multiple tests

, o r the acceptable outcome can be specified as an average over 171

multiple tests

. T he performance standard may also stipulate the maximum number of 172

positive carriers or a minimum log reduction value allowed per test, the number of tests 173

that must be per formed , and the number of labs required to conduct the tests

. For 174

example, f

or the UDM, the current performance standard specifies that a pass ing outcome 175

is no more than 1 positive carrier out of 60 tested per test

. F or product registration, the 176

EPA requi

res the evaluation of three separate batches of product per microbe, where one 177

batch must be 60 days old , and testing may be performed at a single laboratory. 1785

179

For the purpose of this statistical evaluation, the reassessment process assumes that we 180

are reviewing a proposed UDM performance standard which specifies PS

, the allowable 181

number of positive carriers per test , the number of tests that must be passed , and the 182

number of labs needed to conduct the tests . An alternative performance standa rd, based 183

on the typical number of positive carriers over the multiple tests , was also considered

. 184

185

Various outcomes relevant to the effectiveness of disinfectant products are utilized in the 186

modeling process, each is defined below. For the sole purpose of this paper and 187

explaining the calculations , a n ineffective disinfectant product is defined as one which 188

has a true median number of positive carriers which is at least PS + 1 out of a 60 carrier 189

set. In other words, a n ineffective product fails the perfo rmance standard by a t least a 190

single positive carrier in at least 50% of all UDM tests

. Further, a marginally ineffective 191

disinfectant product is one which has a true median number of positive carriers equal to 192

PS + 1 out of a 60 carrier set. For the current performance standard which allows PS = 1 193

positive carriers in each UDM test , a marginally ineffective product has a true median 194

number of posit ive carriers equal to 2

. In contrast, an effective disinfectant product is 195

defined as one which has a true median number of positive carriers less than PS + 1 out 196

of a 60 carrier set

. A highly effective disinfectant product is one which has a true median 197

number of positive carriers equal to 0. In other words, a highly effective product will 198

pass a t least 50% of all UDM tests with 0 positive carriers observed

. 199

200

A proposed UDM performance standard can be evaluated by determining whether it 201

attains acceptably small pass

- error and fail

- error rates

. A pass

- error occurs when an 202

ineffective product passes the given performanc e standard

. A pass

- error is also known as 203

a Type I error in statistical hypothesis testing ( 16

, section 8.3.1 )

. The pass

- error rate

, α, 204

also known as the significance level

, is the probability that a truly marginally ineffective 205

produ ct pass es the performance standard. Error rates for passing other ineffective 206

treatments could also be calculated, but the pass

- error rate for a marginally ineffective 207

product is largest compared to any other ineffective disinfectant. A fail

- error occurs 208

when a n effective product fails the given performance standard ( 15 ). A fail

- error is also 209

known as a Type II error in statistical hypothesis testing ( 16 ). The fail

- error rate

, β, is 210

the pro bability that a truly highly effective product fail s the performance standard

. The 211

error rates for failing other effective treatments could also be calculated, and these would 212

be larger than the fail

- error rate of a n effective disinfectant. 213

214

If an infinite number of UDM test s were applied to a disinfectant product at an infinite 215

number of labs, then the true median of all observed UDM outcomes for that product 216

could be directly calculated

. In this paper, the term truly is used to describe a 217

hypothe tical disinfectant product for which the true median number of positive carriers of 218

UDM outcomes is specified

. For example, in the previous paragraph, a truly highly 219

effective product is defined as a disinfectant product for which the true median of UDM 220

outcomes is equal to zero. 221

222

In this manuscript, we use a hypothesis testing framework to calculate error rates for 223

different performance standards (i.e., rejection regions)

. However, whereas the 2246

conventional hypothesis testing problem is to find a rejection region that achieves 225

spec ified error rates for one set of hypothes e s, our problem was to find the error rates for 226

many specified performance standards and sets of hypotheses. 227

228

Evaluating a performance standard using error rates 229

230

Setting acceptable bounds for pass

- error and fail

- error rate s is not a fixed aspect in 231

modeling, but rather is set by experts, policy makers, and/or scientific advisors. The error 232

rates 1%, 5% and 10% are commonly used for these error types in scientific hypothesis 233

testing. A level of 5 % is used as the baseline error rate for this decision

– making exercise. 234

A pass

- error rate of 5% means that in a long series of tests of a marginally ineffective 235

product at many labs , 95% of the tests will correctly fail the UDM performance standard

. 236

A fail

- error rate of 5% means that in a long series of tests of a highly effective product at 237

many labs, 95% of the tests will correctly pass the UDM performance standard. 238

Regulatory authorities charged with ensuring public safety focus on minimizing pass

239

errors in o rder to prevent accepting an ineffective product. Others may focus on fail

240

errors to avoid removing effective products from the marketplace. An optimal 241

p erformance standard addresses both types of errors effectively. Due to natural variability 242

among micro bial cultures, product formulations, microbiological techniques, etc., it is 243

impossible to eliminate both of these errors; however, the error rates can be reduced to 244

acceptable levels, for example, by conducting multiple UDM tests. 245

246

Previous Models Used for Assessing the UDM and HSCT D ata 247

248

The current performance standards for the UDM and HSCT were set by explicitly 249

considering the pass

- error rate defined above , but not the fail

- error rate. The current 250

UDM performance standard

, set by Ortenzio and Stuart ( 5 )

, was purported to provide “a 251

result at the 95% confidence level ”

. In other words, allowing no more than 1 positive out 252

of 60 treated carriers allow s a 5% pass

- error rate. However, t his claim i s based on 253

unpublished data

. For the HSCT, Rubino et al. ( 10) advocate d that “a performance 254

standard should be no smaller than the expected number of positive carriers and no larger 255

than the upper confidence limit ” . At 95% confidence, such a scheme would allow a pass

256

error rate of between 5% and 50%. The empiri cally observed pass

- error and fail

- error 257

rates for this performance standard can be calculated for data generated in the 10

- lab 258

c ollaborative HSCT study involving 6 test chemicals. The observed proportion of times 259

that a n ineffective disinfectant was passe d was α 1 = 0/120 = 0 .00 ( 10, Table 1, 260

formulations 2, 4 and 5, pooled across all 3 organisms). The observed proportion of times 261

that a n effective disinfectant was failed was β 1 = 12/120 = 0.10 ( 10, Table 1, formulations 262

1, 3 and 6, pooled across all 3 orga nisms). In 2008, based on fail

- error rate calculations, 263

Arlea et al. ( 15 ) advocated modifications to the UDM . Their study suggested that a fail

264

error rate of 10% was acceptable ( 15, Table 1 )

. 265

266

One common feature of the different statistical approaches that have been historically 267

applied to UDM data is that they first transform the number of positive carriers to a 268

decimal value , although the units associated with this new value are not always 269

interpretable with respect to the UDM test

. The first step of the transformation is to 2707

rewrite the number of positive carriers N as the proportion of negative carriers in a 60 271

carrier UDM test, 272

273

274

275

(see Appendix A). Ortenzio & Stuart ( 5 ) performed a probit analysis which appli e s the 276

inverse of the standard normal cu mulative distribution function

, denoted by to the 277

proportion of negative carriers by the following equation, 278

279

(1)

. 280

281

A general linear model (which includes any fixed or random effects of interest) can be 282

fit ted to these probit

- transformed values by m odern statistical software packages via 283

application of iterative ly re weighted least squares ( 17, p 572 )

. Another approach to 284

analyzing the numbers of positive carriers from UDM tests is to fit a logistic regression 285

model

, as Arlea et al. ( 15 ) did in their assessment of the UDM . Th is model requires a 286

logit transform of the number of positive carriers , defined by 287

288

(2)

. 289

290

As with the probit analysis, t he se logit values are analyzed using iterative ly re weighted 291

least squares ( 17, section 14.4 )

. Alternatively, Rubino et al. ( 10 ) analyzed the 292

proportions of negative carriers from a multi

- lab study of the HSCT directly by 293

employing a type of weighted ANOVA that is not advocated by modern statistical 294

thinking, and they performed calculations by hand because computer software was not 295

readily available in 1992 to perform the calculations

. In 1995, Rubino and collab orators 296

( 18) re

- analyzed the HSCT collaborative study data, this time using a n approach similar 297

to the most probable number (MPN) transform of the number of positive carriers N

, 298

299

(3)

. 300

301

The MPN for mula in equation (3) provides a maximum likelihood estimate of the typical 302

number of viable organisms per carrier, averaged across the 60 carrier set ( 19

- 21 )

. In the 303

bioassay literature, the ln(MPN) is known as the complementary log

- log transform ( 22

, 304

p p 55, 113). 305

306

One of the most significant changes to the UDM over the years has been the addition of a 307

process for measuring the number of viable bacteria on dried inoculated carriers, thus 308

providing for a semi

- quantitative test method. In this case, the typical bio

- ch allenge on 309

each of the i noculated carriers can be estimated by 310

311

, 312

3138

where the average is taken over the set of all enumerated untreated control carriers in the 314

test. Similar to how Hamilton et al. (18 ) re

- analyzed the HSCT collaborative study, t he 315

TestLD and the MPN formula in equation (3) can be subtracted to estimate a log 316

reduction (LR) value for each test by 317

318

(4 ) 319

320

( 20

, 23 ). More recently, i n the analysis of the UDM collaborative study (11)

, the MPN 321

transform was used to calculate LRs for each UDM t est using equation (4). Calculating 322

LR s using the MPN formulation has been used to analyze an other AOAC st andard 323

disinfectant test method against spores ( 24 )

. 324

325

Each of these approaches used to analyze the UDM or the HSCT transform the numbers 326

of positive carriers to a continuous scale before performing a statistical analysis. If one 327

were to plot the probit

, logit and

- log 10 (MPN) transforms (see equations (1)

- (3)) as a 328

function of the number of positive carriers N , one would observe that all three produce 329

similar decreasing “sideways S” curves ( e.g., see the comparative plots in Collett ( 22

, p 330

54 ) )

. We prefer to use the MPN transform instead of the other kinds of transforms that 331

could be applied to modern

- day UDM test data . The main reason for this choice is that 332

the availability of control carrier counts enabl e a LR to be calculated f or each test 333

providing results comparable to quantitative disinfectant tests

. Even with control counts 334

available, it is not clear how to calculate a LR from the probit and logit transforms in 335

equations (1)

- (2). 336

337

Model ing Steps

– Use of the UDM Collaborative Study Data 338

339

The primary inputs to the modeling process are: i ) the numbers of p ositive carriers 340

observed in the UDM collaborative study (11)

, in which there were 3 UDM tests 341

performed at each lab using a presumed “ high ” efficacy product; and ii ) the control 342

carrier counts for ea ch UDM test for both P. aeruginosa and S. aureus

. A schematic of 343

the complete modeling process is given in Figure 1. 344

345

Step 1: Calculating log reductions from the number of positive carriers 346

347

The first step of the modeling process wa s to MPN

- transform (via equation (3)) the 348

numbers of positive carriers from the UDM collaborative study to provide estimates of 349

the typical number of viable organisms left on each of the 60 treated carriers . S ince 350

control carriers were also enumerated for each test, then th e LR was calculated via 351

equation (4). 352

353

Step 2: Estimating the variability of the UDM 354

355

In the second step of the modeling process, the LR s calculated from the UDM 356

collaborative study were analyzed for each microbe separately using a one

- way ANOVA 357

with a random effect due to lab ( 11

, 25 ) . The ANOVA provides estimates of two 358

components of varia nce for the LR s for each microbe separately. One estimated 3599

component of variance is due to sources among labs (e.g. different equipment, 360

technicians and geographic lo cations) , denoted by S 2

lab . The second estimated variance 361

component is due to within

- lab sources (e.g. fresh inocula and different environmental 362

conditions on the days on which experiments are run) , denoted by S 2

test

. The values for 363

the variance components S 2

lab and S 2

test found for P. aeruginosa and S. aureus in the 364

UDM collaborative study are provided in Table 1. T he estimated reproducibility standard 365

deviation (SD) is calculated from the variance components by 366

367

(5) S R = [ S 2

lab + S 2

test ] 1/2 368

369

( 25 ). Note that S

test is the repeatability SD of the LRs and is often denoted by S

r . The 370

variance components S 2

lab and S 2

test were used in all subsequent calculations of the 371

expected pass

- error and fail

- error rates . Thus, the variability observed in the UDM 372

collaborative study is assumed to be representative of any UDM test. 373

374

375

Step 3: Calculating the error rates for a single UDM test 376

377

T he third modeling step relies on a specific performance standard for a single UDM test ; 378

namely, a disinfectant passes if no m ore than PS 1 positive carriers out of the 60 carrier set 379

are observed. The subscript of “1” emphasizes that this is a performance standard for a 380

single UDM test rather than multiple UDM tests. For the current UDM performance 381

standard, PS 1 = 1. We assume that ineffective treatments will not meet this performance 382

standard most of the time. In other words, as previously defined, the true median number 383

of positive carriers for a n ineffective treatment is at least PS 1 + 1

. Thus, f or a given 384

perfor mance standard PS 1 , in order to calculate the expected pass

- error and fail

- error 385

rate s , the following competing null and alternative hypotheses were considered 386

387

(6) H 0 : (true median number of positive carriers) ≥ PS 1 + 1 388

H

a : (true median number of positive carriers) < PS 1 + 1

. 389

390

The null hypothesis H 0 presumes that the disinfectant being tested is not efficacious. The 391

goal of the product’s manufacturer is that t he outcome of the UDM test will provide 392

evidence in favor of H

a , that the disinfectant is truly efficacious. Consistent with the 393

current implementation of the UDM performance standard based on a single UDM test 394

outcome, t he null H 0 is rejected in favor of the alternative H

a if the observed number of 395

positive carriers i s less than or equal to PS 1

. 396

397

The statistic T which tests the hypotheses in equation (6) follows Student’s t

- distribution 398

with degrees of freedom df 1 , and non

- centrality parameter λ 1 = 0. (see Appendix A for 399

details) . The Type I pass

- error and Type II fail

- error rates for the t

- test can be found by 400

conventional statistical techniques, and are equal respectively to 401

402

(7) α 1 = Pr (Rejecting H 0 | H 0 : true median N = PS 1 + 1 ) 403

= Pr ( T ≥ t 1 ; df 1 , λ 1 = 0 ). 404

(8) β 1 = Pr (Failing to reject H 0 |H

a : true median N = 0) 40510 = Pr ( T < t 1 ; df

1 , λ 1 ) 406

407

( 16 ). The probability function Pr ( ) is stra ightforward to evaluate using statistical 408

software (see Appendix B). Explicit formulas for these are provided in Appendix A

, as 409

well as other details showing how th is hypothesis testing framework leads to the 410

calculation of α 1 and β 1

. The calculations are completely determined by the available 411

S 2

lab and S 2

test values and the difference between two known LR values. The same 412

degrees of freedom df 1 applies to both the Type I and Type II error rate calculations. Also 413

λ 1 = 0 for all Type I error rate calculations and therefore for all α 1 calculations. A non

414

zero numerical value of λ 1 had to be calculated only for Type II fail

- err or rates β 1 , but the 415

calculations were uncomplicated

. 416

417

Since the calculations of the error rates depend on differences in LRs, i t should be 418

stressed that th e error rates do not depend on the magni tude of the control counts ( cf. 419

equation (4)), or on the magnitude of the LR values. The error rates depend only on the 420

LR differences just described, and on the presumed variability of these LR values. 421

422

Step 4: Calculating the error rates for multiple UDM tests for a single organism us ing a 423

“pass all” approach 424

425

Currently, single UDM tests per organism are commonly used to verify efficacy claims in 426

EPA’s post

- registration stewardship program ; however, f or registration purposes

, K = 3 427

tests per organism are required and a product must all pass K UDM tests in each of L = 1 428

labs. In the context of testing the hypotheses in equation (6) simultaneously for all KL 429

tests , the pass

- error rate for a performance standard that requires such a “pass all” test 430

scheme is 431

432

α KL = Pr (Rejecting H 0 in al l KL tests | H 0 : true median N = PS 1 + 1 ) 433

= Pr (Passing all KL tests | H 0 : true median N = PS 1 + 1 ) 434

(9) = Pr ( T ≥ t 1 in all KL tests | H 0 : true median N = PS 1 + 1). 435

436

When the KL tests are performed independently of each other, then 437

438

(10) α KL = (α 1 ) KL

. 439

440

In the preceding equation, the pass

- error rate for a single test, α 1 , was provided in 441

equation (7). 442

443

The probability that a highly effective product fails to pass all KL tests is 444

445

β KL = Pr (Failing to reject H 0 in at least one test | H

a : true median N = 0) 446

= 1

- Pr (Rejecting H 0 in all KL tests | H

a : true median N = 0) 447

(11) = 1

- Pr ( T ≥ t 1 in all KL tests | H

a : true median N = 0). 448

449

When the KL tests are performed independently of each other, then this equation can be 450

simp lified to 45111

452

(12) β KL = 1

- (1

- β 1 ) KL

. 453

454

In the preceding equation, the fail

- error rate for a single test, β 1 , was provided in equation 455

(8). 456

457

In practice, it is known that UDM tests conducted in the same lab are not statistically 458

independent of each other s ince they may be performed, for example, by the same 459

technician using similar equipment in the same facility and geographic location. T he 460

estimated correlation amongst repeated tests in the same lab is 461

462

(13) r = S 2

lab / ( S 2

lab + S 2

test ) 463

464

( 17 , section 24.1) , which is readily calculated for each organism using the values in Table 465

1 . The pass

- error and fail

- error rates using equations (10) and (12) assume that the KL 466

tests are independent (and uncorrelated). If the correlation in equation (13) is 467

signif

icantly positive, as is the case when there is a significant lab

- to

- lab variance 468

component S 2

lab , then equations (10) and (12) provide incorrect error rates for the 469

performance standard. To properly account for these correlations among tests of the 470

same m icrobe in the same lab, we used a multivariate t

- distribution to calculate the pass

471

error rate α KL and the fail

- error rate β KL using equations (9) and (11) ( 26 )

. The 472

multivariate t

- distribution requires the same inputs t 1, df 1 , and λ 1 as were required for 473

equations (7) and (8). An additional input is the matrix of correlations among the KL 474

tests. S tatistical software for these calculations is readily available (see Appendix B). 475

476

Alternate Step 4: Calculating the error rates for multi ple UDM tests for a single 477

organism using averaging 478

479

A possible alternative to requiring that a disinfectant pass all of multiple UDM tests is to 480

impose a performance standard PS

mean on the mean disinfectant response across the K 481

multiple tests performed at each of L labs. Since it is assumed that ineffective treatments 482

will not meet this performance standard most of the time, then the median number of 483

positive carriers for a n ineffective treatment is at least PS

mean + 1 . Thus, th e hypotheses 484

being tested in this scenario are 485

486

(14) H 0 : (true median number of positive carriers) ≥ PS

mean + 1 487

H

a : (true median number of positive carriers) < PS

mean + 1

. 488

489

In contrast to the pass

- all approach, these hypotheses are tested by calculating the mean 490

response across all KL tests. The chance of a pass

- error when averaging over all KL tests 491

is 492

493

α mean = Pr (Rejecting H 0 | H 0 : true median N = PS

mean +1 ) 494

= Pr (Passing all KL tests on average | H 0 : true median N = PS

mean +1 ) 495

(15) = Pr ( T ≥ t

mean ; df

mean , λ

mean = 0 )

. 496

49712

The expected fail

- error rate for a highly effective product (for which the true median 498

number of positive carriers is 0) i s calculated by 499

500

β

mean = Pr (Failing to reject H 0 | H

a : true median N = 0) 501

(16) = Pr ( T < t

mean ; df

mean , λ

mean ). 502

503

Just as was described for the single test scenario, the above probability calculations 504

correspond to the Type I and Type II error rates for a t

- test of the hypotheses in (14). 505

These calculations depend on Student’s t

- distribution with critical value t

mean , degrees of 506

freedom df

mean , and a non

- centrality parameter λ

mean (see Appendix A). 507

508

Step 5: Calculating the error rates for multiple UDM tests for both P. aeruginosa and S. 509

aureus using a “pass all” approach 510

511

Step 4 of the modeling process considered error rates for each microbe separately ; 512

h owever, the current UDM performance standard, whether at registration or post

513

registration, requires that a disinfectant pass all te sts for all organisms under 514

consideration for a hospital disinfectant claim . To calculate the pass

- error and fail

- error 515

rates for this scenario, appropriate notation must first be developed

. C onsider a 516

performance standard that requires K Pa UDM tests against P. aeruginosa and K Sa UDM 517

tests against S. au r e u s at each of L labs, for a total of K = K Pa + K Sa tests at each of L labs. 518

The hypotheses in equation (6) must now be extended to account for different target 519

numbers of positive carriers for each microbe, 520

521

H 0 : (true median number of positive carriers for a P. aeruginosa test ) ≥ PS Pa + 1 522

or 523

(true median number of positive carriers for an S. au r e u s test ) ≥ PS Sa + 1 524

525

H

a : (true median number of positive carriers for a P. aeruginosa test ) < PS Pa + 1 526

and 527

(true median number of positive carriers for an S. au r e u s test ) < PS Sa + 1

. 528

529

Consistent with the current implementation of the UDM performance standard, H 0 is 530

rejected in favor of the alternative H

a if the obser ved number of positive carriers N Pa in 531

each of the K Pa tests against P. aeruginosa is less than or equal to PS Pa and the observed 532

number of positive carriers N Sa in each of the K Sa tests against S. au r

e u s is less than or 533

equal to PS Sa

. 534

535

The Type I pass

- err or rate α KL for a performance standard that requires that a disinfectant 536

pass all K = K Pa + K Sa tests for both organisms at each of L labs is calculated by 537

538

α KL = Pr (Rejecting H 0 in all KL tests | H 0 : true median N Pa = PS Pa + 1 539

true median N Sa = PS Sa + 1 ) 540

= Pr (Passing all KL tests | H 0 : true median N Pa = PS

Pa + 1 541

true median N Sa = PS Sa + 1 ) 542

( 17 ) = Pr ( T ≥ t 1 in all KL tests | H 0 : true median N Pa = PS

Pa + 1 ) 54313 true median N

Sa = PS Sa + 1 ). 544

545

If it is assumed that, within each lab, the P. aeruginosa test results are independent of the 546

S. aureus test results, then the pass

- error rate in equation (17) can be calculated by 547

548

. 549

550

In the previous equation, is the Type I pass

- error rate for K Pa P. aeruginosa tests 551

and is the pass

- error rate for K S a S. aureus tests, with all K = K Pa + K

Sa tests 552

performed in each of L labs. The rates and can each be calculated using 553

equation ( 9 ). 554

The probability that a highly effective product fails to pass al l KL tests is 555

556

β KL = Pr (Failing to reject H 0 in at least one test | H

a : true median N Pa = 0 557

true median N S a = 0) 558

= 1

- Pr (Rejecting H 0 in all K L tests | H

a : true median N Pa = 0 559

true median N S a = 0) 560

(1 8 ) = 1

- Pr ( T ≥ t 1 in all K L tests | H

a : true median N Pa = 0 561

true median N S a = 0). 562

563

If it is assumed that, within each lab, the P. aeruginosa test results are independent of the 564

S. aureus test results, then this equation can be simplified to 565

566

567

568

where and are the Type II fail

- error rates for the K Pa P. aeruginosa tests and 569

the K

S a S. aureus tests respectively, with all K = K Pa + K Sa tests performed in each of L 570

labs. The fail

- error rate for each organism can be calculated by equation (11). 571

572

We have already explained that tests using the same microbe conducted in the same 573

laboratory may not be uncorrelated

. For similar reasons, P. aeruginosa test results may 574

be correlated with S. au reus test results when the tests are conducted in the same 575

lab oratory

. Although the UDM collaborative study was explicitly designed to reliably 576

estimate the correlation among LRs of tests of the same microbe conducted in the same 577

lab oratory , the data from the UDM collaborat i ve did not provide a reliable estimat e of the 578

correlation among tests of different microbes conducted in the same lab oratory . Thus, we 579

calculated the error rates for a range of hypothetical correlation values: 0.00, 0.25, 0.5, 580

0.95, and then reported the range of error rates found. 581

582

To account for th e between microbe correlation , a multivariate t

- distribution ( 26 ) was 583

used to c alculat e the error rate s for a performance standard which requires that all tests 584

from both organisms must be passed

. The necessary inputs for the calculations using the 585

multivariate t are: t 1 and λ 1 calculated for each organism separately. Unlike the values 586

used in equations (7) and (8) for each microbe separately , the degrees of freedom df 1 = 587

6.9 for P. aer uginosa was used since it was smaller and thus more conservative. The 58814

calculations were also performed for df 1 = 4, the number of labs minus one, which 589

resulted in negligibly different error rates than those reported. Given these inputs, as well 590

as the matrix of correlation s amongst the tests

, statistical software is readily available to 591

perform the calculations (see Appendix B). In some instances, for the between microbe 592

correlation values of either 0.5 or 0.95, the correlation matrix was no longer positive 593

definite, in which case the error rates were not calculable. An intuitive explanation of 594

this failure of the model to generate the error probabilities of interest is that the observed 595

variability of the two organisms is inconsistent with these h igh hypothetical correlations. 596

597

Modeling Assumptions 598

599

Figure 1 provides an overview of the assumptions used at each step of the modeling 600

process. The first critical assumption of the model is that the expected variability (i.e. the 601

values of the variance components S 2

lab and S 2

test ) of any product subjected to the UDM is 602

assumed to be the same as the variability observed in the UDM collaborative study . A 603

possible critique of this assumption is that only a single high efficacy product was used to 604

calculate these variances. Although conventional guidelines for collaborative studies 605

recommend the inclusion of multiple disinfectant formulations ( 25 ), a single disinfectant 606

study can yield results of practical value. If available resources are limit ed

, as was the 607

case for the UDM collaborative study

, then it is advisable to use a single disinfectant in 608

all tests; otherwise, to reduce cost, one would be forced to use less lab oratories in which 609

case the reproducibility estimates S R would be less reliable (i.e., S 2

R would have lower 610

degrees of freedom) . For similarly efficacious products having similar LR values within a 611

narrow range, experience shows that the reproducibility SDs, S R , among the products are 612

also similar

. Highly efficacious treatments produce small S R values ; completely 613

ineffective treatments produce small S R values ; and partially efficacious treatments 614

produce medium or high S R values ( 27

- 2 9 ) . Therefore, w e believe that the 615

reproducibility SD, S R , produced by the high efficacy treatment in the single disinfectant 616

study of the UDM is representative of the reproducibility for most other high efficacy 617

disinfectants and that the observed S R can be used when crafting performance standards. 618

Support for this claim is found in the multi

- lab collaborative study results for the HSCT

. 619

The reproducibility SD

, S R , for the single disinfectant UDM study are consistent with the 620

S R values for two earlier multi

- disinfectant HSCT collaborative studies for P. aeruginosa 621

( 18, 30 ). F or all the different high efficacy disinfectant formulations tested in the UDM 622

and HSCT collaborative studies, the range of S R values was 0.09 to 0.66 and the median 623

S R was slightly less than 0.50. Thus, if an additional single dis infectant collaborative 624

study of the UDM was conducted with P. aeruginosa with a different high efficacy 625

disinfectant, one would expect the resulting S R to be about 0.50. Note that the statistical 626

calculations for the UDM performance standards presently being proposed in this paper 627

are based on S R = 0.53 for P. aeruginosa from the UDM collaborative study. The pass

628

error and fail

- error rat es will be smalle r for more effective disinfectants ( since they are 629

presumed to have smaller variability) and larger for le ss

- effective disinfectants (that 630

exhibit larger variability). 631

A second assumption of importance to the calculation of the error rates is that the LRs, 632

calculated from the control carrier counts and the MPN

- transformed numbers of positive 633

carriers, are normally distributed with homogeneous within

- lab variance component S 2

test 63415

across all labs for each microbe

. By equation (13 ), the homogeneous variance assumption 635

is equivalent to assuming a homogeneous correlation amongst tests using the same 636

microbe in the same lab. T hese assumptions are easily checked with well established 637

diagnostic tools. In our experience, the LR values usually satisfy the assumptions of 638

normality and homogeneity of variance across a variety of different test organisms and 639

standard methods ( 11

, 18 , 2 4, 31

- 34 )

. 640

641

The third assumption, w hen calculating the error rates for a performance standard which 642

require s that all tests from both P. aeruginosa and S. aureus must be passed, was that the 643

LRs from tests of different microbes conducted in the same lab ha ve the same correlation 644

in all labs . Although it was not possible to reliably estimate this correlation from the 645

UDM collaborative study, the inability to find a significant correlation, and the disparate 646

among

- lab variance of each of the two organisms (Table 1) , is suggestive that such a 647

correlation , if it exists, is small. Nonetheless, we calculated the error rates for a wide 648

range of hypothetical correlation values: 0.00, 0.25, 0.5, and 0.95

. 649

650

A fourth critical assumption is that t he pass

- error rate is calculated with respect to a 651

“marg inally ineffective” product that has a median number of positive carriers that 652

exceeds the performance standard by one positive carrier. The pass

- error rate c ould have 653

been calculated for a different ineffective product , for example, of a highly ineffecti ve 654

product with true median number of positive carriers equal to twenty. Nonetheless, the 655

pass

- error rate for a marginally ineffective product is larger than the error rate for any 656

other ineffective product. 657

658

The f

if th and last fundamental assumption i s that the fail

- error rates have been calculated 659

with respect to a highly efficacious disinfectant that has a median number of positive 660

carriers of zero over many tests. O ne could instead calculate the fail

- error rate for a 661

“ marginally efficacious” produc t , for example, when the true median number of positive 662

carriers just meets the allowab le number of positive carriers

. Such a product would have 663

a higher failure rate than a highly effective product. 664

665

Estimates of Log Reduction 666

667

One defining feature of the modeling approach presented in this paper is transforming the 668

number of positive carriers observed in a 60 carrier UDM test to a LR value. The 669

advantages of using the LR instead of analyzing the numbers of positive carriers some 670

other way are: 1) the resu lting value is directly interpretable as the LR of organisms over 671

the entire 60 carrier set; 2) the results of the variability of the method can be directly 672

compared to other studies of disinfectant tests, which are usually analyzed with respect to 673

the LR; 3) for analyses such as ANOVA and t

- tests, appropriate for small samples of 674

normally distributed LRs, can be directly applied; and 4) the normality assumption is 675

easily verified with well known diagnostic tools. 676

677

It is known that the actual LR s for a disinfectant are likely larger than what is estimated 678

by the MPN formula when zero positive carriers are observed. This is the same challenge 679

faced by other possible analyses of UDM data such as probit and logit (see equations (1) 68016

and (2)). If it were poss ible to enumerate organisms on all 60 treated carriers and estimate 681

a LR that way, then statistical theory maintains that this LR , and the LR calculated from 682

the MPN formula, would both estimate the same true mean LR of the disinfectant applied 683

in the UDM. Nonetheless, it is not to be expected that the LR calculated for a disinfectant 684

based on UDM outcomes will be predictive of the LR value obtained for the same 685

disinfectant applied in a different method, such as the ASTM Q uantitative C arrier Disk 686

T est ( 29 )

. 687

688

One alternative to the MPN transform approach is the use of the probit transform (see 689

equation (1)) employed by Ortenzio and Stuart ( 5 ) , which provide s the justification for 690

the current performance standard of passing a disinfectant in a single UDM test

. 691

Unfortunately, the probit response variable, which has units of normal percentiles, has no 692

practical meaning with respect to the UDM or the disinfectant being tested. A second 693

alternative analysis is the logit analysis (in equation (2)). There are many other possible 694

transforms of UDM numbers of positive carriers as well ( 33 ) . Unlike a probit transform, 695

the logit response is interpretable as the log of the odds of getting a negative carrier out of 696

60. Nonetheless, using either the probit or logit models, the variance components are not 697

interpretable with respect to other disinfectant tests which use LRs

, and in contrast to the 698

MPN formula, neither explicitly estimates the number of organisms killed in the test. 699

Like the ANOVA models which are fit to MPN

- transformed UDM data, the iteratively 700

reweighted least squares techniques used to analyze probit and logit transformed data 701

also yield maximum likelihood estimates of model parameters ( 17, section 14.4). 702

However, although ANOVA models of LRs provi de valid statistical inference for 703

variance components and other questions of interest for small numbers of LRs (since LRs 704

tend to be normally distributed), attempting the same inference via probit and logit 705

analyses is only valid for large sample sizes ( 1 7 , section 14.7). A last critique of non

706

MPN transform approach is that to account for the control counts , a covariate would need 707

to be added to the generalized linear model. This imposes the additional assumption that 708

changes in the control carrier coun ts affect the number of positive carriers similarly 709

across all tests

. Use of the LR values calculated by using the MPN formula (equation 710

(4)) has no such additional assumption. 711

712

Results and Discussion 713

714

E rror rates of the current performance standard f or a single UDM test 715

716

When a single UDM test is performed

, the expected pass

- error and fail

- error rates using 717

the current UDM performance standard which allows up to PS 1 = 1 positive carrier can 718

be determined . F or P. aeruginosa , t o calculate the chance of a pass

- error of a n 719

ineffective treatment (w h ich has median number of positive carriers equal to 2 )

, the test 720

statistic critical value is 721

722

t 1 = (LR( 0 )

– LR(2))/ [0.175 + 0.111] 1/2 723

= ( 8.1038

– 7.8783 )/ 0.53 48 724

= 0.4217

. 725

72617

T he variance components in the denominator i n the preceding calculation were obtained 727

from the UDM collaborative study data reproduced in Table 1. T he deg rees of freedom 728

using equation (19) in the Appendix wa s found to be df 1 = 6.9. Applying equation (7 )

, 729

the pass

- error rate for a single UDM test using P. aeruginosa is α 1 = 0.343 . To 730

determine the fail

- error rate, equation (20) in Appendix A shows that the non

- centrality 731

parameter is 732

733

λ 1 = (LR( N = 1)

– LR( N = 2))/ [0.175 + 0.111] 1/2 734

= ( 8.5846

– 7.8783 )/0.53 48 735

= 1.32

. 736

737

Applying equation (8)

, the fail

- error rate is β 1 = 0.182

. Similar calculations for S. aureus 738

yield α 1 = 0.244 and β 1 = 0.064 based on t

- distributions with df 1 = 13.8 and λ 1 = 2.23. 739

The error rates for P. aeruginosa and S. aureus for a single UDM test are presented in the 740

top two rows of Table 2

. The co mmands which generated these values using the 741

statistical software R are provided in Appendix B. The se pass

- error rate s ar e much 742

higher than the 0.05 value arrived at by the unpublished data of Ortenzio & Stuart ( 5 )

, 743

which presumably was based on UDM tests using S. aureus and S. choleraesuis

. The 744

authors hypothesize that this large difference in pass

- error rates is due to th e fact that 745

their calculation s were based on the observed within

- lab variance from a single lab

, 746

whereas our calculations are based on a within

- lab variance averaged across multiple 747

labs. 748

749

Error rates of the current pe rformance standard f or multiple UDM tests 750

751

For product registration, a product must pass all of K = 3 tests

, all in a single lab, with 752

PS 1 = 1, for each test of P. aeruginosa and S. aureus

. 753

754

Based on the UDM collaborative study data set, the lab

- to

- lab variance component for S. 755

aureus was not significant and estimated to be zero . T hus

, one may assume that S. 756

aureus tests conducted within the same laboratory are uncorrelated, and so e qua tion (10) 757

yields a good approximation of the pass

- error rate of ineffective disinfectants which are 758

requi

red to pass all of three S. a u reus tests, 759

760

α K L = (0.2438 ) 3 = 0.01 5

. 761

762

The fail

- error rate for a highly efficacious product applied to three independent S. aureus 763

UDM tests is predicted to be (via equation (12) ) 764

765

β K L = 1

- (1

– 0. 0644 ) 3 = 0.18 1

. 766

767

These calculations are very close to th e ones presented in Table 2. They are not exactly 768

the same since Table 2 was calculated via equations (9) and (11) using a multivariate t 769

calculation

. 770

77118

Similar calculations assuming independent tests cannot be performed fo r P. aeruginosa 772

since the results from the UDM collaborative study found significant lab

- to

- lab 773

variability (see Table 1). Equation (1 3 ) shows that there is a correspondingly high 774

correlation of r = 0.61 amongst P. aeruginosa tests conducted in the same l ab. Thus, the 775

error rates for P. aeruginosa must be found via equations (9) and (11) using a 776

multivariate t calculation. The results of these calculations are provided in Table 2. The 777

code which generated these error rates using the statistical software R is provided in 778

Appendix B. 779

780

Comparing theoretical and empirical UDM error rates 781

782

It is instructive to compare the theoretical pass

- error rates just derived for the current 783

performance standard with the empi rical pass

- error rate ( for an ineffective low efficacy 784

treatment) and the empi rical fail

- error rate ( for an effective high efficacy treatment) 785

observed in the UDM collaborative study . These empirical rates over 15 tests are 786

presented in Table 3

. Unlike the assumed median numbers of positive carriers in Table 2, 787

the observed median values for P. aeruginosa was 1 (i nstead of 0) for the effective 788

treatment and 15 (instead of 2) for the ineffective treatment. For S. aureus , the observed 789

median value was 0 for the effective treatment, and 4 for ineffecti ve t reatment. Thus

, 790

except for the high ly effective treatment applied to S. aureus

, the values presented in 791

Table 3 are estimated lower bounds of the theoretical pass

- error rates and upper bounds 792

of the theoretical fail

- error rates provided in Table 2. Even though only 15 tests were 793

conducted, the resulting bounds agree with the theoretically obtained values, except for 794

the fail

- error rate for the high ly effective treatment applied to S. aureus . A larger 795

collaborative study with many independent tests w ould need to be run, with a high ly 796

effective disinfectant with median number of positive carriers N = 0, and a m arginally 797

ineffective treatment with median number of positive carriers N = 2

, in order to 798

empirically corroborate the theoretical results in Ta ble 2. 799

800

Proposing a new performance standard for the UDM 801

802

T h is paper provide s the explicit calculations necessary to determine the pass

- error and 803

fail

- error rates for performance standards which require passing only a single test

, 804

passing all of multiple tests

, or passing multiple tests on average. In order to find a 805

performance standard which keeps both the pass

- err or and fail

- error rates as low as 806

possible (e.g., <10%) while providing for a minimum amount of testing, a set of po ssible 807

testing scenarios and performance standard s for the UDM was investigated . The set was 808

constructed by varying: the allowable numbers of positive carriers for each test; the 809

numbe rs of tests that must be passed; and different numbers of labs performin g the tests

. 810

The results , calculated via equations (9) and (11), are provided in Table 4 for P. 811

aeruginosa and in Table 5 for S. aureus

. Tables 4 and 5 provide results for a 812

performance standard on a single microbe, and allow a direct comparison to the U DM 813

and HSCT performance standard error rates suggested by other authors ( 5 , 10

, 15 ). Table 814

4 suggest s that for a performance standard based only on tests of P. aeruginosa , a multi

815

lab testing scenario would be critical to maintain a pass

- error rate of 10% or less and a 81619

fail

- error rate of 5% or less (either two tests in each of two labs or one test in each of 817

three labs). 818

819

Table 6 provides the pass

- error rates for a performance standard that requires that all 820

tests from both P. aeruginosa and S. aureus pa ss ( calculated via equations (17) and (18)

. 821

Since the calculation of the error rates in Table 6 requires an estimate of the correlation 822

between microbes , and we do not have a reliable estimate of this correlation, these 823

calculations were performed for a r ange of hypothetical correlation values. T he model ’s 824

output in Table 6 may be used as a tool to develop a practical and informed decision on 825

the best approach to managing registration testing requirements. Table 6 suggests that i n 826

order to simultaneously maintain pass

- error and fail

- error rates less than 5%

, three tests 827

(e.g., testing three pro duction lots o r testing a single lot three time s ) must be run for each 828

organism

, all possibly performed in the same lab

. Using this proposal, a disinfectant 829

would pass each P. aeruginosa UDM test if the number of positive carriers N ≤ 6 (i.e., no 830

more than 6 positive carriers out of 60 tested) and would pass each S. aureus UDM test if 831

N ≤ 3 (i.e., no more than 3 positive carriers out of 60 tested)

, in which case the pass

- error 832

rate of marginally ineffective pr oducts over all six tests is predicted to be between 1

. 2 % 833

and 2

. 8 % and the fail

- error rate o f highly effective products is predicted to be between 834

4

. 9 % and 5

. 0 %

. 835

836

The error rates in Table 6 presume that all future UDM tests applied to any disinfectant 837

exhibit the same variability as observed in the UDM collaborative study, for which the 838

among

- lab variance for S. aureus was estimated to be zero. It is possible that t his 839

variance component of the LRs is under

- estimated since 10 of the 15 S. aureus tests 840

exhibited zero positive carriers. To investigate the influence that a positive among

- lab 841

variance would have on the error rates, the model used to generate Table 6 was rerun, but 842

this time with an among

- lab S. aureus variance equal to the observed among

- experiment 843

variance (i.e., S 2

lab = S 2

test = 0.100). The results are presented in Table 7. Over the range 844

of hypothetical correlations, the range of pass

- error rates for the proposed performance 845

standard increased from 1.2%

- 2.8% to 4.2%

- 10.5%, and the range of fail

- error rates 846

increased from 4.9%

- 5.0% to 9.2%

- 10.6%. Thus, in the presence of a moderate level of 847

among

- lab variability for S

. aureus

, the proposed performance standard maintains a pass

848

error rate of at most 10.5% and a fail

- error rate of at most 10.6%. In this scenario, t o 849

maintain a pass error rate less than 10% or less, multiple tests, possibly performed at 850

different labs, are required (Table 7). 851

852

Another ap proach explored during this exercise was to consider the averaging of UDM 853

data across multiple tests. However, in order to achieve a pass

- error rate at or below 854

10% for P. aeruginosa

, a 10 lab testing scenario was required , with two UDM tests 855

performed at each lab ( these rates were calculated via equations (15) and (16) )

. T his 856

pass

- error rate was attained with only three uncorrelated tests using the “pass

- all” 857

approach. Thus, for the UDM, an averaging performance standard approach appears to 858

be too resou rce intensive

. 859

860

The Tables in this manuscript provide point estimates for the pass

- error and fail

- error 861

rates for different performance standards for the UDM. One could also provide, via 86220

computer simulations, a margin of error and confidence interval for each error rate. Such 863

calculations are beyond the scope of this manuscript. The computer code that generated 864

the error rates in Tables 4

- 7 using the statistical software R is provided in Appendix B. 865

866

Conclusion s 867

868

Concurrent with the development and assessment of new quantitative methodology 869

designed to replace the UDM, the EPA continues to work with AOAC and the 870

stakeholder community to facilitate the introduction, review and publication of editorial 871

and procedural revisions to standard methods. T he current investigation of the UDM 872

performance standard utilizes the best available scientific practices for assessing the 873

available data and is one of many aspects of the method that has been presented for 874

revision

. 875

876

T he statistical evaluation of the multi

- laboratory UDM collaborative data provides a 877

unique opportunity for re

- evaluating the current UDM performance standard. This paper 878

provides the quantitative tools, including computer code, to evaluate a number of opti ons 879

for a new performance standard

. As other authors have proposed, we believe that any 880

proposed performance standard for tests used by regulatory agencies should be based on 881

a quantitative assessment of the pass

- error and fail

- error rates. Based on this a ssessment

, 882

the pass

- error rate for a single UDM test, for either P. aeruginosa or S. aureus , is much 883

higher than originally suggested by previous studies of the UDM and HSCT. This 884

initiative serves to reduce the error rates w hile maintaining a practical level of testing

. 885

Historically, low er pass

- error rates have been achieved by EPA by requiring a registrant 886

to submit efficacy data on three batches of the same product for registration

. 887

Furthermore, d ue to the EPA’s registration specifications for a hospi tal disinfectant which 888

involve efficacy data for S. aureus and P. aeruginosa

, a performance standard should 889

require that a hospital disinfectant pass all tests (six total) against both P. aeruginosa and 890

S. aureus (see Table s 6 and 7 )

. In conjunction with t he number of tests conducted, u se of 891

N ≤ 6 for P. aeruginosa and N ≤ 3 for S. aureus as the passing requirement for each test 892

should improve the overall confidence in the outcome of the test ing scheme

. Thus, two 893

testing scenarios may be considered : A) testing each of three production lots one time per 894

microbe, and B) testing a single production lot three times per microbe. Any of the six 895

test s that fail to meet the performance standard would fail the product. Pass

- error rates of 896

0.3 to 1.0% and fa il

- error rates of 41.6 to 45.1% are predicted using the current 897

performance standard ( i.e., N ≤ 1 for both microbes and three production lots tested per 898

microbe ). When error rates for th e proposed performance standard ( N ≤ 6 for P. 899

aeruginosa and N ≤ 3 for S. aureus ) are calculated using the reproducibility standard 900

deviations observed in the collaborative study

, a low pass

- error rate between 1.2% and 901

2.8% is predicted for marginally ineffective products and the fail

- error rate for highly 902

effective products is predicted to be lowered to approximately 5 % (Table 6)

. T he 903

requirement of N ≤ 1 for each of three tests (three batches of product tested for 904

registration) for S. enterica would remain the same. 905

906

In addition to the proposed new performance standards, m eeting the minimum and 907

maximum carrier count requirements (mean log density of 6

- 7 per carrier) and the use of 90821

the most current and revised version of UDM further strengthens the reliability of the 909

UDM and the confidence of the tes t data

. Furthermore, if the new performance standard 910

is implemented, it would be possible to reassess the performance standard again 911

following the generation of additional data

. 912

913

Based on the information provided in this report , the EPA is seeking AOAC’s technical 914

review of the model and the associated calculations

. Based on this modeling exercise, 915

proposed revisions to Methods 955.15 and 964 .02 are provided. 916

917

The following revisions are proposed following section C ( f ) of each method : 918

919

( g ) Performance sta ndard

. – For method 955.15 ( S. aureus ), S ection C (h ): 920

Performance standard: ≤ 3 positive (growth) carriers out of 60 tested. Conduct 921

t hree separate UDM tests , one test conducted per day

. Each test must meet the 922

performance standard. Testing at a single laboratory is acceptable

. 923

924

( g ) Performance Standard

. – For method 964.02 ( P. aeruginosa ), Section C (h): 925

Performance standard: ≤6 positive (growth) carriers out of 60 tested. Conduct 926

three separate UDM tests, one test conducted per day. Each test must meet the 927

performance standard. Testing at a single laboratory is a cceptable

. 928

929

Acknowledgements 930

931

This work was performed under contract EP

- W

- 08

- 014 between Montana State 932

University and EPA OPP Microbiology Laboratory Branch (MLB)

. The authors 933

gratefully acknowledge substantial contributions from Dr. Darla Goeres at the Center for 934

Biofilm Engineering at Montana State University in the preparation of this manuscript. 935

936

References 937

938

( 1 ) Official Methods of Analysis (20 12 ) 18 th Ed., AOAC INTE RNATIONAL

, 939

Gaithersburg, MD, Method 955.14 940

( 2 ) Official Methods of Analysis (20 12 ) 18 th Ed., AOAC INTERNATIONAL, 941

Gaithersburg, MD, Method 955.15 942

( 3 ) Official Methods of Analysis (20 12 ) 18 th Ed., AOAC I NTERNATIONAL, 943

Gaithersburg, MD, Method 964.02 944

( 4 ) Tomasino, S.F., Pines, R.M., & Hamilton, G.C. (2012) J. AOAC Int

. 95 (in press) 945

( 5 ) Ortenzio, L.F., & Stuart, L. S. (1961) J. AOAC Int

. 44 , 416

- 421 946

( 6 ) Morgan, B.J. (1992) Analysis of Quantal Response Data, Chapman & Hall, New 947

York, NY 948

( 7 ) Cole, E.C., & Rutala, W.A. (198 8 ) J. AOAC Int. 71 , 1 187

- 1 194 949

( 8 ) Rutala, W.A., & Cole, E.C. (1987) Infect. Control 8

, 501

- 506 950

( 9 ) Official Methods of Analysis (2009) 18 th Ed., AOAC INTERNATIONAL, 951

Gaithersburg, MD, Method 991.47 952

(10) Rubino, J.R., Bauer, J. M., Clarke, P.H ., Woodward, B.B., Porter, F.C., & Hilton, 953

H.G. (1992) J. AOAC Int. 75

, 635

- 645 95422

(11) Tomasino, S.F., Parker, A.E., Hamilton, M.A.

, & Hamilton, G.C. (2012) J. AOAC 955

Int

. 96 ( in press ) 956

( 12) Standard Operating Procedure (MB

- 05) for conducting the Use

- diluti on method, 957

U.S. E nvironmental Protection Agency

- Office of Pesticide Programs, Biological 958

and Economic Analysis Division, Microbiology Laboratory Branch 959

(13) Tilt, N.

, & Hamilton, M.A. (1999) J. AOAC Int. 82 , 384

- 389 960

(14) Trout, J. (1985) J. AOAC Int

. 68 ,763

- 765 961

( 15 ) Arlea, C., King, S., Bennie, B., Kemp, K., Mertz, E., & Staub, R. (2008) J. AOAC 962

Int. 91 , 152

- 158 963

( 16 ) Casella & Berger (1990) Statistical Inference, Duxbury Press, Belmont, CA 964

(17) Neter, J., Kutner, M.H., Wasserman, W., & Nachtsheim, C.J. (1996) Applied 965

Linear Statistical Models , McGraw

- Hill Co., Boston, MA 966

(18) Hamilton, M.A., DeVries, T.A., & Rubino, R.R. (1995) J. AOAC Int

. 78 , 1102

- 967

1109 968

( 19 ) Hamilton, M. A. (2011) The P/N formula for the log reduction when using a semi

969

quantitative disinfectant test of type SQ1, KSA

- SM

- 08

, Feb 1, 2011, Center for 970

Biofilm Engineering at Montana State University, Bozeman, MT. 971

http://www.biofilm.montana.edu/resources/knowledge_sharing_articles 972

( 20 ) Cochran, W.G. (1950) Biometrics 6 , 105

- 116 973

( 21 ) McCrady, M.H. (1915) J Infect. Dis

. 17 , 183

- 212 974

( 22 ) Collett, D. (1991) Modelling Binary Data. Chapman & Hall, New York, NY 975

( 23 ) Blodgett, R.J. (2006) Bacteriological Analytical Manual , FDA, 976

http://www.fda.gov/Food/SceinceResearch/LaboratoryMethods/BacterilogicalAna 977

lytical ManualBAM/default.htm , Appendix 2 978

( 24 ) Tomasino, S.F., & Hamilton, M.A. (2006) J. AOAC Int. 89 , 1373

- 1397 979

(25 ) Official Methods of Analysis (2005) 18th Ed., AOAC INTERNATION AL, 980

Gaithersburg, MD, Appendix D 981

( 2 6 ) Genz, A., & Bretz, F. (2009) Computation of Multivariate Normal and t 982

Probabilities Springer

- Verlage, Heidelberg, Germany. ISBN 978

- 3

- 642

- 01688

- 2 983

( 27 ) Bloomfield , S.F., & Looney, E. ( 1992 ) J. Appl. Bacteriol. 73 , 87

- 93 984

( 28 ) Bloomfield, S.F., Arthur, M., Van Klingeren, B., Pullen, W., Holah, J.T., & Elton, 985

R. (199 4 ) J. Appl. Bacteriol. 76 , 8 6

- 9 4 986

( 29 ) Springthorpe, V.S.

, & Sattar, S.A. (2005) J. AOAC Int. 88 , 182

- 201 987

( 30 ) Hamilton, M. A. (2001) “Statistical report: prelimi nary analysis of the second 988

collaborative study of the hard surface carrier test,” Aug 31, 2001

, EPA contract 989

#68

- W

- 99

- 015 990

( 31 ) Tomasino, S.F., Pines, R .M., Cottrill, M. P., & Hamilton, M.A. (2008) J. AOAC 991

Int. 91 , 833

- 852 992

( 32 ) Tomasino, S.F., Pines, R.M., & Hamilton, M.A. (2009) J. AOAC Int. 92 , 1531

- 993

1540 994

( 33 ) Hamilton, M.A.

, & DeVries, T.A. (1996) Biometrics 52 , 1112

- 1120 995

( 34 ) Buckingham

- Meyer, K

. , Goeres, D., & Hamilton, M. (2007) J. Microbiol. Meth. 996

70 , 236

- 244 997

( 3 5 ) McCullagh, P.

, & Nelder J. (1989) Generalized Linear Models , Chapman & Hall, 998

New York, NY 999

( 3 6 ) Nijhuis, M.B.

, & Van den Heuvel, E. R. (2007) J Biopharm. Stat. 17, 123

– 142 100023

( 3 7 ) Mee, R. (1984) Technometrics 26 , 251

- 254 1001

( 3 8 ) R Development Core Team (2010)

, R: A language and environment for statistical 1002

computing

, R Foundation for Statistical Computing, Vienna, Austria

, ISBN 3

- 1003

900051

- 07

- 0, http://www.R

- project.org 1004

( 39 ) Genz, A., Bretz, F., Miwa, T., Mi, X., Leisch, F., Scheipl, F., & Hothorn, T. 1005

(2011) mvtnor m: Multivariate Normal and t Distributions

, R package version 0.9

- 1006

96

, http://CRAN.R

- project.org/package=mvtnorm 1007

1008

1009

1010 Figure 1

. An overview of the modeling process and assumptions used to calculate the 1011

two error rates. The number of positive carriers out of 60 observed in a UDM test is 1012

denoted by N . The maximum allowable number of positive carriers allowed by a 1013

performance standard is denoted by PS . A log reduction is denoted by LR. 1014

1015

Table 1

. Variance components and reproducibility SD of LR for each of the two 1016

microbes in the UDM collaborative study ( 11, Table 6 ). 1017

Organism Among

lab

variance

( S 2

lab ) Within

lab

variance

( S 2

test ) Ratio

S 2

lab / S 2

test Intra

- lab

correlation Reproducibility

SD

P. aeruginosa 0.175 0. 111 1.5766 0.6119 0.5348

S. aureus 0. 000 0. 100 0.0000 0.0000 0.3162

1018

101924

Table 2

. Estimated error rates, expressed as percentages, for the current UDM 1020

performance standard (which allows up to 1 positive carrier out of a 60 inoculated and 1021

treated set). “Three tests in one lab” is the current registration performance standard; 1022

“one test” is the current post

- registration performance standard. 1023

Organism PASS ALL of this Performance s tandard Pass

- error Fail

- error

number of t ests required to PASS rate (α) rate (β)

each test (median N = 2) (median N = 0)

Pa 1 test N ≤ 1 34.30% 18.19%

Sa 1 test N ≤ 1 24.38% 6.44%

Pa 3 tests in 1 lab N ≤ 1 15.50% 33.45%

Sa 3 tests in 1 lab N ≤ 1 1.59% 18.01%

1024

1025

Table 3

. Empirical error rates, expressed as percentages, for the current UDM 1026

performance standard (which allows up to 1 positive carrier out of a 60 inoculated and 1027

treated set) (11, Tables 8 and 9, Figures 1 and 2). Unlike the assumed median numbers of 1028

positive c arriers in Table 2, the observed median values for the high efficacy treatment 1029

applied to P. aeruginosa was 1 (instead of 0), and 15 (instead of 2) for the low efficacy 1030

treatment. For S. aureus , the observed median values were 0 for the high efficacy 1031

trea

tment, and 4 for the low efficacy treatment. Thus, the values in this table are bounds 1032

for the theoretical rates given in Table 2. Rates are reported as percentages. 1033

1034

Organism PASS ALL of this Perfo rmance s tandard Pass

- error Fail

- error

number of t ests required to PASS rate (α) rate (β)

each test

Pa 1 test N ≤ 1 ≥ 0% (0/15) ≤ 26.7% (4/15)

Sa 1 test N ≤ 1 ≥ 20% (3/15) ≤ 33.3% (5/15)

Pa 3 tests in 1 lab N ≤ 1 ≥ 0% (0/15) ≤ 60% (9/15)

Sa 3 tests in 1 lab N ≤ 1 ≥ 0% (0/15) 60% (9/15)

1035

103625

Table 4

. Estimated error rates, expressed as percentages, for hypothetical 1037

performance standards for P. aeruginosa that require a disinfectant to pass all of 1038

multiple tests as the numbers of labs, numbers of tests at each lab, and the 1039

allowable number of p ositive carriers ( N ) out of a 60 carrier set per test is varied. 1040

The number of labs is critical due to the large lab

- to

- lab variance for P. 1041

aeruginosa

. 1042

Num.

labs Num. tests

at each lab PASS ALL

of this

num. tests Performance

s tandard to

PASS each

test Pass

- error

rate (α) Fail

- error

rate (β)

1 1 1 N ≤ 1 34.30% 18.19%

1 1 1 N ≤ 2 39.38% 9.23%

1 1 1 N ≤ 3 41.94% 5.42%

1 1 1 N ≤ 4 43.49% 3.47%

1 1 1 N ≤ 5 44.52% 2.35%

1 1 1 N ≤ 6 45.26% 1.67%

1 3 3 N ≤ 1 15.50% 33.44%

1 3 3 N ≤ 2 19.20% 18.88%

1 3 3 N ≤ 3 21.23% 11.81%

1 3 3 N ≤ 4 22.49% 7.89%

1 3 3 N ≤ 5 23.36% 5.53%

1 3 3 N ≤ 6 23.97% 4.02%

2 1 2 N ≤ 1 12.02% 32.91%

2 1 2 N ≤ 2 15.64% 17.56%

2 1 2 N ≤ 3 17.68% 10.64%

2 1 2 N ≤ 4 18.98% 6.80%

2 1 2 N ≤ 5 19.87% 4.66%

2 1 2 N ≤ 6 20.53% 3.30%

2 2 4 N ≤ 1 4.62% 47.10%

2 2 4 N ≤ 2 6.58% 27.43%

2 2 4 N ≤ 3 7.78% 17.25%

2 2 4 N ≤ 4 8.57% 11.52%

2 2 4 N ≤ 5 9.14% 8.04%

2 2 4 N ≤ 6 9.55% * 5.81% *

3 1 3 N ≤ 1 4.26% 44.92%

3 1 3 N ≤ 2 6.24% 25.12%

3 1 3 N ≤ 3 7.47% 15.35%

3 1 3 N ≤ 4 8.29% 10.03%

3 1 3 N ≤ 5 8.88% 6.89%

3 1 3 N ≤ 6 9.31% 4.91%

104326

Table 5

. Estimated error rates, expressed as percentages, for hypothetical 1044

performance standards for S. aureus that require a disinfectant to pass all of 1045

multiple tests as the numbers of tests and the allowable number of positive 1046

carriers ( N

) out of a 60 carrier set per test is varied. The error rates are 1047

representative of either single or multiple lab scenarios (due to the insignificant 1048

lab

- to

- lab var iance for S. aureus ). 1049

PASS ALL of this

num. t ests Performance s tandard

req uired to PASS each

test Pass

- error rate (α) Fail

- error rate (β)

1 N ≤ 1 24.38% 6.44%

1 N ≤ 2 32.15% 1.28%

1 N ≤ 3 36.32% 0.34%

1 N ≤ 4 38.89% 0.11%

2 N ≤ 1 6.15% 12.43%

2 N ≤ 2 10.46% 2.54%

2 N ≤ 3 13.27% 0.68%

2 N ≤ 4 15.18% 0.22%

3 N ≤ 1 1.59% 18.01%

3 N ≤ 2 3.43% 3.78%

3 N ≤ 3 4.87% 1.01%

3 N ≤ 4 5.94% 0.33%

105027

Table 6

. Estimated error rates, expressed as percentages, for hypothetical 1051

performance standards that require a disinfectant to pass all of multiple tests for 1052

both microbes as the hypothetical correlation between the LRs of tests of the 1053

different microbes, the num bers of labs, and the numbers of tests for each microbe 1054

at each lab is varied. For the “current performance standard,” the allowable 1055

number of positive carriers ( N ) out of a 60 carrier set per test is 1 for P. 1056

aeruginosa and 1 for S. aureus; for the “prop osed performance standard” the 1057

allowable number of positive carriers is 6 for P. aeruginosa and 3 for S. aureus

. 1058

“Not calculable” scenarios are denoted by “NC” and indicate high correlation 1059

values that are inconsistent with the observed variability for ea ch organism 1060

separately

. 1061

1062

Current

Performance

Standard Proposed

Performance

Standard

Correlation

between

Microbes Num.

Labs Num.

P.a. tests

at ea. lab Num.

S.a. tests

at ea. lab Pass

error

rate

(α) Fail

error

rate

(β) Pass

error

rate

(α) Fail

error

rate

(β)

0 1 1 1 8.9% 23.3% 16.7% 2.0%

0.25 1 1 1 11.9% 22.4% 20.4% 2.0%

0.5 1 1 1 15.2% 21.1% 24.5% 1.9%

0.95 1 1 1 23.9% 18.2% 34.8% 1.7%

0 1 2 2 1.5% 36.2% 4.2% 3.6%

0.25 1 2 2 3.1% 33.9% 7.1% 3.5%

0.5 1 2 2 5.1% 31.2% 10.5% 3.4%

0.95 1 2 2 NC NC NC NC

0 1 3 3 0.3% 45.1% 1.2% 5.0%

0.25 1 3 3 1.0% 41.6% 2.8% 4.9%

0.5 1 3 3 NC NC NC NC

0.95 1 3 3 NC NC NC NC

0 2 1 1 0.9% 41.0% 2.8% 4.0%

0.25 2 1 1 1.6% 39.5% 4.2% 3.9%

0.5 2 1 1 2.5% 37.6% 6.0% 3.8%

0.95 2 1 1 6.0% 33.0% 12.2% 3.3%

0 2 2 2 0.0% 58.8% 0.2% 7.1%

0.25 2 2 2 0.1% 55.9% 0.5% 6.9%

0.5 2 2 2 0.3% 52.4% 1.1% 6.6%

0.95 2 2 2 NC NC NC NC

0 3 1 1 0.1% 54.4% 0.5% 5.9%

0.25 3 1 1 0.2% 52.7% 0.9% 5.8%

0.5 3 1 1 0.4% 50.5% 1.5% 5.7%

0.95 3 1 1 1.6% 45.1% 4.3% 4.9%

106328

Table 7

. Assuming an among

- lab variance for S. aureus of the same magnitude as 1064

the observed within

- lab variance, these are the estimated error rates, expressed as 1065

percentages, for hypothetical performance standards that require a disinfectant to 1066

pass all of multip le tests for both microbes as the hypothetical correlation 1067

between the LRs of tests of the different microbes, the numbers of labs, and the 1068

numbers of tests for each microbe at each lab is varied. For the “current 1069

performance standard,” the allowable numb er of positive carriers ( N ) out of a 60 1070

carrier set per test is 1 for P. aeruginosa and 1 for S. aureus; for the “proposed 1071

performance standard” the allowable number of positive carriers is 6 for P. 1072

aeruginosa and 3 for S. aureus . “Not calculable” scenari os are denoted by “NC” 1073

and indicate high correlation values that are inconsistent with the observed 1074

variability for each organism separately

. 1075

1076

Current

Performance

Standard Proposed

Performance

Standard

Correlation

between

Microbes Num.

Labs Num.

P.a. tests

at ea. lab Num.

S.a. tests

at ea. lab Pass

error

rate

(α) Fail

error

rate

(β) Pass

error

rate

(α) Fail

error

rate

(β)

0 1 1 1 11.08% 29.44% 18.4% 4.4%

0.25 1 1 1 14.42% 27.80% 22.2% 4.3%

0.5 1 1 1 18.11% 25.75% 26.4% 4.0%

0.95 1 1 1 28.31% 19.53% 37.5% 3.0%

0 1 2 2 3.76% 43.56% 7.6% 7.8%

0.25 1 2 2 6.17% 40.51% 11.0% 7.5%

0.5 1 2 2 9.22% 36.87% 15.0% 6.9%

0.95 1 2 2 NC NC NC NC

0 1 3 3 1.83% 52.30% 4.1% 10.6%

0.25 1 3 3 3.57% 48.36% 6.9% 10.1%

0.5 1 3 3 6.01% 43.72% 10.4% 9.2%

0.95 1 3 3 NC NC NC NC

0 2 1 1 1.32% 49.92% 3.4% 8.5%

0.25 2 1 1 2.19% 47.62% 5.0% 8.3%

0.5 2 1 1 3.42% 44.66% 7.0% 7.9%

0.95 2 1 1 8.21% 35.12% 14.1% 5.9%

0 2 2 2 0.16% 67.72% 0.6% 14.9%

0.25 2 2 2 0.42% 64.25% 1.2% 14.4%

0.5 2 2 2 0.91% 59.84% 2.3% 13.4%

0.95 2 2 2 NC NC NC NC

0 3 1 1 0.17% 64.27% 0.6% 12.5%

0.25 3 1 1 0.35% 61.84% 1.1% 12.2%

0.5 3 1 1 0.67% 58.61% 1.9% 11.6%

0.95 3 1 1 2.43% 47.60% 5.3% 8.7%

107729

Appendix A

- Discussion and details for selected modeling steps 1078

1079

Previous Models Used for Assessing the UDM and HSCT Data 1080

1081

In order to evaluate equations (1)

- (3) at the extreme cases of N = 0 or N = 60, the 1082

proportion of negative carriers p N that we consider includes a 1⁄2 in the numerator and a 1 1083

in the denominator

. Morgan ( 6

, p 58)

, McCullagh & Nelder ( 3 5 , p 117) and Collett ( 22

, 1084

p 80) show that the addition of these terms is appropriate. 1085

1086

Step 3: Calculating the error rates for a single UDM test 1087

1088

For the competing null and alternative hypotheses in equation (6) , t he rejection region 1089

RR 1 is 1090

1091

RR 1 = { All N such that N ≤ PS 1 }, 1092

1093

where N is the number of positive carriers observed in a single UDM test of 60 carriers. 1094

Since the LR transform ( via equation (4)) is a monotonically decreasing function of N for 1095

a fixed TestLD , and because the s ymmetrically distributed LR s have median equal to the 1096

mean, the n the hypotheses in equation (6) can be rewritten in a mathematically equivalent 1097

form as 1098

1099

H 0 : μ LR ≤ LR( PS 1 + 1 ) 1100

H

a : μ LR > LR( PS 1 + 1 ). 1101

1102

In these new hypotheses, the true mean LR is denoted by μ LR

. The rejection region with 1103

respect to the LRs is 1104

1105

RR 1 = { All LRs such that LR ( N ) ≥ LR( PS 1 ) } 1106

= {( LR( N )

– LR( PS 1 + 1))/

S R ≥ (LR( PS 1 )

– LR( PS 1 + 1))/ S R } 1107

= { T ≥ t 1 }. 1108

1109

The bold font indicates random variables throughout th e sequel

. The pass

- error rate can 1110

now be calculated just as a Type I error is calculated in statistical hypothesis testing 1111

(Casella & Berger, section 8.3.1). The random variable 1112

1113

T = ( LR( N )

– LR( PS 1 + 1))/ S R 1114

1115

is only approximately a t ( L

- 1) random variable, even when the LRs are normal, since 1116

( L

- 1) S

2

R /E( S 2

R ) is only approximately a χ 2 ( L

- 1) random variable ( 17 , p 971). Nijhuis 1117

& Van den Heuvel ( 3 6 ) showed that this approximation is particularly good and has a 1118

negligible effect on pr obability calculations. The critical value

, t 1 , for the test statistic is 1119

completely determined by the performance standard and the observed reproducibility SD 1120

observed in the collaborative study, 1121

1122

t 1 = (LR( PS 1 )

– LR( PS 1 + 1))/ S R

. 112330

Given a performance standard PS 1 , the LRs in the numerator can be calculated by 1124

equation (4 ). The standard deviation S R in the denominator is the reproducibility SD of 1125

the LRs across multiple labs as in equation ( 5 ) , and estimated from the UDM 1126

collaborati ve study ( Table 1 )

. T he pass

- error rate α 1 of a single test can now be 1127

calculated by application of equation (7) , which requires probability calculations based 1128

on a t

- distri bution with non

- centrality parameter λ 1 = 0, and degrees of freedom 1129

1130

( 19 )

. 1131

1132

In equation (19)

, I = 5 is the number of labs in the UDM collaborative study; J = 3 is the 1133

number of repeated tests in each lab ; and R = S 2

lab / S 2

test , where both S 2

lab and S 2

test are the 1134

variance components from the UDM collaborative study (see Table 1 )

. Not to be 1135

confused with the well known result that the standard error of the mean LR across the I 1136

labs, [( S 2

test

/ J + S 2

lab ) / I ] 1/2 , has degrees of freedom equal to I

- 1 (17) , equation (19) gives 1137

the degrees of freedo m associated with S R . Equation (19) can be derived directly using 1138

Satterthwaite’s formula applied to the independent mean squares and the ir degrees of 1139

freedom as in Mee ( 3 7 )

. Equations (5) and (19) make clear precisely how the variance 1140

components S 2

lab and S 2

test from the UDM collaborative study are used to calculate the 1141

pass

- error rate α 1

. 1142

1143

The expected fail

- error rate β 1 of highly efficacious treatments i s calculated just as a 1144

Type II error is in statistical hypothesis testing when the LRs are normal using equation 1145

(8) , which uses a non

- central t

- distribution, with degrees of freedom df 1 calculated as in 1146

equation (19)

, and non

- centrality parameter 1147

1148

(20) λ 1 = (LR(0)

– LR( PS 1 + 1 ) ) / S R

. 1149

1150

In the numerator in equation (20) , we utilize the assumption that the median number of 1151

positive carriers is zero for a highly effective treatment. The preceding equation, as well 1152

as equations (5) and (19) , make clear precisely how the variance components S 2

lab and 1153

S 2

test from the UDM col laborative study are used to calculate the fail

- error rate β 1

. 1154

1155

Alternate Step 4: Calculating the error rates for multiple UDM tests for a single 1156

organism using averaging 1157

1158

The hypotheses being tested in this scenario are given in equation (14)

. If the geometric 1159

mean MPN across the multiple tests attains the performance standard, then the 1160

disinfectant passes. That is, the rejection region RR

mean for this test is 1161

1162

RR

mean = { (geometric mean MPN ) ≤ PS

mean }. 1163

1164

We consider the geometric mean MPN across K tests at each of L labs instead of the 1165

arithmetic mean for three reasons. First, the geometric mean MPN can be converted to 1166

the typical number of positive carriers over many UDM tests; secondly, the geometric 116731

mean is directly transformed to the mean LR for each lab using equation (4) when the 1168

LRs are symmetrically distributed; and thirdly, experience shows that the LRs are 1169

normally distributed

. The hypotheses in equation (14) can be written in a mathematically 1170

equivalent form with respect to the mean LR, 1171

1172

H 0 : μ LR ≤ LR( PS

mean + 1 ) 1173

H

a : μ LR > LR( PS

mean + 1 ). 1174

1175

In these hypotheses, the true mean LR is denoted by μ LR

. The corresponding rejection 1176

region with respect to the arithmetic mean of the LRs across K tests at each of L labs is 1177

1178

RR

mean = { (arithmetic mean LR) ≥ LR( PS

mean ) }. 1179

1180

The critical value of the test statistic corresponding to this rejection region is 1181

1182

t

mean = (LR( PS

mean )

– LR( PS

mean + 1))/ SE

mean . 1183

1184

Since multiple tests are available, this test statistic differs in a single fundament al respect 1185

from the critical value t 1 for the single test performance standard presented earlier. The 1186

denominator is now occupied by SE

mean , the standard error of the mean LR over K future 1187

tests performed at each of L future labs

. To use all available da ta, SE

mean is found by 1188

pooling the standard error obtained from the UDM collaborative study with the standard 1189

error to be observed in the K future tests performed at each of L future labs. E ach of the 1190

variables necessary to evaluate SE

mean are defined nex t . The “future” standard error to be 1191

observed in the multiple UDM tests of the disinfectant will be calculated by 1192

1193

SE future = [ S 2

future, l ab /

L + S 2

future, t est

/( LK ) ] 1/2

, 1194

1195

which has df future = L

- 1 degrees of freedom ( 17 , pp 958

- 976) , with the variance 1196

components S 2

future, l ab and S 2

future, t est calculated from the future UDM test requiring K tests 1197

performed at each of L future labs. The standard error estimated from the UDM 1198

collaborative study is 1199

1200

SE collab = [ S 2

lab / L + S 2

test /( LK ) ] 1/2

. 1201

1202

The variance components S 2

lab and S 2

test in this equation are the same as those calculated 1203

from the UDM collaborative study (see Table 1). In other words, SE collab provides our 1204

best guess for SE future , the standard error of the mean LR in a future study of K tests in 1205

each of L labs. The degrees of freedom associated with SE collab is given by 1206

1207

(21)

. 1208

1209

In equation (21)

, I = 5 is the number of labs in the UDM coll aborative study; J = 3 is the 1210

number of repeated tests in each lab in the UDM collaborative study ; R = S 2

lab / S 2

test , 121132

where both S 2

lab and S 2

test are the variance components from the UDM collaborative 1212

study; and K is the number of tests in each lab in a future test of the disinfectant (e.g., by 1213

enforcement officials) of the disinfectant

. Equation (21) , found via Satterthwaite’s 1214

approximation, is a modification of Mee’s ( 3 7 ) formula (cf. equation (19) )

. The result ing 1215

pooled standard error of the mean is 1216

1217

(22) SE

mean = 1218

1219

The degrees of freedom associated with SE

mean is 1220

1221

(23) df

mean = df

collab + L

- 1. 1222

1223

I t is assume d that the K future UDM tests in each of L labs will yield the same variability 1224

as that observed in the UDM collaborative, S 2

future, l ab = S 2

lab and S 2

future, t est = S 2

test . In this 1225

case, equation (22) reduces to 1226

1227

SE

mean = SE collab

, 1228

1229

with degrees of freedom calcula ted as in equations (21) and (23)

. 1230

1231

Now the expected pass

- error and fail

- error rate when averaging over all KL tests can be 1232

formulated as in equations (15) and (16) using a non

- central t

- distribution s , with degrees 1233

of freedom df

mean calculated as in equation s (21) and (23) . The non

- centrality parameter 1234

in equation (16) is 1235

1236

λ

mean = (LR( N= 0)

– LR( N=PS

mean + 1 )/ S E

mean )

. 1237

1238

1239

Appendix B

- Computer code 1240

1241

T he error rate calculations presented in this manuscript were generated using the 1242

statistical software package R ( 3 8 ) , package mvtnorm ( 26, 39 )

. The R code is presented 1243

in the same order that the m odeling steps were presented earlier

, by first showing how to 1244

calculate the error rates for a performance standard on a single UDM test

, then for a 1245

performance standard which requires passing all of multiple tests

, and then based on 1246

pas s ing multiple tests on average. The following R code should be sequentially entered 1247

directly into R ’s command line

. 1248

1249

The following R function generates th e LR associated with a number of positive carriers 1250

as in equation (4), 1251

1252

GetMPNLR<

- function(N,TestLD=6 .5 ,NumTot=60) 1253

{ 1254

LR = TestLD

- log10(

- log((NumTot

- N+.5)/(NumTot + 1))) 1255

return(LR) 1256

} 125733

1258

For a single UDM test, the pass

- error rate α 1 for the current performance standard is 1259

calculated for each microbe using equation (7) by: 1260

1261

PS 1 = 1 # This corresponds to the current UDM performance standard 1262

SR = 0.5348 # For P. aeruginosa , use SR = 0.3162 for S. aureus 1263

df 1 _Pa = 6.9 # For P. aeruginosa 1264

df1_Sa = 13.8 #

, F or S. aureus 1265

LR.PS 1 = GetMPNLR(PS 1 ) 1266

LR.PS 1 p lus 1 = GetMPNLR(PS 1 +1) 1267

t1 _Pa = (LR.PS 1

- LR.PS 1 p lus 1)/SR 1268

alpha 1 _Pa = 1

- pt(t1 _Pa ,df 1 _Pa ) 1269

t1_Sa = (LR.PS1

- LR.PS1plus 1) /0.3162 1270

alpha1 _Sa = 1

- pt(t1 _Sa ,df1 _Sa ) 1271

1272

1273

For a single UDM test, the fail

- error rate β 1 is calculated using equations (8) and (20) by: 1274

1275

LR. 0 = GetMPNLR(0) 1276

lambda1 _Pa = (LR.0

- LR.PS 1 plus1)/SR 1277

beta1 _Pa = pt(t1 _Pa ,df 1 _Pa ,ncp=lambda1 _Pa ) # for P. aeruginosa 1278

1279

lambda1 _Sa = (LR.0

- LR.PS1plus1)/ 0.3162 1280

beta1 _S a = pt(t1 _S a ,df1 _S a ,ncp=lambda1 _S a ) # for S. aureus 1281

1282

When a performance standard requires that a disinfectant passes all of K L uncorrelated 1283

U DM tests ( this means that each test is performed in a different lab in the case of P. 1284

aeruginosa ), then the pass

- error rate α K L and the fail

- error rate β K L are calculated using a 1285

multivariate t via equation s (9) and (11) by : 1286

1287

library(mvtnorm) 1288

K = 1 # Number of P. aeruginosa tests at each lab 1289

L = 3 # Number of labs 1290

TotTests = K*L # Total n umber of tests 1291

alphaK L = pmvt(lower=rep(t1 _Pa

, TotTests ),upper=rep(Inf, TotTests ),df=df1 _Pa ) 1292

betaKL = 1

- pmvt(lower=rep(t1_Pa, TotTests ),upper=rep(Inf, TotTests),df=df1_Pa,delta=lambda1_Pa) 1293

1294

The error rate calculations are more complicated when multiple tests of a single microbe 1295

are performed in the same lab in the presence of a significant among

- lab variance 1296

component S 2

lab , in which case the correlation matrix amongst the tests must be input ted 1297

in order to evaluate equations (9) and (11)

. For example , the two error rates for two P. 1298

aeruginosa tests performed in each of two labs are calculated by: 1299

1300

# #### An R function to g enerate the correlation matrix for tests of a single microbe 1301

GenMatri x<

- function(NumTests,NumLabs=1,VarLa b=.175,VarExp=.111 ) 1302

{ 1303

Z = matrix(0,NumTests,(NumTests+1)) 1304

Z[,1]=1 1305

for (i in 1:NumTests) 1306

{ 1307

Z[i,i+1]=1 130834 } 1309

Psi=diag(c(rep(VarLab,1),rep(VarExp,NumTests))) 1310

Vblk=Z%*%Psi%*%t(Z) 1311

N = NumTests*NumLabs # total number of data points across all labs and exps 1312

V=matrix(0,N,N) 1313

for (i in 0:(NumLabs

- 1)) 1314

{ 1315

index=(i*NumTests+1):((i+1)*NumTests) 1316

V[index,index]=Vblk # construct the covariance matrix first 1317

} 1318

V = V/V[1,1] 1319

return(V) 1320

} 1321

##### End of function 1322

1323

L = 2 # Number of labs 1324

K = 2 # Number of tests in each of the labs 1325

TotTests = K*L 1326

R _Pa = GenMatrix(K,L) 1327

alphaK L = pmvt(lower=rep(t1 _Pa

, TotTests ),upper=rep(Inf, TotTests ),df=df 1 _Pa ,corr=R _Pa ) 1328

betaKL = 1

- pmvt(lower=rep(t1_Pa, TotTests),upper=rep(Inf,TotTests),df=df1_Pa,corr=R_Pa,delta=lambda1_Pa) 1329

1330

When a performance standard requires that all tests from both P. aeruginosa and S. 1331

aureus must be passed, the error rate calculations via equations (17) and (18) look similar 1332

as for the single microbe case, but the correlation matrix is more complicated. For 1333

example, the two error rates for two tests of each microbe performed in each of two labs 1334

are calculated by: 1335

1336

##### An R function to g enerate the correlation matrix for tests of two microbes 1337

GenMatrix_Microbe<

- 1338

function(NumTests,NumLabs,CorrMicrobe,VarLab =c(.175,0),VarExp=c(.111,.1) ) 1339

{ 1340

# NumTests is a 2x1 vector, NumTests[1] s p e cifies number of tests for Pa in each lab 1341

# NumTests[2] sepcifies number of tes ts for Sa in each lab 1342

# 1343

# VarLab is a 2x1 vector, VarLab[1] is for Pa, VarLab[2] is for Sa 1344

# VarExp is a 2x1 vector, VarExp[1] is for Pa, VarExp[2] is for Sa 1345

PaBlock=GenMatrix(NumTests[1],1,VarLab[ 1],VarExp[1] ) 1346

SaBlock=GenMatrix(NumTests[2],1,VarLab[ 2],VarExp[2] ) 1347

CovMicrobe = CorrMicrobe*sqrt((VarLab[1] + VarExp[1])*(VarLab[2] + VarExp[2])) 1348

Vblk = matrix(0,sum(NumTests),sum(NumTests)) 1349

index1=1:NumTests[1] 1350

index2=(NumTests[1]+1):sum(NumTests) 1351

Vblk[index1,index1]=PaBlock 1352

Vblk[index2,index2]=SaBlock 1353

Vblk[index1,index2]=CorrMicrobe 1354

Vblk[index2,index1]=CorrMicrobe 1355

N = sum(NumTests)*NumLabs # total number of data points across all labs and exps 1356

V=matrix(0,N,N) 1357

for (i in 0:(NumLabs

- 1)) 1358

{ 1359

index=(i*sum(NumTests)+1):((i+1)*sum(NumTests)) 136035 V[index,index]=Vbl k 1361

} 1362

return(V) 1363

} 1364

##### End of function 1365

1366

L = 2 # Number of labs 1367

K _Pa = 2 # Number of P. aeruginosa tests in each of the labs 1368

K_Sa = 2 # Number of S. aureus tests in each of the labs 1369

R = GenMatrix_Microbe(c(K_Pa,K_Sa),L,0.25) 1370

vecblk = c(rep(t1 _Pa

, K_Pa ),rep(t 1_ Sa ,K_Sa )) 1371

lower.vec = rep(vecblk,L ) 1372

deltablk = c(rep( lambda1_Pa

, K_Pa ),rep( lambda1_Sa

, K_Sa )) 1373

delta.vec = rep(deltablk,L ) 1374

Tot Tests = (K_Pa + K_Sa) * L 1375

alpha KL = pmvt(low er=lower.vec,upper=rep(Inf,Tot Tests),df= df 1 _Pa ,corr=R) 1376

beta KL = 1

- pmvt(lower=lower.vec,upper=rep(Inf,TotTests),df= df 1 _Pa, corr=R,delta=delta.vec) 1377

1378

The error rates for a performance standard on the average over K multiple tests performed 1379

at each of L labs are calculated using equations (15)

, (16)

, (21) and (22) : 1380

1381

PSmean = 1 # This corresponds to the current UDM performance standard 1382

L = 3 # Number of labs required by the performance standard 1383

K = 10 # Number of future tests in each lab 1384

S2_lab = 0.175 # For P. aeruginosa , use S2_lab = 0 .000 for S. aureus 1385

S2_test = 0.111 # For P. aeruginosa , use S2_ test = 0.100 for S. aureus 1386

dfcollab = 3.06 1387

SEmean = sqrt(S2_lab /L + S2_test /(L*K)) 1388

dfmean = dfcollab + L

- 1 1389

LR.PSmean = GetMPNLR(PSmean) 1390

LR.PSmeanplus1 = GetMPNLR(PSmean+1) 1391

tmean = (LR.PSmean

- LR.PSmeanplus1)/SEmean 1392

alpha_mean = 1

- pt(tmean,dfmean) 1393

lambda_mean = (LR.0

- LR.PSmeanplus1)/SEmean 1394

beta_mean = pt(tmean,dfmean,ncp=lambda

_ mean) 1395

1396