Center for Biofilm Engineering

Statistical results: 5 lab 

collaborative study for 

comparing “current” & 

“modified” UDM methods

Statisticians: Al Parker, CBE/MSU; 

G. Cord Hamilton & Marty Hamilton, BSSA

1

AOAC Committee M 27 September 2010

MSU co-authors: Darla Goeres & 

Diane Walker, CBE/MSU
2

DISCLAIMER

This information is distributed solely for the 

purpose of pre-dissemination peer review under 

applicable information quality guidelines. It has 

not been formally disseminated by EPA. It does 

not represent and should not be construed to 

represent any Agency Determination or Policy.

The information provided should not be cited or 

referenced.
Acknowledgments 

3

Support:

EPA Contract #EP-W-08-014 with Montana State 

University (MSU) and MSU Sub-contract G220-10-

W2933 with Big Sky Statistical Analysts, LLC

EPA/OPP/BEAD Microbiology Laboratory:

Steve Tomasino, Senior Scientist

Mark Rindal, Contract Officer

Luisa C. Samalot-Freire, Microbiologist
Study design

4

3 test days (repeats) for each microbe in each 

laboratory, leading to a total of 24 tests in each of 

the 5 laboratories

4 tests (2 methods, each applied to 2 efficacy 

levels) were conducted side-by-side on one test day 

in a laboratory
Purposes of study

5

A “Modified” version of the 60 carrier Use 

Dilution Method (UDM) was proposed as an 

alternative to the “Current” version 

The study was conducted to:

evaluate each of the methods (Modified and 

Current) with respect to specified attributes

test the equivalency of the two methods with 

respect to specified characteristics 
Study factors (levels)

6

Lab (110, 112, 114, 116, 119)

Microbe: Staphylococcus aureus & 

Pseudomonas aeruginosa (Sa and Pa)

Efficacy level (Low & High)

Method (Current & Modified)

Replicate (1, 2, 3)
Schedule for a laboratory

7

240 treated coupons and 6 to 12 control coupons per test day

Microbe Test Day 1 (Rep #1) Test Day 2 (Rep #2) Test Day 3 (Rep #3)

S. aureus High efficacy High efficacy High efficacy

Test 1: Current Test 1: Current Test 1: Current

Test 2: Modified Test 2: Modified Test 2: Modified

Low efficacy Low efficacy Low efficacy

Test 3: Current Test 3: Current Test 3: Current

Test 4: Modified Test 4: Modified Test 4: Modified 

Test Day 4 (Rep #1) Test Day 5 (Rep #2) Test Day 6 (Rep #3) 

P. aeruginosa High efficacy High efficacy High efficacy

Test 1: Current Test 1: Current Test 1: Current

Test 2: Modified Test 2: Modified Test 2: Modified

Low efficacy Low efficacy Low efficacy

Test 3: Current Test 3: Current Test 3: Current

Test 4: Modified Test 4: Modified Test 4: Modified 
Control carrier counts

8

viable cell enumerations on 3 untreated (control) 

carriers for each method on each test day; often 

two separate 3-carrier enumerations were done 

during a test day

concurrent control carrier enumerations make 

the UDM a semi-quantitative test and allow 

efficacy to be measured with a log reduction 

value (via the P/N LR formula)
Data inspection 

9

No outliers were identified, no observations were 

rejected

Statisticians inspected the data for inconsistent or 

unusual values; the main tools were careful reading, 

cross-tabulation tables, and individual value plots 

Some minor, correctable errors (mainly pertaining to 

an occasional incorrect date for a test day) were 

discovered and corrected; questions about individual 

data points were resolved by the EPA QA Officer

Throughout the analysis, diagnostic plots were 

constructed and data checks were conducted
Statistical analysis: focused on desirable 

attributes of a disinfectant test 

10

Responsiveness – the test method 

correctly detects increased efficacy

Repeatability – The efficacy result is about 

the same among repeated tests within a lab

Reproducibility – The efficacy result is 

about the same among labs 

Resemblance - control carriers hold about 

the same, suitable number of bacteria
Three efficacy response variables 

11

“Number of positive carriers” among the 60 

treated carriers, a quantitative measure

log reduction (“LR”), a quantitative measure 

based on comparing the number of viable 

cells on treated carriers to the number on 

untreated carrier

“Pass/Fail” outcome, a binary measure, 

where Pass resulted if and only if there were 

0 or 1 positive carriers among the 60
Outline 

12

Control carrier counts

Number of positive carriers among 60 

treated carriers per test

log reduction – efficacy measure that takes 

account of the number of positive carriers in 

comparison to the control carrier counts

pass/fail outcome – the conventional 

performance standard for the UDM
1. Resemblance - analysis of viable 

cell counts on control carriers

13
Nomenclature: control carrier data

14

LD denotes the log10-transformed density for 

a control carrier

Density (CFU/carrier), recorded for each 

carrier in the set of three control carriers for 

each test (for Test 1 & Test 2 every test day, 

sometimes for Test 3 & Test 4 also)

In this presentation, all resemblance 

analyses are based on the Test LD values

Test LD denotes the mean LD for the set of 

3 control carriers (carrier counts) in a test
List of resemblance analyses

15

Estimates of mean Test LD, averaged over 

all tests for each microbe and method

Variance components, Sr and SR of Test LD

Plots of LD and Test LD values
Control Test LD (carrier counts);

each point is for one test

16

Microbe

Method

Lab

Pa Sa

Current Modified Current Modified

110 112 114 116 119 110 112 114 116 119 110 112 114 116 119 110 112 114 116 119

7.6

7.2

6.8

6.4

6.0

C

o

ntrol T

e

st L

D
Mean control Test LD (carrier counts)

17

Mean

Test LD

95% Confidence Interval

Microbe Method SEM lower limit upper limit

Pa Current 6.71 0.14 6.31 7.10

Pa Modified 6.68 0.09 6.43 6.92

Sa Current 6.64 0.12 6.31 6.97

Sa Modified 6.73 0.09 6.47 6.99
Variability results for Test LD (carrier counts);

previous results (Tomasino et al., 2009)

18

Microbe Method Mean Sr

* SR**

Pa Current 6.71 0.25 0.38

Pa Modified 6.68 0.36 0.37

Sa Current 6.64 0.16 0.30

Sa Modified 6.73 0.14 0.24

Results from Tomasino et al. (2009)

Pa Current 6.65 0.22 0.29

Sa Current 6.63 0.24 0.32

* Sr = repeatability standard deviation (within-lab variability) of the Test LD

**SR = reproducibility standard deviation (total variability) of the Test LD
2. Analysis of the 

“number of positive carriers” 

19
List of analyses based on the number 

of positive carriers

20

Responsiveness of the number of positive 

carriers

Plots of observed values

Mean number of positive carriers, averaged 

over all tests for each microbe and method
21

Number of positive carriers for Pa

(line is the mean, averaged across labs)
22

Number of positive carriers for Sa

(line is the mean, averaged across labs)
23

Responsiveness of the “number of 

positive carriers” for Pa

(lines connect the means for 3 tests)

For each Method, responsiveness = mean (Low Efficacy No. Pos. – High Efficacy No. Pos.) 

is highly statistically significantly positive (p < 0.001).
24

Responsiveness of the “number of 

positive carriers” for Sa

(lines connect the means for 3 tests)

For each Method, responsiveness = mean (Low Efficacy No. Pos. – High Efficacy No. Pos.) 

is highly statistically significantly positive (p < 0.001).
3. Analysis of the log reductions (LR)

25
log reduction (LR) efficacy

26

All conventional attributes, including the 

repeatability and reproducibility standard 

deviations, can be calculated for LR

LR is the mean log density on the treated 

carriers subtracted from the mean log density 

on the control carriers for the same test; it is 

the conventional efficacy measurement

A larger LR indicates higher disinfectant efficacy
List of P/N LR analyses

27

Variance components, Sr , and SR of 

P/N LR for each microbe, efficacy level, 

and method

Plots of P/N LR values

Calculation of the mean P/N LR for each 

microbe, efficacy level and method

Responsiveness
P/N LR for Pa

(each point is for one test)

28

Efficacy level

Method

Lab

Low High

Current Modified Current Modified

110 112 114 116 119 110 112 114 116 119 110 112 114 116 119 110 112 114 116 119

9.5

9.0

8.5

8.0

7.5

7.0

6.5

6.0

P/N

L

R

Pa
29

P/N LR for Sa

(each point is for one test)

Efficacy level

Method

Lab

Low High

Current Modified Current Modified

110 112 114 116 119 110 112 114 116 119 110 112 114 116 119 110 112 114 116 119

9.5

9.0

8.5

8.0

7.5

7.0

6.5

6.0

P/N

L

R

Sa
Mean P/N LR

30

Efficacy

level

Mean

P/N LR

95% Confidence Interval

Microbe Method SEM lower limit upper limit

Pa Low Current 7.25 0.24 6.58 7.92

Pa Low Modified 7.11 0.47 5.81 8.40

Pa High Current 8.26 0.21 7.69 8.84

Pa High Modified 8.09 0.15 7.66 8.52

Sa Low Current 7.79 0.13 7.44 8.15

Sa Low Modified 8.00 0.13 7.65 8.35

Sa High Current 8.48 0.08 8.25 8.70

Sa High Modified 8.49 0.13 8.12 8.85
Methods comparison: mean P/N LR

31

P/N LR Difference: Current - Modified

Efficacy

level

Mean P/N LR 90% Confidence Interval

Microbe Current Modified Mean lower limit upper limit

Pa Low 7.25 7.11 0.15 -0.60 0.90

Pa High 8.26 8.09 0.17 -0.35 0.70

Sa Low 7.79 8.00 -0.21 -0.48 0.07

Sa High 8.48 8.49 -0.01 -0.25 0.23
Responsiveness of P/N LR for Pa

(lines connect the means of 3 tests)

32

Method

Lab

Efficacy level

Current Modified

110 112 114 116 119 110 112 114 116 119

L H L H L H L H L H L H L H L H L H L H

9.5

9.0

8.5

8.0

7.5

7.0

6.5

6.0

P

/

N

L

R

Pa

For each Method, responsiveness = mean (High Efficacy LR – Low Efficacy 

LR) is highly statistically significantly positive (p < 0.001).
33

Responsiveness of P/N LR for Sa

(lines connect the means of 3 tests)

Method

Lab

Efficacy level

Current Modified

110 112 114 116 119 110 112 114 116 119

L H L H L H L H L H L H L H L H L H L H

9.5

9.0

8.5

8.0

7.5

7.0

6.5

6.0

P

/

N

L

R

Sa

For each Method, responsiveness = mean (High Efficacy LR – Low Efficacy 

LR) is highly statistically significantly positive (p < 0.001).
Overall mean responsiveness of LR

34

Low High

8.6

8.4

8.2

8.0

7.8

7.6

7.4

7.2

7.0

Efficacy level

Mean P/N LR

Current

Modified

Method

Low High

8.6

8.4

8.2

8.0

7.8

7.6

7.4

7.2

7.0

Efficacy level

Mean P/N LR

Current

Modified

Method

Pa Sa
Variability results for P/N LR

35

For each method, all repeatability and reproducibility standard deviations are acceptably small

The observed repeatability and reproducibility standard deviations are slightly smaller for the 

Current method than for the Modified method

Efficacy Variances

Microbe level Method VRep(Lab) VLab Sr

* SR**

Pa Low Current 0.023 0.284 0.15 0.55

Pa Low Modified 0.180 1.025 0.42 1.10

Pa High Current 0.111 0.175 0.33 0.53

Pa High Modified 0.245 0.037 0.50 0.53

Sa Low Current 0.070 0.059 0.26 0.36

Sa Low Modified 0.116 0.041 0.34 0.40

Sa High Current 0.100 0.000 0.32 0.32

Sa High Modified 0.103 0.050 0.32 0.39

* Sr = repeatability standard deviation (within-lab variability) of the P/N LR **SR = reproducibility standard deviation (total variability) of the P/N LR
4. Analysis of the Pass/Fail outcomes 

36
List of analyses based on the 

Pass/Fail outcome

37

Responsiveness of the number of positive 

carriers

Plots to describe the Pass/Fail observations

Mean percentage of tests producing Pass for 

each efficacy level, microbe, and method
Pass/Fail outcome: “Pass” if the number of 

positive carriers out of 60 is either 0 or 1; 

i.e., the point is below the line at 1.5

(percentage Pass, averaged across labs)

38

(0/15 = 0.0%) (2/15 =13.3%) (11/15 = 73.3%) (6/15 = 40.0%)
39

Number of positive carriers for Pa

(line is the mean, averaged across 15 tests)
40

Pass/Fail concordance or 

discordance per test

concordant pairs discordant pairs

Microbe Efficacy

Current 

Method 

Pass %

Modified 

Method 

Pass %

Both 

Tests 

Pass

(a)

Current 

Fails, 

Modified 

Fails

(d)

Current 

Passes, 

Modified 

Fails

(b)

Current 

Fails, 

Modified 

Passes

(c)

Pa High 73% 40% 5 3 6 1

Pa Low 0% 13% 0 13 0 2

Sa High 67% 80% 10 3 0 2

Sa Low 20% 13% 0 10 3 2

For each Microbe, both methods were highly statistically significantly responsive (p < 0.001).
Conclusions: Control carrier counts

41

The two methods produced similar overall 

means of Test LD

For both methods, Test LD was more 

variable for Pa than for Sa.

Both methods exhibited small repeatability 

and reproducibility SDs for Test LD
Conclusions: efficacy outcomes

42

Over the 15 Current method tests of the High 

efficacy level, the mean number of positives (1.3 

for Pa & 0.9 for Sa) could be considered a Pass.

Both methods were responsive

The two methods produced similar efficacy 

outcomes on the average

The P/N LR values for both methods exhibited small 

repeatability & reproducibility SDs relative to other 

standardized disinfectant test methods

The observed repeatability and reproducibility SDs 

for P/N LR were smaller for the Current than for the 

Modified
The End

43