# Center for Biofilm Engineering 

# Statistical results: 5 lab collaborative study for comparing “current” & “modified” UDM methods
Statisticians : Al Parker, CBE/MSU; G. Cord Hamilton & Marty Hamilton, BSSA

1 AOAC Committee M 27 September 2010 

# MSU co
- authors: Darla Goeres & Diane Walker, CBE/MSU2 

# DISCLAIMER This information is distributed solely for the purpose of pre
- dissemination peer review under applicable information quality guidelines. It has not been formally disseminated by EPA. It does not represent and should not be construed to represent any Agency Determination or Policy . The information provided should not be cited or referenced.

# Acknowledgments

3 

# Support: EPA Contract #EP
- W
- 08
- 014 with Montana State University (MSU) and MSU Sub

- contract G220
- 10
- W2933 with Big Sky Statistical Analysts, LLC EPA/OPP/BEAD Microbiology Laboratory: Steve Tomasino, Senior Scientist Mark Rindal , Contract Officer Luisa C. Samalot
- Freire, Microbiologist

# Study design

4 

# 3 test days (repeats) for each microbe in each laboratory, leading to a total of 24 tests in each of the 5 laboratories 4 tests (2 methods, each applied to 2 efficacy levels) were conducted side

- by
- side on one test day in a laboratory

# Purposes of study

5 

# A “Modified” version of the 60 carrier Use Dilution Method (UDM) was proposed as an alternative to the “Current” version The study was conducted to: evaluate each of the methods (Modified and Current) with respect to specified attributes test the equivalency of the two methods with respect to specified characteristics

# Study factors (levels)

6 

# Lab (110, 112, 114, 116, 119) Microbe : Staphylococcus aureus & Pseudomonas aeruginosa (Sa and Pa) Efficacy level (Low & High) Method (Current & Modified)

Replicate (1, 2, 3)

# Schedule for a laboratory

7 

## 240 treated coupons and 6 to 12 control coupons per test day

**Microbe** Test Day 1 (Rep #1) Test Day 2 (Rep #2) Test Day 3 (Rep #3) S. aureus High efficacy High efficacy High efficacy Test 1: Current Test 1: Current Test 1: Current Test 2: Modified Test 2: Modified Test 2: Modified

Low efficacy Low efficacy Low efficacy Test 3: Current Test 3: Current Test 3: Current Test 4: Modified Test 4: Modified Test 4: Modified Test Day 4 (Rep #1) Test Day 5 (Rep #2) Test Day 6 (Rep #3) P. aeruginosa High efficacy High efficacy High efficacy Test 1: Current Test 1: Current Test 1: Current Test 2: Modified Test 2: Modified Test 2: Modified

Low efficacy Low efficacy Low efficacy Test 3: Current Test 3: Current Test 3: Current Test 4: Modified Test 4: Modified Test 4: Modified

# Control carrier counts

8 

# viable cell enumerations on 3 untreated (control) carriers for each method on each test day; often two separate 3
- carrier enumerations were done during a test day concurrent control carrier enumerations make the UDM a semi
- quantitative test and allow efficacy to be measured with a log reduction value (via the P/N LR formula)

# Data inspection

9 

# No outliers were identified, no observations were rejected Statisticians inspected the data for inconsistent or unusual values; the main tools were careful reading, cross - tabulation tables, and individual value plots Some minor, correctable errors (mainly pertaining to an occasional incorrect date for a test day) were discovered and corrected; questions about individual data points were resolved by the EPA QA Officer
Throughout the analysis, diagnostic plots were constructed and data checks were conducted

# Statistical analysis: focused on desirable attributes of a disinfectant test

10 

# Responsiveness
– the test method correctly detects increased efficacy Repeatability
– The efficacy result is about the same among repeated tests within a lab Reproducibility
– The efficacy result is about the same among labs Resemblance
- control carriers hold about the same, suitable number of bacteria

# Three efficacy response variables

11 

# “Number of positive carriers” among the 60 treated carriers, a quantitative measure log reduction (“LR”), a quantitative measure based on comparing the number of viable cells on treated carriers to the number on untreated carrier “Pass/Fail” outcome, a binary measure, where Pass resulted if and only if there were 0 or 1 positive carriers among the 60

# Outline

12 

# Control carrier counts Number of positive carriers among 60 treated carriers per test log reduction
– efficacy measure that takes account of the number of positive carriers in comparison to the control carrier counts pass/fail outcome
– the conventional performance standard for the UDM

# 1. Resemblance
- analysis of viable cell counts on control carriers

13

# Nomenclature: control carrier data

14 

# LD denotes the log 10 

# - transformed density for a control carrier Density (CFU/carrier), recorded for each carrier in the set of three control carriers for each test (for Test 1 & Test 2 every test day, sometimes for Test 3 & Test 4 also)

In this presentation, all resemblance analyses are based on the Test LD values Test LD denotes the mean LD for the set of 3 control carriers (carrier counts) in a test

# List of resemblance analyses

15 

# Estimates of mean Test LD , averaged over all tests for each microbe and method Variance components, S r 

# and S R 

# of Test LD Plots of LD and Test LD values

# Control Test LD (carrier counts); each point is for one test

16 M i c r o b e M e t h o d L a b Sa Pa M o d i f i e d C u r r e n t M o d i f i e d C u r r e n t 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 7 . 6
7 . 2
6 . 8
6 . 4
6 . 0ControlTestLD

# Mean control Test LD (carrier counts)

17 Mean **Test LD** 95% Confidence Interval Microbe Method SEM lower limit upper limit Pa Current 6.71 0.14 6.31 7.10 Pa Modified 6.68 0.09 6.43 6.92 Sa Current 6.64 0.12 6.31 6.97 Sa Modified 6.73 0.09 6.47 6.99

# Variability results for Test LD (carrier counts); previous results (Tomasino et al., 2009)

18 

## Microbe Method Mean S r* 

## S R ** 

## Pa Current 6.71 0.25 0.38 Pa Modified 6.68 0.36 0.37 Sa Current 6.64 0.16 0.30 Sa Modified 6.73 0.14 0.24 Results from Tomasino et al. (2009) 

## Pa Current 6.65 0.22 0.29 Sa Current 6.63 0.24 0.32

* 

### S r 

### = repeatability standard deviation (within - lab variability) of the Test LD ** 

### S R 

### = reproducibility standard deviation (total variability) of the Test LD

# 2. Analysis of the “number of positive carriers”

19

# List of analyses based on the number of positive carriers

20 

# Responsiveness of the number of positive carriers Plots of observed values Mean number of positive carriers, averaged over all tests for each microbe and method

### 21 

# Number of positive carriers for Pa (line is the mean, averaged across labs)

### 22 

# Number of positive carriers for Sa (line is the mean, averaged across labs)

### 23 

# Responsiveness of the “number of positive carriers” for Pa (lines connect the means for 3 tests)

For each Method, responsiveness = mean (Low Efficacy No. Pos.
– High Efficacy No. Pos.) is highly statistically significantly positive (p < 0.001).

### 24 

# Responsiveness of the “number of positive carriers” for Sa (lines connect the means for 3 tests)

For each Method, responsiveness = mean (Low Efficacy No. Pos.
– High Efficacy No. Pos.) is highly statistically significantly positive (p < 0.001).

# 3. Analysis of the log reductions (LR)

25

# log reduction (LR) efficacy

26 

# All conventional attributes, including the repeatability and reproducibility standard deviations, can be calculated for LR LR is the mean log density on the treated carriers subtracted from the mean log density on the control carriers for the same test; it is the conventional efficacy measurement A larger LR indicates higher disinfectant efficacy

# List of P/N LR analyses

27 

# Variance components, S r 

# , and S R 

# of P/N LR for each microbe, efficacy level, and method Plots of P/N LR values Calculation of the mean P/N LR for each microbe, efficacy level and method Responsiveness

# P/N LR for Pa (each point is for one test)

28 Eff i ca cy l e ve l M e t h o d L a b H i g h L ow M o d i f i e d C u r re n t M o d i f i e d C u r re n t 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 9 . 5 9 . 0 8 . 5 8 . 0 7 . 5 7 . 0 6 . 5 6 . 0P/NLR P a

### 29 

# P/N LR for Sa (each point is for one test)

Eff i ca cy l e ve l M e t h o d L a b H i g h L ow M o d i f i e d C u r re n t M o d i f i e d C u r re n t 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 9 . 5 9 . 0 8 . 5 8 . 0 7 . 5 7 . 0 6 . 5 6 . 0P/NLR S a

# Mean P/N LR

30 Efficacy level Mean P/N LR 95% Confidence Interval Microbe Method SEM lower limit upper limit Pa Low Current 7.25 0.24 6.58 7.92 Pa Low Modified 7.11 0.47 5.81 8.40 Pa High Current 8.26 0.21 7.69 8.84 Pa High Modified 8.09 0.15 7.66 8.52 Sa Low Current 7.79 0.13 7.44 8.15 Sa Low Modified 8.00 0.13 7.65 8.35 Sa High Current 8.48 0.08 8.25 8.70 Sa High Modified 8.49 0.13 8.12 8.85

# Methods comparison: mean P/N LR

31 P/N LR Difference: Current - Modified Efficacy level Mean P/N LR 90% Confidence Interval Microbe Current Modified Mean lower limit upper limit 

### Pa Low 7.25 7.11 0.15 - 0.60 0.90 Pa High 8.26 8.09 0.17 - 0.35 0.70 Sa Low 7.79 8.00 - 0.21 - 0.48 0.07 Sa High 8.48 8.49 - 0.01 - 0.25 0.23

# Responsiveness of P/N LR for Pa (lines connect the means of 3 tests)

32 M e t h o d L a b Ef f i ca cy l e ve l M o d i f i e d C u r re n t 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 H L H L H L H L H L H L H L H L H L H L 9 . 5 9 . 0 8 . 5 8 . 0 7 . 5 7 . 0 6 . 5 6 . 0P/NLR 

### P a

For each Method, responsiveness = mean (High Efficacy LR
– Low Efficacy LR) is highly statistically significantly positive (p < 0.001).

### 33 

# Responsiveness of P/N LR for Sa (lines connect the means of 3 tests)

M e t h o d L a b Ef f i ca cy l e ve l M o d i f i e d C u r re n t 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 1 1 9 1 1 6 1 1 4 1 1 2 1 1 0 H L H L H L H L H L H L H L H L H L H L 9 . 5 9 . 0 8 . 5 8 . 0 7 . 5 7 . 0 6 . 5 6 . 0P/NLR 

### S a

For each Method, responsiveness = mean (High Efficacy LR
– Low Efficacy LR) is highly statistically significantly positive (p < 0.001).

# Overall mean responsiveness of LR

34 H i g h L o w 8 . 6 8 . 4 8 . 2 8 . 0 7 . 8
7 . 6 7 . 4
7 . 2 7 . 0 Eff i c a cy l eve lMeanP/NLR

Cu r r e n t M

o d i f i e d M

e t

h o d

H i g h L o w 8 . 6 8 . 4 8 . 2 8 . 0 7 . 8
7 . 6 7 . 4
7 . 2 7 . 0 Eff i c a cy l eve lMeanP/NLR

Cu r r e n t M

o d i f i e d M

e t

h o d Pa Sa

# Variability results for P/N LR

35 For each method, all repeatability and reproducibility standard deviations are acceptably small The observed repeatability and reproducibility standard deviations are slightly smaller for the Current method than for the Modified method 

### Efficacy Variances Microbe level Method V Rep (Lab) 

### V Lab 

### S r* 

### S R ** 

### Pa Low Current 0.023 0.284 0.15 0.55 Pa Low Modified 0.180 1.025 0.42 1.10 Pa High Current 0.111 0.175 0.33 0.53 Pa High Modified 0.245 0.037 0.50 0.53 Sa Low Current 0.070 0.059 0.26 0.36 Sa Low Modified 0.116 0.041 0.34 0.40 Sa High Current 0.100 0.000 0.32 0.32 Sa High Modified 0.103 0.050 0.32 0.39 * S r = repeatability standard deviation (within - lab variability) of the P/N LR
** S R = reproducibility standard deviation (total variability) of the P/N LR

# 4. Analysis of the Pass/Fail outcomes

36

# List of analyses based on the Pass/Fail outcome

37 

# Responsiveness of the number of positive carriers Plots to describe the Pass/Fail observations Mean percentage of tests producing Pass for each efficacy level, microbe, and method

# Pass/Fail outcome: “Pass” if the number of positive carriers out of 60 is either 0 or 1; i.e., the point is below the line at 1.5 (percentage Pass, averaged across labs)

38 

## (0/15 = 0.0%) (2/15 =13.3%) (11/15 = 73.3%) (6/15 = 40.0%)

### 39 

# Number of positive carriers for Pa

(line is the mean, averaged across 15 tests)

### 40 

# Pass/Fail concordance or discordance per test concordant pairs discordant pairs

Microbe Efficacy Current Method Pass % Modified Method Pass % Both Tests Pass

(a) Current Fails, Modified Fails

(d) Current Passes, Modified Fails

(b) Current Fails, Modified

Passes (c) Pa High 73% 40% 5 3 6 1 Pa Low 0% 13% 0 13 0 2 Sa High 67 % 80% 10 3 0 2 Sa Low 20% 13% 0 10 3 2

For each Microbe, both methods were highly statistically significantly responsive (p < 0.001).

# Conclusions: Control carrier counts

41 

# The two methods produced similar overall means of Test LD For both methods, Test LD was more variable for Pa than for Sa. Both methods exhibited small repeatability and reproducibility SDs for Test LD

# Conclusions: efficacy outcomes

42 

# Over the 15 Current method tests of the High efficacy level, the mean number of positives (1.3 for Pa & 0.9 for Sa) could be considered a Pass. Both methods were responsive The two methods produced similar efficacy outcomes on the average The P/N LR values for both methods exhibited small repeatability & reproducibility SDs relative to other standardized disinfectant test methods The observed repeatability and reproducibility SDs for P/N LR were smaller for the Current than for the Modified

# The End

43