2011 UDM/OECD Technical Workshop

Reassessment of the UDM 

Performance Standard

For Official Use Only/Do not Cite or Distribute 1

Stephen F. Tomasino, Ph.D.

U.S. Environmental Protection Agency

Biological and Economic Analysis Division

Office of Pesticide Programs

Microbiology Laboratory
2011 UDM/OECD Technical Workshop

Presentation Overview

2

• Overall goals of the 

reassessment 

• 2009 collaborative study and 

associated data

• Basic concepts of the 

statistical analysis 

• Proposal/recommendations

• Path forward

For Official Use Only/Do not Cite or Distribute
2011 UDM/OECD Technical Workshop

Overall Goals of the Reassessment

• Address concerns with the UDM while the Agency evaluates and 

implements a new strategy/methodology 

• Use the best data available and apply the best available statistics

• Increase our confidence in the outcome of the product performance testing 

while maintaining adequate stringency

• Use a bundled, multi-faceted approach 

• Incorporate UDM revisions 

• Align the post registration testing with registration requirements as much as 

possible

• Seek independent technical assessment, e.g., AOAC

• If approved, provide detailed communication and instructions to end-users

• Additional data generation by EPA on the UDM is highly unlikely 

For Official Use Only/Do not Cite or Distribute 3
2011 UDM/OECD Technical Workshop

The Collaborative Study – key aspects

• EPA works with the Efficacy Working Group

• October 2008 – technical survey submitted to labs

– Collect information on how labs currently conduct the UDM

– Identify potential sources of variability 

• Method modifications identified – targeting the preparation of 

test cultures

• Goal to reduce variability and improve the performance of the 

method

• Perform equivalency testing to demonstrate what effect, if any, 

the modifications have on the outcome of the efficacy test.

For Official Use Only/Do not Cite or Distribute 4
2011 UDM/OECD Technical Workshop

• Input from AOAC provided 

• January-March 2009 – EPA leads effort to develop 

the study protocol

• Montana State University contract used to develop 

the data analysis section of the protocol

• Industry provided input to the test design and 

statistical methodology

• August 2009 – collaborative study launched

For Official Use Only/Do not Cite or Distribute 5

The Collaborative Study – key aspects
2011 UDM/OECD Technical Workshop

• Use of MLB’s SOP MB-05 as the current method

– The modified version included all proposed modifications (i.e., 

each modification was not evaluated separately)

• The modifications (MLB SOP MB-05 and the following):

– Use of frozen stock cultures instead of stock cultures grown and 

stored on agar slants and stabs

– Reduce the number of 24 hr subculture periods from 3 to 1 

– Use of an agar-based procedure to replace the current liquid 

culture protocol. 

For Official Use Only/Do not Cite or Distribute 6

The Collaborative Study – key aspects
2011 UDM/OECD Technical Workshop

• 5 representative laboratories

• Two microbes - S. aureus and P. aeruginosa

• One quaternary ammonium product

• Hard water and organic burden included

• A mean minimum level of 6 logs per carrier – test 

cultures for the modified version were adjusted (diluted) 

based on optical density of the broth from the current 

method

• One subculture tube (neutralizer) used for recovery per 

carrier

For Official Use Only/Do not Cite or Distribute 7

The Collaborative Study – key aspects
2011 UDM/OECD Technical Workshop

• The study design called for testing the product 

at two levels of efficacy (high and low) with 

the current and modified UDM in a side-byside fashion.

• Three replications (test days)

• 24 60-carrier tests per lab

• Order of microbe randomized

For Official Use Only/Do not Cite or Distribute 8

The Collaborative Study – key aspects
2011 UDM/OECD Technical Workshop

UDM Test Cultures

Pseudomonas

Staphylococcus Pseudomonas pellicle

For Official Use Only/Do not Cite or Distribute
2011 UDM/OECD Technical Workshop

10

Microbe Method Mean LD Sr SR

P. aeruginosa Current 6.71 0.25 0.38

P. aeruginosa Modified 6.68 0.36 0.37

S. aureus Current 6.64 0.16 0.30

S. aureus Modified 6.73 0.14 0.24

Statistical summary of the control counts 

For Official Use Only/Do not Cite or Distribute

The Collaborative Study – data
2011 UDM/OECD Technical Workshop

Control carrier log densities (mean +/- 1 SEM)

For P.a., SEM = 0.14; for S.a., SEM = 0.12.

The Collaborative Study – data

For Official Use Only/Do not Cite or Distribute
2011 UDM/OECD Technical Workshop

12

Laboratory Replication Current Method Modified Method

110

1 0 (Pass) 10 (Fail)

2 0 (Pass) 11(Fail)

3 0 (Pass) 3 (Fail)

112

1 1 (Pass) 1 (Pass)

2 1 (Pass) 2 (Fail)

3 1 (Pass) 1 (Pass)

114

1 3 (Fail) 1 (Pass)

2 2 (Fail) 2 (Fail)

3 1 (Pass) 0 (Pass)

116

1 1 (Pass) 0 (Pass)

2 2 (Fail) 2 (Fail)

3 1 (Pass) 1 (Pass)

119

1 5 (Fail) 4 (Fail)

2 1 (Pass) 5 (Fail)

3 1 (Pass) 3 (Fail)

11/15 Pass 6/15 Pass

Number of positive carriers and Pass/Fail outcome for the P. 

aeruginosa/high QAC efficacy treatment – the label claim

For Official Use Only/Do not Cite or Distribute

The Collaborative Study – data
2011 UDM/OECD Technical Workshop

The Collaborative Study – data

13

Efficacy

Method

Lab

Low High

Current Modified Current Modified

110 112 114 116 119 110 112 114 116 119 110 112 114 116 119 110 112 114 116 119

60

50

40

30

20

10

0

Number of positive carriers

18.8

26.3

1.3 3.1

P. aeruginosa
2011 UDM/OECD Technical Workshop

14

Lab

Method

Efficacy

110 112 114 116 119

Current Modified Current Modified Current Modified Current Modified Current Modified

Low High Low High Low High Low High Low High Low High Low High Low High Low High Low High

60

50

40

30

20

10

0

Number of positive carriers

Responsiveness for P. aeruginosa where each point represents 

the number of positive carriers from a single test per method.

For Official Use Only/Do not Cite or Distribute

The Collaborative Study – data
2011 UDM/OECD Technical Workshop

The Collaborative Study – conclusions

• Based on the Srand SR, the control Test LDs for P. aeruginosa were slightly 

more variable compared to S. aureus; however, both methods exhibited 

acceptable and small Sr and SR.

• Across microbes and methods, the highest degree of variability was noted 

at the low efficacy level – this was not an unexpected result. 

• For P. aeruginosa, the mean number of positive carriers for the low 

efficacy treatment was 18.8 and 26.3 for the current and modified methods, 

respectively. 

• In contrast, for P. aeruginosa, the high efficacy treatment yielded a mean of 

1.3 positives for the current method and 3.1 positives for the modified 

method.

• With the exception of laboratory 116 for P. aeruginosa, both methods 

exhibited the ability to detect the presumed efficacy-response relationship.

15
2011 UDM/OECD Technical Workshop

The Collaborative Study – conclusions

• The Srand SR were less for S. aureus tests than for the 

corresponding P. aeruginosa tests.

• For each combination of microbe and efficacy level, 

both methods produced acceptably small Srand SR

values for LR.

• Across the combinations of microbe and efficacy level

– the Sr values ranged from 0.15 to 0.33 for the current 

method and from 0.32 to 0.50 for the modified method

– the SR values ranged from 0.32 to 0.55 for the current 

method and from 0.39 to 1.10 for the modified method. 

• Overall, the method modifications did not reduce 

method variability.

16
2011 UDM/OECD Technical Workshop

For Official Use Only/Do not Cite or Distribute

The Collaborative Study – further analysis

High Efficacy Treatment
2011 UDM/OECD Technical Workshop

Carrier counts

in combination 

with the 

number of 

positives 

log reduction 

(LR)

For reassessment of the performance standard, conversion of number 

of positives to log reduction values – use of the MPN formula

Use of the MPN 

formula 

For Official Use Only/Do not Cite or Distribute
2011 UDM/OECD Technical Workshop

Use of the MPN formula

• Widely used by FDA – see Blodgett, R.J., (2006) Most 

Probable Number Determination from Serial Dilutions. 

FDA’s Bacteriological Analytical Manual 

• Used by EPA in the past – see Tomasino and Hamilton 

(2006). Modification to the AOAC Sporicidal Activity 

of Disinfectants Test: Collaborative Study. JAOAC Int. 

89:1373-1397

• See MSU’s “Knowledge Sharing Article” KSA-SM-08 

for complete details on the P/N formula at 

http://www.biofilm.montana.edu/content/ksa-sm-08 

For Official Use Only/Do not Cite or Distribute 19
2011 UDM/OECD Technical Workshop

within-lab variability

Identifying sources of variability

For Official Use Only/Do not Cite or Distribute
2011 UDM/OECD Technical Workshop

Among-lab variability

Identifying sources of variability

For Official Use Only/Do not Cite or Distribute
2011 UDM/OECD Technical Workshop

Within-lab variability

Among-lab variability

Reproducibility SD = 0.5348 

Identifying sources of variability

Using an assessment of 

variability, we can calculate 

the probability of events 

through modeling 

For Official Use Only/Do not Cite or Distribute
2011 UDM/OECD Technical Workshop

Important Questions to Ask

• How often does a 

marginally ineffective 

product pass?

– using the current

performance standard

– using a new performance 

standard

– in combination with the 

number of tests 

conducted 

• How often does a highly 

effective product fail?

– using the current

performance standard

– using a new performance 

standard

– in combination with the 

number of tests 

conducted 

For Official Use Only/Do not Cite or Distribute 23
2011 UDM/OECD Technical Workshop

Proposal!

• S. aureus

– 0-3/60 positives is a pass

– > 3/60 positives is a 

failure 

– Must conduct 3 

independent tests; each 

must pass

– Must have a minimum 

of 6 logs of bacteria per 

carrier

• P. aeruginosa

– 0-6/60 positives is a pass

– > 6/60 positives is a 

failure 

– Must conduct 3 

independent tests; each 

must pass

– Must have a minimum 

of 6 logs of bacteria per 

carrier

24

Performance standard for Salmonella enterica will not change. 

For Official Use Only/Do not Cite or Distribute
2011 UDM/OECD Technical Workshop

Overall Recommendations

Increased 

confidence 

in the 

outcome

Revise 

performance 

standards 

Require multiple 

testing of the 

same product

Employ other 

enhancements to 

the UDM such as 

the min/max 

control counts

For Official Use Only/Do not Cite or Distribute 25
2011 UDM/OECD Technical Workshop

AOAC approval and outreach

• Two manuscripts will be prepared:

– First paper: Collaborative data and method performance

• Submit to JAOAC by 01/31/2012

– Second paper: Statistical model and the reassessment of the 

performance standard 

• Submit to JAOAC by 02/29/2012

– Independent panel of experts, likely from AOAC 

Committee M, will be assigned for technical review

– Follow the official AOAC method revision process 

• Concurrently, seek comment from stakeholders and 

other federal agencies (e.g., CDC)

For Official Use Only/Do not Cite or Distribute 26
2011 UDM/OECD Technical Workshop

QUESTIONS/COMMENTS 

For Official Use Only/Do not Cite or Distribute 27