U.S. ENVIRONMENTAL PROTECTION AGENCY 

OFFICE OF INSPECTOR GENERAL 

Catalyst for Improving the Environment 

Audit Report 

Measuring and Reporting 

Performance Results for the 

Pollution Prevention Program 

Need Improvement 

Report No. 09-P-0088 

January 28, 2009 
Report Contributors:	 Gail Saunders 

 Anthony Grear 

 Patrick Gilbride 

Abbreviations 

EPA U.S. Environmental Protection Agency 

FY Fiscal Year 

OIG Office of Inspector General 

OMB Office of Management and Budget 

OPPTS Office of Prevention, Pesticides, and Toxic Substances 

P2 Pollution Prevention 

PART Program Assessment Rating Tool 

PIP Program Improvement Plan 

QAPP Quality Assurance Project Plan 

QMP Quality Management Plan 

Cover photo:	 Within the Office of Pollution Prevention and Toxics’ Design for the 

Environment Branch of the Economics, Exposure, and Technology Division, 

the Lead-Free Solder initiative works to phase out the use of lead-based solder 

in electronics. (EPA photo) 
U.S. Environmental Protection Agency 09-P-0088 

January 28, 2009 Office of Inspector General 

At a Glance
 

Catalyst for Improving the Environment 

Why We Did This Review 

The U.S. Environmental 

Protection Agency’s (EPA’s) 

Pollution Prevention (P2) 

Program was rated moderately

effective by the Office of 

Management and Budget 

(OMB) in Fiscal Year 2006,

receiving the third highest

rating awarded to EPA

programs using the 

Program Assessment 

Rating Tool (PART). We 

sought to verify the accuracy

of P2 Program data provided 

for the PART assessment and 

determine what actions have 

been taken to address the 

recommendations in the PART 

program improvement plan. 

Background 

The Pollution Prevention Act 

of 1990 established policy for 

controlling industrial pollution

at its source. EPA uses the P2 

Program to facilitate adoption 

of source reduction techniques 

by businesses, EPA, and other 

federal agencies. EPA uses 

the P2 Program to accomplish 

this. PART is a diagnostic

tool designed to assess the

management and performance 

of federal programs. 

For further information, 

contact our Office of 

Congressional, Public Affairs 

and Management at 

(202) 566-2391. 

To view the full report, 

click on the following link: 

www.epa.gov/oig/reports/2009/ 

20090128-09-P-0088.pdf 

Measuring and Reporting Performance Results for the 

Pollution Prevention Program Need Improvement

 What We Found 

The P2 Program’s data provided in response to the OMB PART assessment 

generally addressed the PART questions and supported the moderately effective 

rating received. However, we noted several weaknesses: 

•	 The P2 Program’s Fiscal Year 2006 PART performance measures were not 

designed to report on the program’s impacts to human health and the 

environment. P2 Program managers believed that reductions in discharges 

and emissions of pollutants represent the best measures that can be supported 

by data obtainable on a program-wide basis and acknowledge that additional 

outcome measures are needed to assess impacts on human health and the 

environment associated with hazardous materials reductions. 

•	 The P2 Program’s verification and validation procedures did not ensure the 

accuracy of performance data. P2 program managers had no assurance that 

performance results data obtained from voluntary partnerships with industry 

and other organizations were accurate. The Program’s Fiscal Year 2006 

performance results were not reported consistently and contain inaccuracies. 

Strengthening data controls would provide P2 managers with improved 

program performance data. 

•	 While the P2 Program has completed several interim PART follow-up 

actions, some of its actions to address its program improvement plan have 

been slow. In addition, the plan did not address all deficiencies identified in 

the PART assessment. 

What We Recommend 

We recommend that EPA continue efforts to develop performance indicators that 

measure impacts on human health and the environment; require the development 

of a P2 Division Quality Assurance Project Plan for data collection and reporting; 

and develop a program improvement plan to address all deficiencies identified in 

the PART assessment. EPA concurred with our recommendations and has 

developed reasonable completion dates for each recommendation. 
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY 

WASHINGTON, D.C. 20460 

OFFICE OF 

INSPECTOR GENERAL 

January 28, 2009 

MEMORANDUM 

SUBJECT:	 Measuring and Reporting Performance Results for the Pollution 

Prevention Program Need Improvement 

 Report No. 09-P-0088 

FROM:	 Melissa M. Heist 

Assistant Inspector General for Audit 

TO: Jim Jones 

Acting Assistant Administrator for the Office of Prevention, Pesticides, 

and Toxic Substances 

This is our report on the U.S. Environmental Protection Agency’s (EPA’s) Pollution Prevention 

Program’s Fiscal Year 2006 Program Assessment Rating Tool audit conducted by EPA’s Office 

of Inspector General (OIG). This report represents our final position on the subjects reported. It 

contains findings that describe the issues the OIG has identified and corrective actions the OIG 

recommends. This report represents the opinion of the OIG and does not necessarily represent 

the final EPA position. Final determinations on matters in this report will be made by EPA 

managers in accordance with established audit resolution procedures. 

The estimated cost of this report – calculated by multiplying the project’s staff days by the 

applicable daily full cost billing rates in effect at the time – is $428,435 

Action Required 

In accordance with EPA Manual 2750, you are required to provide a written response to the 

findings and recommendations in this report within 90 days of the report date. You should 

include a corrective action plan for agreed upon actions, including milestone dates. We have no 

objections to the further release of this report to the public. This report will be available at 

http://www.epa.gov/oig. 

If you or your staff have any questions regarding this report, please contact me at (202) 566-0899; 

or Patrick Gilbride, Director of Risk and Program Performance Audits, at (303) 312-6969 or 

gilbride.patrick@epa.gov. 
Measuring and Reporting Performance Results for the 09-P-0088 

Pollution Prevention Program Need Improvement 

Table of Contents 

Chapters 

1 Introduction .......................................................................................................... 1 

Purpose............................................................................................................. 1 

Background ................................................................................................... 1 

 Noteworthy Achievements.............................................................................. 3 

Scope and Methodology................................................................................. 3 

2 PART Information Supports Rating .................................................................... 5 

Information Supports Rating........................................................................... 5 

Several Weaknesses Noted ........................................................................... 6 

3 Performance Measures Need to Better Address Impacts ................................. 7 

P2 Program Needs Better Performance Measures........................................ 7 

Defining a Universe of P2 Pollutants Difficult ................................................. 9 

Recommendations ......................................................................................... 10 

Agency Response and OIG Comments ........................................................ 11 

4 P2 Program Needs to Strengthen Internal Controls 

over Data Collection.............................................................................................. 12 

Collecting Data from Multiple Sources Challenging......................................... 12 

Inconsistent and Inaccurate Reporting of Results ........................................... 13 

Data Quality Requirements Not Met............................................................... 15 

 Recommendations ........................................................................................ 16 

Agency Response and OIG Comments ........................................................ 16 

5 Program Improvement Plan: Progress Slow and Not All 

 Weaknesses Addressed ....................................................................................... 18
 

Progress to Address PIP Slow ....................................................................... 18 

Additional Program Improvements Needed.................................................... 21

 Recommendations ......................................................................................... 22 

Agency Response and OIG Comments ........................................................ 22 

Status of Recommendations and Potential Monetary Benefits................................. 23 

Appendices 

A EPA’s Hierarchy of Indicators.................................................................................. 24 

B Agency Response to Draft Report........................................................................... 25 

C Distribution .............................................................................................................. 34 
09-P-0088 

Chapter 1

Introduction 

Purpose 

Using the Program Assessment Rating Tool (PART), the Office of Management 

and Budget (OMB) assessed the U.S. Environmental Protection Agency’s 

(EPA’s) Pollution Prevention (P2) Program in Fiscal Year (FY) 2006. OMB 

found the program to be performing moderately effective, awarding the program 

the third highest rating (82 percent) among the 53 EPA programs assessed. Our 

objectives were to: 

•	 verify the accuracy of the P2 Program’s information/data provided in 

response to the PART assessment, and 

•	 determine what actions the P2 Program has taken to address 

recommendations in the program improvement plan. 

Background 

The Pollution Prevention Act of 1990 

The Pollution Prevention Act of 1990 established pollution prevention as the 

national policy reducing industrial pollution at its source – in other words, to keep 

pollutants from entering the environment. EPA, through its P2 Program, works to 

reduce pollution before it occurs by encouraging and supporting innovative 

changes in industrial production and use of raw materials. The following table 

shows FY 2006 actual expenditures and FY 2007 and 2008 President’s Budget 

figures: 

Table 1-1: Pollution Prevention Program President’s Budget (Millions) 

Fund 2006 

Actual 

2007 

Budget 

2008 

Budget 

Environmental Program and Management $17,744.8 $21,292.4 $19,935.0 

State and Tribal Assistance Grants 4,192.6 5,940.0 5,940.0 

Total $21,937.4 $27,232.4 $25,875.0 

Source: FY 2008 Annual Performance Plan and Congressional Justification (EPA’s Proposed 

Budget). 

The Office of Pollution Prevention and Toxics, within the Office of Prevention, 

Pesticides, and Toxic Substances (OPPTS), is responsible for the P2 Program. 

One way the Office of Pollution Prevention and Toxics encourages industry and 

others to implement P2 activities is through voluntary programs. The voluntary 

pollution prevention activities (identified in the PART as “Centers of Results”) 

include: 

1 
09-P-0088 

1.	 Design for the Environment: A partnership program that works with a 

broad range of stakeholders to use the unique tools and expertise of the 

New Chemicals Program to inform businesses of safer chemicals. 

2.	 Environmentally Preferable Purchasing: A federal government-wide 

program managed by the Pollution Prevention Division that requires and 

assists federal agencies in the purchasing of environmentally preferable 

products and services. 

3.	 Green Chemistry: A national Pollution Prevention Act and Toxic 

Substance Control Act based program that focuses on P2 through the 

environmentally conscious design of safer and economically competitive 

chemical products and processes. The program includes the Presidential 

Green Chemistry Challenge Awards for rewarding technologies that have 

demonstrated P2 benefits across the life cycle, including the design, 

manufacture, and use of chemical products. 

4.	 Green Suppliers Network: A collaborative venture between industry 

and EPA that works with all levels of the manufacturing supply chain to 

achieve environmental and economic benefits. 

5.	 Hospitals for a Healthy Environment/Partnership for Sustainable 

Healthcare: A program working with the healthcare industry to reduce 

its environmental impact. It became EPA’s first voluntary program to 

become an independent non-profit organization in 2006. The program 

was initially named Hospitals for a Healthy Environment, but took on the 

new name Partnership for Sustainable Healthcare when the initial program 

became independent. 

6.	 Pollution Prevention Resource Exchange: Provides national-level 

support and coordination of the Pollution Prevention Resource Exchange 

network through the provision of P2 information to facilitate State and 

tribal P2 technical assistance providers, and records the results of State 

P2 programs through a National P2 results system. 

7.	 P2 Program in EPA’s 10 Regional Offices: Report results from 

matching grants to State/tribal P2 programs, source reduction grants to 

promote P2 practices by industry, and direct regional efforts to facilitate 

business P2 practices and interstate coordination on promoting P2. 

Program Assessment Rating Tool 

PART, developed by OMB, is a diagnostic tool and management control process 

to assess program performance and drive a focus on results. PART is designed to 

provide a consistent approach to assessing and rating programs across the Federal 

Government. PART reviews overall program effectiveness, from how well a 

program is designed to how well it is implemented and what results are achieved. 

2
 
09-P-0088 

PART uses a standard questionnaire. For each question, agencies provide short 

answers and a detailed explanation with supporting evidence. Once each 

assessment is completed, a rating is assigned and, if necessary, a program 

improvement plan is developed to monitor and improve program performance. 

The P2 Program is one of 53 EPA programs assessed using the PART tool. It 

received a score of moderately effective based on a rating of 82 percent, the third 

highest among all EPA programs assessed. According to OMB, a program rated 

moderately effective has set ambitious goals and is well-managed. Moderately 

effective programs likely need to improve their efficiency or address other areas 

in the program’s design or management to achieve better results. 

Noteworthy Achievements 

The Design for the Environment center, working with the automotive refinishing 

industry, developed best practices that help small businesses reduce emissions of 

toxic chemicals. Building on a successful approach, Design for the Environment 

worked collaboratively with EPA’s Office of Air Quality Planning and Standards 

and EPA regional offices to base an area source regulation on this P2 program. 

The regulation became effective on January 9, 2008. 

Hospitals for a Healthy Environment became EPA’s first voluntary program to 

become an independent non-profit organization in 2006. Although the new entity, 

a 50l(c)(3) non-profit organization, is totally independent and has no legal 

connection to the Agency, it retained the name and has an agenda and mission 

consistent with EPA’s goals. 

Scope and Methodology 

We conducted our field work from October 2007 through April 2008 in accordance 

with generally accepted government auditing standards. Those standards require 

that we plan and perform the audit to obtain sufficient, appropriate evidence to 

provide a reasonable basis for our findings and conclusions based on our audit 

objectives. We believe that the evidence obtained provides a reasonable basis for 

our findings and conclusions based on our audit objectives. 

To accomplish our objectives, we reviewed the P2 Program’s responses to the 

OMB PART questionnaire and analyzed the evidence the program provided in 

support of those responses. We reviewed the supporting evidence for each of the 

PART questions to ensure that it adequately supported the program’s response 

and to determine whether it supported the OMB score received. In support of 

EPA’s mission “To Protect Human Health and the Environment,” we assessed 

whether the program’s performance measures adequately measure progress 

toward protecting human health and the environment. 

3
 
09-P-0088 

To test internal controls, we met with officials from each of the seven Centers of 

Results to identify their voluntary partners. For each center, we flow charted their 

organizational processes and documented their program data sources and uses. 

We obtained, analyzed, and traced the performance results to supporting 

documentation to verify and validate performance results reported for FY 2006. 

We met with the P2 coordinators for Regions 3 and 8 to obtain, review, and 

analyze the grantee data flow. We obtained the regional grantees’ program 

performance reports and traced the program results to supporting documentation. 

We reviewed the program improvement plan developed by OMB and the P2 

Program and obtained and analyzed actions taken and planned to address the plan. 

We briefed P2 Program managers on our preliminary results to obtain their 

feedback and ensure that we accurately and completely captured their P2 efforts. 

Because we looked at the first PART review of the P2 Program, no follow-up was 

needed. 

4
 
09-P-0088 

Chapter 2

PART Information Supports Rating 

The P2 Program’s data provided in response to the OMB PART Assessment 

generally addressed the PART questions and supported the moderately effective 

rating received. Moderately effective means the P2 Program has set ambitious 

goals and is well-managed. Despite receiving a moderately effective score, 

several internal control weaknesses were identified that need to be addressed. 

Information Supports Rating 

Our review of the P2 Program’s responses and the supporting data for each of the 

four PART categories disclosed the following. 

1. Program Purpose and Design: The program received a score of 

100 points in this section. We reviewed the documents provided as evidence 

and found they support the OMB PART assessment score. The P2 Program 

has been designed to address the requirements of the Pollution Prevention Act. 

2. Strategic Planning: The program received 88 out of 100 points in this 

section. We reviewed the documents provided as evidence and found they 

support the OMB PART assessment score. At 12 points each, no points were 

awarded for PART question 2.6, “Are independent evaluations of sufficient 

scope and quality conducted on a regular basis or as needed to support 

program improvements and evaluate effectiveness and relevance to the 

problem, interest, or need?” While the P2 program has conducted evaluations, 

including two conducted by EPA’s Office of Policy, Economics and 

Innovation, they did not meet OMB’s criteria for independence. 

3. Program Management: The program received 70 out of 100 points for 

this section. We found that the quality of the data supporting the P2 PART 

performance measures impacts the program’s performance measurement and 

accomplishment reporting. At 10 points each, no points were awarded for the 

following questions: 

•	 “Does the agency regularly collect timely and credible performance 

information, including information from key program partners, and use 

it to manage the program and improve performance?” (Question 3.1) 

•	 “Are federal managers and program partners held accountable for cost, 

schedules and performance results?” (Question 3.2) 

•	 “Does the program collect grantee performance data on an annual 

basis and make it available to the public in a transparent and 

meaningful manner?” (Question 3.CO3) 

5
 
09-P-0088 

4. Program Results/Accountability: The program received 80 out of 100 

points for this section. We found that the documents provided as evidence 

support the OMB PART assessment score in this section. At 20 points each, 

no points were awarded for question 4.5, “Do independent evaluations of 

sufficient scope and quality indicate that the program is effective and 

achieving results?” OMB made a determination that the evaluations 

conducted by the program lacked sufficient independence to receive credit 

under the PART. As a result of our review, we believe P2 should take 

additional steps to demonstrate relative effectiveness of its results in 

protecting human health and the environment, as discussed later in this report. 

Several Weaknesses Noted 

In addition to reviewing the PART and the P2 Program’s evidence provided in 

support of its responses to the PART questions, we also examined the internal 

controls used to capture and report program accomplishments. Government 

Accountability Office Standards for Internal Controls in the Federal Government 

state that internal controls are an integral component of an organization’s 

management that provide reasonable assurance that the program operates 

effectively and efficiently. We found the following weaknesses: 

•	 The P2 Program’s FY 2006 PART approved performance measures were 

not designed to report on the effects on human health and the 

environment. 

•	 The P2 Program’s verification and validation procedures did not ensure 

the accuracy of performance data. 

•	 The PART program improvement plan did not address all the deficiencies 

identified. 

Details on our findings and recommendations for these three issues are in 

Chapters 3 through 5, respectively. 

6
 
09-P-0088 

Chapter 3

Performance Measures Need to 

Better Address Impacts 

The P2 Program’s two FY 2006 PART-approved long-term performance 

measures are not designed to report on the program’s impact to human health or 

environmental conditions. The two FY 2006 PART measures used by the P2 

Program are: 

•	 Cumulative pounds of hazardous materials reduced by P2 Program 

participants. 

•	 Cumulative business, institutional, and government cost reduced by P2 

Program participants. 

According to the Chief, Planning and Assessment Branch, the lack of reliable data 

limits the P2 Program’s ability to measure its impacts on human health and the 

environment. This limits EPA’s ability to measure progress toward meeting the 

Agency’s goal of protecting human health and the environment. 

P2 Program Needs Better Performance Measures 

Federal agencies are expected to demonstrate how their activities contribute to 

achieving agency or government-wide goals. The Government Performance and 

Results Act requires federal agencies to report annually on their progress in 

achieving their agency and program goals. 

EPA categorizes its performance goals and measures in a framework known as 

the “hierarchy of indicators” (see Figure 3-1; Appendix A provides more details). 

Figure 3-1: EPA Hierarchy of Indicators 

Level 

1 2 3 4 5 6 

Actions by 

EPA, States, 

tribes, or other 

governmental 

bodies 

Actions or 

responses 

by 

regulated 

parties 

Discharges 

and 

emissions of 

pollutants 

Concentrations 

of pollutants in 

the 

environment 

Pollutants 

absorbed by 

the human 

body 

Environmental 

risk or impacts 

to the ecology 

or human 

health and/or 

welfare 

Source: Government Accountability Office Report, GAO/RCED-00-77, EPA Faces Challenges 

in Developing Results-Oriented Performance Goals and Measures, April 2000 

This framework ranks the performance goals and measures according to their 

impact on improving human health and the environment. At the top of the 

hierarchy, level 6 indicators are reductions of environmental risks or impacts to the 

ecology or to human health and/or welfare. Level 3 indicators are reductions in 

7 
09-P-0088 

discharges and emissions of pollutants. The P2 Program’s two FY 2006 PARTapproved long-term end outcome performance measures fall into the level 3 

category, which limits the P2 Program’s ability to relate program results to impacts 

on human health and the environment. P2 Program managers believe that the 

lower level outcome measures represent the highest level measure relating to the 

program’s hazardous materials reduction mission that can be supported by data 

obtainable on a program-wide basis. However, the P2 Program does have two 

higher level Government Performance and Results Act performance measures 

(reductions in energy use and water use) that are included in the Program’s annual 

President’s Budget and Performance Accountability Report but were initially 

rejected as PART measures during the FY 2006 PART assessment. OMB 

approved the water savings measure and an additional higher-level measure 

“reductions in green house gas emissions” as PART measures in the 2008 PART 

Spring Update. 

The seven Centers of Results logic models further illustrate that the P2 Program’s 

FY 2006 PART measures were not designed to measure impacts to human health 

or the environment (see example Figure 3-2). A logic model is an evaluation tool 

used to describe a program’s components and desired results. It explains the 

strategy or logic by which the program expects to achieve results. Logic models 

help articulate a program’s expected outcomes by linking program resources and 

activities to program outcomes. While not inherently measureable, the centers’ 

logic models do highlight movement towards sustainability through behavior 

change as well as within center mission and goal statements. We reviewed each 

of the logic models and found that none directly linked their resources to 

effectiveness in protecting human health and the environment. Instead they 

focused on hazardous materials reductions and cost savings, which equate to 

level 3 on EPA’s hierarchy of indicators. 

The Chief, Planning and Assessment Branch, told us that due to a lack of reliable 

and readily available data on a program-wide basis required to support higher 

level measures, level 3 measures are generally accepted by the P2 Program as the 

highest available for measuring results associated with the program’s hazardous 

materials reduction mission. He indicated that while some P2 results centers such 

as Design for the Environment and Green Chemistry do have access to these data, 

obtaining them on a program-wide basis would be challenging and require OMB 

approval of Information Collection Requests and consent by voluntary partners 

and State P2 Programs to provide the data. The Chief further indicated that 

measuring improvements to health and ecological conditions is challenging 

because no methodology exists for modeling changes in these conditions based on 

reductions to any and all pollutants. Available methodologies are only equipped 

to measure hazard or risk change on a chemical-by-chemical basis where the 

chemical of concern is known. According to the Chief, additional methodologies 

and data are needed to establish a direct cause-and-effect relationship between the 

program’s activities and the resulting changes in the environment and human 

health. 

8
 
09-P-0088 

Figure 3-2: Pollution Prevention Resource Exchange Program Logic Model 

P2Rx Center 

Support 

Develop grant 

policy 

Announce grant 

availability 

Review proposals & 

award grants 

Continually improve 

grants process 

Coordinate & meet 

with Regional staff 

Monitor Centers, 

develop metrics, & 

track performance 

P2Rx RFPs 

P2Rx Grants 

P2Rx Grants 

oversight & 

management 

documentation 

Coordination and 

meetings with 

Centers’ staff 

Consolidated data 

from grantees on 

performance 

States and state 

entities eligible for 

grants 

P2Rx Centers 

Reduced hazardous 

materials released, 

incorporated into 

products, or used in 

processes 

Reduced, 

conserved, or offset 

use of energy 

Conservation of 

water 

Cost savings 

through pollution 

prevention 

improvements 

Improved 

sustainability of 

businesses, 

universities, federal 

state and local 

government 

P2RX centers and 

TAPs assist more 

clients 

Customers: 

-incorporate a P2Rx 

search in their work 

-Improve their ability 

& skill in accessing 

P2 info 

-Use P2 tools more 

often 

-Implement P2Rxsourced P2 practices 

Increased demand for 

& improved service 

delivery by TAPs 

Increased 

collaboration among 

organizations with 

shared goals 

More user sessions, 

increased user 

sessions per dollar 

spent 

Improvements in 

program efficiency 

and effectiveness 

Increased demand for 

sustainable products 

& services 

EPA Regions 

and State 

Grantees 

State & local P2 

providers 

Matching funds 

and in-kind 

services and 

information 

Centers’ design 

EPA Regional 

project offices 

P2Rx Center 

Operation 

Develop & support 

P2RX Web site 

Provide P2 TA & 

content expertise 

(e.g., peer review) 

Conduct training 

Provide information 

management 

Perform networking 

Develop 

partnerships to 

promote P2 activity 

Collect data & 

measure results 

P2Rx Websites 

Information 

collections: 

-sector resources, 

news, directory of P2 

programs, etc 

Technical assistance 

Training 

Regional 

communication and 

information sharing: 

- project updates, 

meeting coordination 

Partnerships with 

NPPR, CA Centers, 

others 

Data on activities, 

Web use measures, 

etc. 

Businesses 

Universities and 

schools 

Federal, state and 

local governments 

EPA HQ and 

Regions 

STAG (PPIN) 

funds 

EPM funds 

EPA HQ and 

Regional staff 

Centers’ design 

POLLUTION PREVENTION RESOURCE EXCHANGE (P2Rx) PROGRAM LOGIC MODEL 

INPUTS ACTIVITIES OUTPUTS CUSTOMERS ATTITUDE BEHAVIOR CONDITION 

OUTCOMES 

Acronym Key: 

P2Rx:Pollution Prevention Resource Exchange STAG: State and Tribal Grant EPM:Environmental Program Management TA: Technical Assistance 

NPPR:National Pollution Prevention Roundtable CA:Compliance Assistance OECA:Office of Enforcement and Compliance Assurance TAP: TA Provider 

Increase in 

grantees and TA 

providers’ ability to 

reach and assist 

clients 

Increase 

awareness of P2 

Increase ability to 

understand P2 and 

willingness to use 

P2 tools 

JULY 2007 

MISSION: To advance pollution prevention and resource conservation as effective and efficient environmental practices, by 

fostering the exchange of pollution prevention information, enhancing networking opportunities and providing technical assistance. 

Source: Chief, Planning and Assessment Branch, OPPTS 

We recognize that the P2 Program’s effectiveness in reducing human health and 

environmental risk associated with hazardous materials reductions is difficult to 

evaluate. The program nonetheless recognizes the value in doing so. As the 

former P2 Director stated, to evaluate program effectiveness, programs must be 

able to show the outcomes from their activities. That focus enables one to 

demonstrate changes in understanding and behavior of the target audience and, 

ultimately, quantify the effects on the environment and human health. 

Defining a Universe of P2 Pollutants Difficult 

The P2 Program measures hazardous materials source reduction progress in 

relation to targeted goals rather than the actual or estimated universe of pollutants 

to be addressed. According to the Chief, Planning and Assessment Branch, there 

is no known source of reliable information for the P2 Program to use that 

identifies the universe of hazardous materials generated and/or used in the United 

States, against which to demonstrate the program’s relative effectiveness. We 

recognize that it may be difficult to identify a universe of pollutants across all 

media and uses. However, providing a complete picture of the program’s 

9
 
09-P-0088 

performance results could provide a better perspective of the P2 Program’s 

effectiveness and progress in addressing source reduction. 

We found that the FY 2006 P2 Program’s PART results do not provide a 

complete picture of the program. Each of the seven Centers of Results focuses on 

particular chemicals and specified uses. They report program accomplishments 

based on those chemicals as they relate to the specified uses. For example, at the 

end of FY 2006, the Partnership for Sustainable Healthcare program reported 

actual results of 17.6 million pounds of regulated medical waste and hazardous 

waste reduced by hospitals participating in the program. However, the 

Partnership did not report 5.0 million pounds of the same wastes increased at 

those participating hospitals during the same reporting period as a result of 

increased production. The program should make it clear in its results reporting 

that it does not account for increased waste generation resulting from increased 

production levels. 

As another example, the Design for the Environment program reported 

124.7 million pounds of hazardous material reductions in FY 2006, representing 

approximately 33 percent of the year’s total reductions. Program 

accomplishments were the result of its two sub-programs – Lead-Free Solder 

Initiative and Formulator. However, the Lead-Free Solder Initiative reported 

reductions of lead solder that make up a substantial portion of the Design for the 

Environment total. This accomplishment data is provided by a trade association 

and represents the amount of lead-free solder shipped globally during calendar 

year 2006. That amount, however, does not represent the amount of lead solder 

reduced; rather the program is using that amount as a conservative proxy for the 

amount of lead solder use actually reduced. Providing such information would 

give stakeholders an improved perspective of the extent of the threat that lead 

solder continues to pose. 

Recommendations 

We recommend that the Assistant Administrator for the Office of Prevention, 

Pesticides, and Toxic Substances: 

3-1 Focus on the select P2 pollutants where health effects are known and 

develop higher level performance indicators that address reductions of 

environmental risks or impacts to human health. 

3-2 Using the results of Recommendation 3-1, design a strategy for 

developing P2-wide PART higher level measurements that could be 

used as a model to further develop P2 Program-wide higher level 

measurements. 

10
 
09-P-0088 

3-3 Report program accomplishments as they relate to the universe for the 

targeted chemical in the specified uses addressed by the P2 Program to 

provide a better perspective of the program’s relative effectiveness. 

Agency Response and OIG Comments 

OPPTS concurred with all of our recommendations in this area. 

We evaluated the program’s planned actions for each recommendation and 

concluded that, if implemented according to their descriptions, they could 

reasonably be expected to correct the concerns we identified. We also analyzed 

the corresponding planned completion dates and similarly found them to be 

reasonable. 

In response to Recommendation 3-1, OPPTS indicated its intention to develop an 

initial indicator tracking one or more health effects associated with at least one 

chemical that the program is currently focusing on by March 2009, and 

subsequently begin gathering data to support tracking the indicator on an annual 

basis. 

In response to Recommendation 3-2, OPPTS indicated its intention to include in 

EPA’s September 2009 submission to Congress of its 2014 Strategic Plan at least 

one new higher-level Strategic Measure and associated long-term performance 

targets. OPPTS will propose adding the new Strategic Measure and associated 

annual performance targets to the 2009 Fall PART Update and assess 

opportunities for developing additional higher-level measures and targets in future 

years. 

In response to Recommendation 3-3, OPPTS indicated that in September 2008, it 

included a proposed new Pollution Prevention Program Strategic Measure in 

EPA's 2014 Strategic Plan Change Document (circulated for public comment). 

The proposed strategic measure tracks the program's results in reducing 

production and use of a priority set of hazardous chemicals which will enable the 

program to define associated universe(s) that will provide the recommended 

perspective on the program's relative effectiveness. In spring 2009, OPPTS and 

Office of the Chief Financial Officer will determine whether to include the 

proposed measure in public comment drafts of the Strategic Plan and whether to 

submit to OMB as formal PART measures. 

11
 
09-P-0088 

Chapter 4

P2 Program Needs to Strengthen 

Internal Controls over Data Collection 

The P2 Program needs to strengthen internal controls over data collection and 

reporting of accomplishments. OPPTS’s Quality Management Plan (QMP) 

requires meeting quality standards and establishing acceptance criteria. In 

addition, Government Accountability Office internal control standards require 

relevant, reliable, and timely data. The P2 Program had established quality 

assurance procedures and policies governing its contractor and grantee supported 

collection and use of environmental data, but had not established similar 

procedures and policies governing work performed by EPA staff. While the 

Office of Pollution Prevention has established broad controls through the QMP, it 

did not develop a Quality Assurance Project Plan (QAPP) that would have 

included data quality objectives. As a result, EPA did not have assurance that 

performance results were accurately reported and EPA had limitations on 

demonstrating program effectiveness. According to the P2 Program, they have a 

QAPP that is currently being used in at least one region. However, the program 

acknowledged that QAPPs are not in place in all regions. 

Collecting Data from Multiple Sources Challenging 

OPPTS’s QMP states that all environmental data activities generated by or for 

OPPTS shall meet quality standards, and that appropriate acceptance criteria will 

be established prior to any environmental data activity. In addition, the 

Government Accountability Office’s Standards for Internal Control in the 

Federal Government states that for an entity to run and control its operations, it 

must have relevant, reliable, and timely communications relating to internal as 

well as external events. It further states management should ensure there are 

adequate means of communicating with, and obtaining information from, external 

stakeholders that may have a significant impact on an agency achieving its goals. 

The Office of Pollution Prevention and Toxics relies on voluntary programs to 

encourage and carry out pollution prevention activities. The seven Centers of 

Results work in partnership with a broad range of stakeholders to reduce risk by 

preventing pollution at its source. The Centers collect and aggregate performance 

data from voluntary partners at other federal agencies, States, tribes, and industry. 

•	 P2 Programs in EPA’s 10 regional offices collect and aggregate 

performance data from States, tribes, and academia (typically grantees), as 

well as through direct interaction with entities implementing pollution 

prevention activities. 

12
 
09-P-0088 

•	 Partnership for Sustainable Healthcare performance data are manually 

extracted from hospitals’ recognition award applications. 

•	 Green Chemistry extracts performance data from award nomination 

packages. Data is reported for the winners of the awards. 

•	 Pollution Prevention Resource Exchange collects performance data from 

State technical assistance providers who submit performance data to the 

eight P2 resource regional exchange centers. 

•	 Environmentally Preferable Purchasing electronically extracts 

performance data from the Federal Electronics Challenge partners and 

Green Electronics Council trade association data which is run through a 

peer reviewed Electronics Environmental Benefits Calculator. 

•	 Design for the Environment, through its partnerships with industry 

representatives and environmental groups, manually collects, aggregates, 

and calculates performance results data. Performance data are also 

gathered from trade associations. 

•	 Green Suppliers Network, through a partnership with the National Institute 

of Standards and Technology, extracts performance results from surveys 

of companies that the institute reviews for lean manufacturing processes. 

Each of the seven Centers of Results aggregates its P2 performance results data 

from these sources and submits the data to Office of Pollution Prevention and 

Toxics’ Planning and Assessment Branch for consolidation and reporting. In 

addition, the Branch annually extracts from the Agency Commitment System 

direct contributions to pollution prevention from EPA’s regional offices. 

Inconsistent and Inaccurate Reporting of Results 

We found that the P2 Program had not established controls for the consistent 

reporting of State and tribal program results through its regional grant programs. 

We met with two regional P2 coordinators to discuss and review their data 

sources used in reporting grant results and found that there was no consistent 

format for reporting grant results. Also, verification and validation procedures 

did not ensure the accuracy of grantee results submitted by the States or the 

facilities claiming the accomplishments. The P2 Program has indicated they are 

taking steps to improve the consistency and transparency of annual regional 

results, including: development of QAPP guidance currently being used by 

several regions; issuance of new guidance to grant applicants on measuring and 

reporting the results from grant awards; standardization of a model spreadsheet 

for reporting outcome performance results; and posting of performance results on 

the P2 Website. 

In the spring 2007 PART update to the program improvement plan, the P2 

Program reported that each region will use a standard spreadsheet beginning in 

2008 to display grant and direct project annual results by measure. Implementing 

a standard format for reporting, along with procedures, should facilitate 

consistency in reporting regional P2 annual results. 

13
 
09-P-0088 

The P2 Program’s Partnership for Sustainable Healthcare reported $6.3 million in 

business, institutional, and government costs reduced by P2 program participants. 

We were unable to substantiate these cost savings from the documents provided 

as support. The partnership’s cost savings are manually extracted from the 

participating hospital’s recognition award applications. The P2 Program does not 

have written procedures for its EPA staff for extracting cost savings from the 

participating hospitals’ award applications. We randomly reviewed 11 

(23 percent) of the 47 award applications that reported cost savings, and were 

unable to arrive at the cost savings reported by the P2 Program for seven of the 

applications. According to a program manager, limited resources and short 

reporting timeframes affect data quality. 

At the end of FY 2006, P2 reported actual program results of 383 million pounds 

of hazardous materials reduced and $85.7 million in cost reductions by program 

participants (see Figures 4-1 and 4-2). 

 Figure 4-1: FY 2006 Hazardous Materials Reductions 

Hazardous Materials Reduced 

(Million - Pounds) 

0 

50 

100 

150 

GC DfE Reg PSH EPP GSN P2Rx 

Prior Year Reductions 

(Recurring) 

2006 Reductions 

(Actual) 

Off 

Figure 4-2: FY 2006 Cost Reductions 

0 

10 

20 

30 

40 

50 

EPP R eg 

Off 

PSH G C G SN P2R x D fE 

B usin ess, In stitu tio n al an d G o vernm en t C o sts S aved 

(M illio ns) 

Source: OIG analysis of P2 FY 2006 results data 

DfE Design for the Environment Center of Results 

EPP: Environmentally Preferable Purchasing Center of Results 

GC: Green Chemistry Center of Results 

GSN: Green Supplies Network Center of Results 

P2Rx: Pollution Prevention Resource Exchange Center of Results 

PSH: Partnership for Sustainable Healthcare Center of Results 

Reg Off: Regional Offices’ Direct Contributions 

14
 
09-P-0088 

Representing approximately 38 percent of the total 383 million pounds of 

hazardous materials reduced in FY 2006, the Green Chemistry program reported 

145.6 million pounds reduced. We reviewed documentation supporting Green 

Chemistry’s 145.6-million-pound annual reduction claimed in FY 2006 and found 

that only 5.8 million pounds were actually reduced by the 2006 Presidential Green 

Chemistry Award winner in 2006. The remaining 139.9 million pounds were 

reductions realized as far back as 1996 and projected through FY 2011 – the end 

of the Agency’s 2006-2011 strategic planning cycle. For example, the 5.8 million 

pounds of hazardous materials reductions reported by the 2006 award winners 

will be reported as reductions every year through FY 2011. Green Chemistry 

projections include a company’s annual reductions in subsequent years as current 

year reductions rather than reporting actual annual accomplishments. We did not 

test the soundness of this recurring results methodology for projecting annual 

program accomplishments out for approximately 15 years. We were advised by 

P2 that they conducted a consultation with EPA’s Science Advisory Board on 

September 3, 2008 to obtain input on their methodology for measuring program 

results, and are awaiting the Board’s formal communication of recommendations. 

Data Quality Requirements Not Met 

The ultimate success of a program or project depends on the adequacy, 

sufficiency, and quality of the environmental data collected and used in decision 

making. EPA Requirements for Quality Management Plans (March 2001) states 

that EPA organizations are required to establish, implement, and document their 

quality system in a QMP. The QMP is supported by project-specific QAPPs, 

which require all work performed by, or on behalf of, EPA involving the 

collection of environmental data be implemented in accordance with an Agencyapproved QAPP. A QAPP defines and documents how specific data collection 

activities shall be planned, implemented, and assessed during a particular project. 

We reviewed OPPTS’s QMP for the Pollution Prevention Division (June 2003), 

and found that P2’s activities involved the collection and analysis of 

environmental data as well as analyzing environmental data previously collected. 

At the time of our review, we did not find that a QAPP was developed for P2 

projects. Developing and implementing an effective QAPP may have prevented 

many of the data quality issues we identified with the P2 Program. Although 

OPPTS has developed a broader QMP, we found it was not being fully 

implemented by all P2 Centers. Further, more detailed procedures, which would 

be included in a QAPP, were not developed. As a result, P2's and the program 

partner’s verification and validation procedures did not ensure the accuracy of 

performance data used for reporting on program accomplishments. 

As P2 performance data is communicated internally and externally to Congress 

and other stakeholders, the data should be relevant, reliable, and timely. The 

Pollution Prevention Division needs to implement the requirements of the QMP to 

ensure the quality of its environmental data. 

15
 
09-P-0088 

Recommendations 

We recommend that the Assistant Administrator for the Office of Prevention, 

Pesticides, and Toxic Substances: 

4-1 Revise the P2 Program procedures for calculating program results to 

collect and use current year data from its voluntary partners to 

calculate annual program results. 

4-2 Require the development of QAPPs for collecting environmental data. 

4-3 Implement controls to ensure full implementation of the Pollution 

Prevention Division’s QMP. 

Agency Response and OIG Comments 

OPPTS concurs with all of our recommendations in this area. 

We evaluated the Program’s planned actions for each recommendation and 

concluded that, if implemented according to their descriptions, they could 

reasonably be expected to correct the concerns we identified. We also analyzed 

the corresponding planned completion dates and similarly found them to be 

reasonable. 

In response to Recommendation 4-1, OPPTS indicated it met with EPA's Science 

Advisory Boards in September 2008 to obtain advice on how to more fully reflect 

its benefits by measuring its recurring as well as new annual results and is 

developing measures that clearly distinguish the two. 

In response to Recommendation 4-2, OPPTS explained that although the 

development of QAPPs for contractors and grantees has been a tenet of the 

OPPTS Quality Assurance policy for a number of years and all P2 Program 

Centers have been in compliance with this policy, outside organizations that 

voluntarily submit data to the Program have generally not submitted associated 

QAPPs. We maintain that at the time of completion of our audit we had not 

received evidence that any QAPPs were in effect for the Program whether for 

contractors and grantees, regions or centers, or voluntary partners. In response to 

our draft report issued to the P2 Program on November 3, 2008, we received from 

OPPTS a document entitled “QAPP Lite” which was reportedly in place for at 

least Region 1. This document, in our opinion, did not approach the purpose and 

standards prescribed by the QMP. The P2 Program also contends that two of its 

centers have data validation procedures for information submitted by partners and 

in nominations for program awards. These procedures are not, however, a 

substitute for a high quality QAPP. 

16
 
09-P-0088 

In response to Recommendation 4-3, OPPTS indicated it is in the process of 

developing a set of Standard Operating Procedures to ensure implementation of 

the QMP. 

17
 
09-P-0088 

Chapter 5

Program Improvement Plan: Progress Slow and 

Not All Weaknesses Addressed 

While P2 has completed many of its interim actions to address the program 

improvement plan (PIP), the plan did not address all the deficiencies identified 

during the PART assessment. The P2 Program received a PART score of zero on 

critical questions regarding program evaluations, data quality, and management 

accountability during its FY 2006 PART assessment. Nonetheless, no actions 

were identified to correct these shortcomings. To maximize the effectiveness of 

the PART tool and ensure the P2 Program is operating efficiently and effectively, 

the PIP should address the cause of the deficiencies identified during the 

assessment process. 

Progress to Address PIP Slow 

In response to each of the PART areas assessed, the program office and OMB 

identify actions to improve the program’s performance through a PIP. Agency 

steps to complete these actions and improve performance are tracked through the 

President’s Management Agenda scorecard for the Performance Improvement 

Initiative, as well as through PART updates. As a result of the 2006 PART 

assessment, OMB examiners and P2 Program managers developed the following 

activities for the P2 Program: 

•	 Evaluate Science Advisory Board report recommendations for improving 

performance measures to better demonstrate P2 results. 

•	 Identify and reduce barriers associated with core EPA activities that limit 

implementation of pollution prevention practices by industry. 

•	 Develop additional P2 Program efficiency measures to expand the portion 

of the program’s resources that are addressed. 

•	 Fully implement GranTrack and P2 State Reporting System. Obtain 

consistent 2007 results from regions. 

We reviewed the P2 Program’s 2007 PART spring PIP update and discussed the 

actions taken with P2 Program managers. Although the P2 Program has 

completed many of its interim actions to address the PIP, 2 years later only one of 

the corrective actions had been fully completed and implemented. Details on 

what we found for each of the four activities listed above follow. 

18
 
09-P-0088 

Evaluate Science Advisory Board report recommendations for 

improving performance measures to better demonstrate P2 results 

Although the P2 Program initiated actions to secure a commitment with the 

Science Advisory Board in FY 2006 for assistance in making improvements in 

reporting accomplishments and developing measures, an actual commitment was 

not received until April 2008 for a September 2008 consultation, almost 2 years 

later. 

On June 14, 2006, the P2 Program provided the Science Advisory Board an 

overview on the program and the measurement issues they face. They also 

provided questions for the Science Advisory Board to consider pertaining to 

projecting recurring results, capturing the impact of indirect assistance, and using 

environmental calculators. In the 2007 PART spring update, the P2 Program 

indicated it had received a commitment from the Science Advisory Board to 

assess and review P2 Program performance measures. However, we found that 

due to scheduling problems with the Science Advisory Board and the P2 Program 

Division Director’s retirement, the P2 Program had not received assistance from 

the Science Advisory Board. 

In April 2008, the P2 Program and the Science Advisory Board were able to 

schedule and subsequently conduct a consultation to assess the methodology 

surrounding the use of recurring results in performance measurement and to 

explore the feasibility of adding higher level indicators (4, 5, or 6) from the 

Agency’s Hierarchy of Indicators. According to the P2 Program the Science 

Advisory Board also examined how to measure outcome data from indirect 

program interventions. Indirect interventions are not on-site and include 

dissemination of P2 information through various means. The program anticipates 

formal recommendations from the Board to be provided to the EPA Administrator 

before the end of 2008, and causing a further extension of the associated PART 

PIP to evaluate those recommendations until the 2009 spring PART update. 

Identify and reduce barriers associated with core EPA activities that 

limit implementation of pollution prevention practices by industry 

The Pollution Prevention Division conducted a self assessment and developed a 

draft report, “Evaluation of EPA Efforts to Integrate Pollution Prevention Policy 

throughout EPA and at Other Federal Agencies” (dated March 27, 2008). The 

evaluation assessed the current state of P2 integration within EPA programs, 

including barriers and Toxic Release Inventory source reduction reports. The 

evaluation focused on Pollution Prevention Act provisions that address integrating 

P2 policy into federal programs authorized under other statutes. The P2 Program 

concluded that most of the Agency’s program activities and efforts to promote P2 

among other EPA programs and at other federal agencies are in basic conformity 

with the Act’s provisions for P2 integration but there is clearly room for 

improvement. The reported recommendations were: 

19
 
09-P-0088 

•	 Take short-term action on several steps to improve the implementation 

of pollution prevention policy; 

•	 Review the Agency’s information collection practices pertaining to P2, 

with the aim of improving overall ease of data collection, coordination 

between systems, and availability of information once collected; 

•	 Address the need for meaningful P2 guidance in the Agency’s 

Economic Impact Analysis Guidance; 

•	 Begin in the short term two processes for more thoroughly improving 

implementation of pollution prevention policy; and 

•	 Begin longer-term processes for addressing more far-reaching positive 

improvements in the implementation of pollution prevention policy. 

The P2 Program provided the draft report to the States and noted in its response to 

our draft report that it had received feedback in the summer 2008. The P2 

Program plans to incorporate key recommendations into its strategic plan. 

Develop additional P2 Program efficiency measures to expand the 

portion of the program’s resources that are addressed 

According to the Chief, Planning and Assessment Branch, the P2 Program has 

developed a revised Design for the Environment efficiency measure which 

increased the scope of the program in the fall 2007. Another measure, assessing 

the efficiency of the Environmentally Preferable Purchasing program, will be 

introduced in the fall 2008. These measures will cover more of the P2 Program’s 

resources, as recommended by OMB. 

Fully implement GranTrack and P2 State Reporting System. Obtain 

consistent 2007 results from regions 

GranTrack is a Web-based tool designed to help EPA regional and headquarters 

project officers manage grants and access information on pollution prevention 

projects around the country. According to the Chief, Planning and Assessment 

Branch, GranTrack is one of several options currently being assessed to track P2 

results from State Technical Assistance Grants and Source Reduction Assistance 

Grants. While the Fiscal Year 2006 PIP recommendation was to obtain consistent 

2007 results from the regions, a decision on whether GrantTrack will be used had 

not been made as of fall 2008. 

The State P2 Reporting system, “P2 Results,” has been fully implemented. The 

first report using performance information from this system has been compiled 

and is planned for release in the summer 2008. These data come from States and 

businesses through Pollution Prevention Resource Exchange centers and the 

report will be issued by the National Pollution Prevention Roundtable. 

20
 
09-P-0088 

To obtain consistent 2007 results from regions, the program initiated a reporting 

methodology in a standard spreadsheet. Beginning in 2008, each P2 region will 

be required to use a standard spreadsheet in addition to P2 results to display grants 

and project annual results by measure. 

The Program also issued guidance, “Draft 2008 Measurement Guidance for EPA 

P2 Coordinators,” on developing FY 2008 commitments to EPA’s Agency 

Commitment System, to further ensure consistency in reporting results. While 

labeled draft, the Chief, Planning and Assessment Branch, indicated the guidance 

is a “living document,” and as new and better ways are discovered to improve 

measurement methodologies they will be incorporated into the measurement 

guidance. P2 Program managers met with regional P2 coordinators in May 2008 

and plan to issue a final version of the guidance incorporating feedback from the 

P2 coordinators in the fall 2008. 

Additional Program Improvements Needed 

The PIP developed by OMB and the P2 Program following the P2 PART 

Assessment did not address all the deficiencies identified. The PART tool is used 

to assess the program’s performance and drive improvements. Therefore, to 

maximize its effectiveness, the PIP should address the deficiencies identified 

during the assessment process. Correcting the deficiencies for the critical 

questions that scored zero in the PART assessment can help improve the P2 

Program’s overall performance. Details are in Table 5-1. 

Table 5-1: P2 Program’s FY 2006 PART Questions Receiving a Score of Zero 

Question # PART Questions Critical Areas 

2.6 Are independent evaluations of sufficient scope and quality 

conducted on a regular basis or as needed to support program 

improvements and evaluate effectiveness and relevance to the 

problem, interest, or need? 

Program 

Evaluations 

3.1 Does the agency regularly collect timely and credible performance 

information, including information from key program partners, and 

use it to manage the program and improve performance? 

Data Quality 

3.2 Are Federal managers and program partners held accountable for 

cost, schedules and performance results? 

Management 

Accountability 

3.CO3 Does the program collect grantee performance data on an annual 

basis and make it available to the public in a transparent and 

meaningful manner? 

Data Quality 

4.5 Do independent evaluations of sufficient scope and quality indicate 

that the program is effective and achieving results? 

Program 

Evaluations 

Source: OIG review and analysis of the P2 Program’s FY 2006 PART Assessment 

21
 
09-P-0088 

Recommendations 

We recommend that the Assistant Administrator for the Office of Prevention, 

Pesticides, and Toxic Substances: 

5-1 Have the P2 Program plan and schedule periodic evaluations of its 

pollution prevention efforts to identify areas where program 

improvements are needed and implement needed corrective actions. 

5-2 Develop actions to address the PART questions where the P2 Program 

was not awarded any points during the FY 2006 PART Assessment 

and include the PIP in the next PART update to promote continual 

P2 Program improvements. 

Agency Response and OIG Comments 

OPPTS concurs with all of our recommendations. 

We evaluated the program’s planned actions for each recommendation and 

concluded that, if implemented according to their descriptions, they could 

reasonably be expected to correct the concerns we identified. We also analyzed 

the corresponding planned completion dates and similarly found them to be 

reasonable. 

In response to Recommendation 5-1, OPPTS reiterated that it has conducted two 

extensive program evaluations addressing two of its Centers of Results that were 

completed on schedule and within budget. We maintain that the P2 Program 

could benefit from planned and scheduled evaluations of their voluntary 

programs. 

In response to Recommendation 5-2, OPPTS indicated it will ensure that all 

proposed replacements for completed follow-up actions are targeted at 

unaddressed weaknesses identified in the Pollution Prevention Program PART 

Assessment until all weaknesses have been fully addressed or, in the case of the 

PART requirements for independent program evaluations, addressed to the extent 

deemed practicable given program resource allocations and influence over 

decisions made by independent evaluating entities. 

22
 
09-P-0088 

Status of Recommendations and 

Potential Monetary Benefits 

POTENTIAL MONETARY 

RECOMMENDATIONS BENEFITS (in $000s) 

Planned 

Rec. Page Completion Claimed Agreed To 

No. No. Subject Status1 Action Official Date Amount Amount 

3-1 10 Focus on the select P2 pollutants where health
 

effects are known and develop higher level 

performance indicators that address reductions of
 

environmental risks or impacts to human health.

3-2 10 Using the results of Recommendation 3-1, design a 

strategy for developing P2-wide PART higher level 

measurements that could be used as a model to 

further develop P2 Program-wide higher level 

measurements.

3-3 11 Report program accomplishments as they relate to 

the universe for the targeted chemical in the 

specified uses addressed by the P2 Program to 

provide a better perspective of the program’s 

relative effectiveness.

4-1 16 Revise the P2 Program procedures for calculating 

program results to collect and use current year 

data from its voluntary partners to calculate annual 

program results.

4-2 16 Require the development of QAPPs for collecting 

environmental data.

4-3 16 Implement controls to ensure full implementation of 

the Pollution Prevention Division’s QMP.

5-1 22 Have the P2 Program plan and schedule periodic 

evaluations of its pollution prevention efforts to 

identify areas where program improvements are 

needed and implement needed corrective actions.

5-2 22 Develop actions to address the PART questions 

where the P2 Program was not awarded any points 

during the FY 2006 PART Assessment and include 

the PIP in the next PART update to promote 

continual P2 Program improvements.

1	 O = recommendation is open with agreed-to corrective actions pending 

C = recommendation is closed with all agreed-to actions completed 

U = recommendation is undecided with resolution efforts in progress 

 Assistant Administrator,
 

Office of Prevention,
 

Pesticides, and 

Toxic Substances 

 Assistant Administrator,
 

Office of Prevention,
 

Pesticides, and 

Toxic Substances 

 Assistant Administrator,
 

Office of Prevention,
 

Pesticides, and 

Toxic Substances 

 Assistant Administrator,
 

Office of Prevention,
 

Pesticides, and 

Toxic Substances 

 Assistant Administrator,
 

Office of Prevention,
 

Pesticides, and 

Toxic Substances 

 Assistant Administrator,
 

Office of Prevention,
 

Pesticides, and 

Toxic Substances 

 Assistant Administrator,
 

Office of Prevention,
 

Pesticides, and 

Toxic Substances 

 Assistant Administrator,
 

Office of Prevention,
 

Pesticides, and 

Toxic Substances 

March 2009 

and 

ongoing 

Fall 2009 

Fall 2009 

Fall 2009 

2009 

2009 

Spring 2009 

Spring 2010 

23
 
09-P-0088 

Appendix A 

EPA’s Hierarchy of Indicators 

Level 

Description of 

Indicator 

Type of performance 

goal and measure 

generally associated 

with the indicator 

Example of 

performance goal 

Example of 

performance measure 

6 Environmental risk or 

impacts to the 

ecology or human 

health and/or welfare 

End outcome Restore watersheds to their 

designated uses 

Number of river miles, lake 

acres, and estuary square 

miles that will be restored 

to their designated uses 

5 Pollutants absorbed 

by the human body 

End outcome Reduce pesticide poisonings 

by 5 percent 

Pesticide poisonings will be 

reduced by 5 percent 

4 Concentrations of 

pollutants in the 

environment 

End outcome Maintain healthy air quality for 

levels of carbon monoxide, 

sulfur dioxide, nitrogen dioxide, 

and lead 

All areas currently meeting 

national ambient air quality 

standards will continue to 

maintain healthful 

standards for carbon 

monoxide, sulfur dioxide, 

nitrogen dioxide, and lead 

3 Discharges and 

emissions of 

pollutants 

End outcome Reduce discharges of toxic air 

pollutants by 4 million pounds 

per year 

Discharge of toxic air 

pollutants will be reduced 

by 4 million pounds per 

year 

2 Actions or responses 

by regulated parties 

Intermediate outcome At least 100 drinking water 

systems eligible for Drinking 

Water State Revolving Funds 

will have initiated operations 

that protect human health and 

welfare 

At least 100 eligible 

drinking water systems will 

initiate operations to protect 

human health and welfare 

1 Actions by EPA, 

States, tribes, or 

other governmental 

bodies 

Output Prepare final rules for disposal 

of lead-based paint debris and 

establish standards regarding 

hazardous levels of lead in 

paint, dust, and soil 

The lead debris disposal 

rule and lead hazardous 

standards rule will be 

completed by 

September 30, 2000 

R1

 Research and 

development 

Output Develop a conceptual model 

for developing watershed 

assessment techniques that 

would assist local, regional, 

and national environmental 

decision-makers in maintaining 

the ecological integrity of a 

watershed 

A model to assess the 

exposure of wildlife to 

multimedia environmental 

contaminants (i.e., in the 

soil, water, food, and air) 

will be released 

Source: Government Accountability Office Report, GAO/RCED-00-77, EPA Faces Challenges in Developing 

Results-Oriented Performance Goals and Measures, April 2000 

1

 Level “R” was established for research and development efforts to differentiate from other types of environmental 

activities. 

24
 
09-P-0088 

Appendix B 

Agency Response to Draft Report 

December 4, 2008 

MEMORANDUM 

SUBJECT: Draft Audit Report: Measuring and Reporting Performance 

Results for the Pollution Prevention Program Need Improvement 

 Assignment No. 2007-00087 

FROM: James B. Gulliford 

 Assistant Administrator 

TO: Patrick Gilbride, Director 

 Risk and Program Performance 

Office of Inspector General (OIG) 

Thank you for providing an opportunity to review the Draft Audit Report: Measuring and 

Reporting Performance Results for the Pollution Prevention Program Need Improvement and for 

your consideration of the comments we offered on the prior Discussion Draft. 

As noted in our previous comments, we appreciate the quality and extent of OIG’s work 

in researching the Pollution Prevention (P2) Program. While the requests for information were 

extensive, we recognize that these requests reflected the depth of the inquiry and diligence in 

pursuing issues to proper conclusion. 

We appreciate and concur with OIG’s overall finding that the P2 Program is deserving of 

the high rating it received from the Office of Management and Budget (OMB) in its Performance 

Assessment Rating Tool (PART) assessment, which as you know ranks as the third highest 

rating received among all 53 EPA programs evaluated by OMB to date. OPPTS values OMB’s 

PART assessments as opportunities to collaborate in assessing and improving the soundness of 

our programs’ design and management and their ability to deliver meaningful and important 

results. In my view, this result, in combination with OMB’s similarly high ratings for the Office 

of Pollution Prevention and Toxics’ (OPPT’s) Lead Risk Reduction Program and Chemical Risk 

Review and Reduction Program PARTs – also among the top five EPA-rated programs – is 

reflective of OPPT’s long-standing commitment to and recognition of the importance and value 

of results-oriented program design and management. We have developed a strong portfolio of 

quantifiable, results-oriented Government Performance and Results Act (GPRA) and PART 

performance measures. We were early embracers of program Logic Modeling, with models 

developed for all of our programs. We believe in the value of program evaluation, having been 

selected for four of the Office of Policy, Economics and Innovation’s (OPEI) annual Program 

Evaluation Competitions, including two evaluations of elements of the P2 Program. 

25
 
09-P-0088 

We don’t do these things simply to achieve good PART ratings. We do them because we 

genuinely believe that they make our programs better and more effective. Similarly, we believe 

that our engagement with your auditors and the resulting Audit report will contribute 

significantly towards our goal of continuous program improvement. 

We also concur with many of your more detailed findings, and believe that implementing 

all of your recommendations will contribute significantly to the improvement of the P2 

Program’s ability to measure and report on its important results. We appreciate your 

incorporation of many of our comments on the Discussion Draft, and find the quality of this 

Draft Report to be much improved. We continue to have concerns, however, with the way some 

information is presented in the Draft Report. Our concerns again fall into three categories: 

•	 Some minor editorial corrections are needed; 

•	 Some factual information (data references and explanations, etc.) and statements ascribed 

to individuals or program managers in general need correction; 

•	 We disagree with some of your interpretations and associated findings. 

To assist you in finalizing the report, we’ve again compiled a Track Changes version 

showing all of the corrections we think are needed as well as our remaining suggestions for 

further improvement (see P2 PART.Draft OIG Audit Report.OPPTS Suggested Changes.12-04-

08). We continue to recognize that this is OIG’s report, not an OPPTS report, but hope you can 

give additional consideration to this final set of priority concerns in finalizing the report. We’re 

attaching a document providing explanations for all of our substantive concerns/suggestions to 

help you assess them (see P2 PART.Draft OIG Audit Report.OPPTS Change Explantions.12-04-

08.doc). 

Per your request, we are also providing an attachment indicating our concurrence with 

each proposed recommendation, including planned completion dates for all recommendations 

and corrective actions already initiated or planned (see P2 PART.Draft OIG Audit Report.OPPTS 

Concurrences & Schedules.12-04-08.doc). 

In the case of one of the concerns we communicate in the Track Changes and Change 

Explanations documents, regarding your report’s suggestion on page 11 that increased waste 

generation resulting from production/service level increases should count against the program's 

hazardous materials reduction results, we state that this proposed approach: 

"completely distorts the public benefit obtained through the investment of public 

resources. Reviews of performance measurement literature and practices by other 

programs, governments and industry support the P2 Program’s approach on this issue and 

do not support the approach suggested in your language. This issue is critically important 

in supporting sound and cost-effective public policy and investment decisions. If you 

continue to disagree with us on this point, I recommend we conduct an in-person 

discussion and include representatives from other EPA programs that would be 

negatively impacted by the adoption of your suggested approach." 

26
 
09-P-0088 

Since your report’s suggested approach is not manifested in any of its formal 

recommendations, we are not deterred from concurring with them. But we do believe that 

publication of this measurement approach in an OIG Final Report would have negative 

consequences for many EPA programs and believe the issue merits being raised up the 

management chain if we are unsuccessful in resolving it at this stage. 

OPPTS again expresses appreciation for the quality work that went into this report and 

values all of the recommendations. My staff and I are available to assist in further refinement of 

the report into its final form. Please contact Mike Burns (202-564-8273) if additional assistance 

is needed. 

Attachments: 

P2 PART.Draft OIG Audit Report.OPPTS Suggested Changes.12-04-08.doc 

P2 PART.Draft OIG Audit Report.OPPTS Change Explantions.12-04-08.doc 

P2 PART.Draft OIG Audit Report.OPPTS Concurrences & Schedules.12-04-08.doc 

P2 Regional Offices.Region 1 QAPP.10-30-07.doc 

cc: Jim Jones 

 Charles Auer 

 Wendy Cleland-Hamnett 

 Bruce Berkley 

 Thomas Tillman 

 Robert Lee 

 Barbara Cunningham

 Mike Burns 

27
 
09-P-0088 

Draft Office of Inspector General Audit Report: 

Measuring and Reporting Performance Results for the Pollution Prevention Program Need 

Improvement 

Assignment No. 2007-00087 

OPPTS Concurrences with Recommendations and Proposed Implementation Schedules: 

As noted in the OPPTS memorandum responding to Patrick Gilbride, Director, Risk and 

Program Performance, Office of Inspector General (OIG), OPPTS concurs with all of OIG’s 

recommendations presented in its Draft Audit Report: Measuring and Reporting Performance 

Results for the Pollution Prevention Program Need Improvement 

(Assignment No. 2007-00087), and believe that implementing them will contribute significantly 

to the improvement of the Pollution Prevention (P2) Program’s ability to measure and report on 

its important results. 

Specific responses and implementation schedules for each recommendation are provided 

below. 

3-1: Focus on the select P2 pollutants where health effects are known and develop higher 

level performance indicators that address reductions of environmental risks or impacts to 

the ecology or human health. 

OPPTS concurs with this recommendation: 

o	 By March, 2009, the P2 Program will develop an initial indicator tracking one or 

more health effects associated with at least one chemical that the program is 

currently focusing on. Likely candidates for this initial indicator development are 

diisocyanates used in auto refinishing operations. Diisocyanates are the leading 

cause of occupational asthma and also a skin sensitization concern. The P2 

Program is already tracking reductions in the use of these chemicals in these 

settings under its current Level 3 outcome measure targeting reductions in pounds 

of hazardous materials. 

o	 Commencing in 2009 after the initial higher level indicator is developed, the P2 

Program will gather data to support tracking the indicator on an annual basis. 

o	 In future years, the program will explore whether other higher-level could also be 

developed based on this model. 

3-2: Using the results of Recommendation 3-1, design a strategy for developing P2-wide 

PART higher level measurements that could be used as a model to further develop P2 

Program-wide higher level measurements. 

OPPTS concurs with this recommendation: 

28
 
09-P-0088 

o	 The P2 Program will include in EPA’s September, 2009, submission to Congress 

of its 2014 Strategic Plan at least one new higher-level Strategic Measure with an 

associated long-term performance target, drawing on the information developed in 

support of the higher level indicator to be developed under Recommendation 3-1 

above. 

o	 The P2 Program will subsequently propose adding the new Strategic Measure as a 

PART Long-Term measure in the 2009 Fall PART Update, and propose 

associated Annual performance targets needed to achieve the Long-Term target. 

o	 In future years, the P2 Program will expand existing performance targets to reflect 

expanded implementation across the program’s Centers, and assess opportunities 

for developing additional higher-level measures based on this model. 

3-3: Report program accomplishments as they relate to the universe for the targeted 

chemical in the specified uses addressed by the P2 Program to provide a better perspective 

of the program’s relative effectiveness. 

OPPTS concurs with this recommendation, having previously at our own initiation 

commenced taking steps towards its implementation: 

o	 September, 2008: OPPTS included a proposed new Pollution Prevention Program 

Strategic Measure in EPA's 2014 Strategic Plan Change Document, circulated for 

public comment, tracking the program's results in reducing production and use of 

a priority set of hazardous chemicals which, once defined through EPA's 

Chemical Assessment and Management Program (ChAMP) and other prioritysetting mechanisms, will enable the program to define associated universe(s) that 

will provide the recommended perspective on the program's relative effectiveness. 

o	 Spring, 2009: OPPTS/OCFO will determine whether to include the proposed 

measure in public comment drafts of the Strategic Plan and whether to submit to 

OMB as formal PART measures, making necessary refinements as understanding 

improves regarding the priority setting processes and the measurement mechanics. 

o	 September, 2009: Final Long-Term Measures will be included in the Final EPA 

2014 Strategic Plan submitted to Congress. 

o	 Fall, 2009: Final Long-Term and Annual Measures will be submitted to OMB for 

incorporation as formal PART Measures 

4-1: Revise the P2 Program procedures for calculating program results to collect and use 

current year data from its voluntary partners to calculate annual program results. 

OPPTS concurs with this recommendation, having previously at our own initiation 

commenced taking steps towards its implementation: 

29
 
09-P-0088 

o	 September 3, 2008: Conducted consultation with EPA's Science Advisory 

Boards' Environmental Engineering Committee to obtain advice on how to more 

fully reflect the Pollution Prevention Program's benefits by measuring its 

recurring as well as new annual results. 

o	 October/November, 2008: Developing proposed Pollution Prevention Program 

Strategic Measures for consideration for inclusion in the EPA 2014 Strategic Plan 

and as Long-Term and Annual PART Measures, clearly distinguishing measures 

tracking new annual results from measures that include recurring results 

stemming from prior years' program interventions. 

o	 Spring, 2009: OPPTS/OCFO will determine which of the above measures to 

include in public comment drafts of the Strategic Plan and which to submit to 

OMB as formal PART measures. 

o	 September, 2009: Final Long-Term Measures will be included in the Final EPA 

2014 Strategic Plan submitted to Congress. 

o	 Fall, 2009: Final Long-Term and Annual Measures will be submitted to OMB for 

incorporation as formal PART Measures. 

4-2: Require the development of QAPPs for collecting environmental data. 

OPPTS concurs with this recommendation, having previously at our own initiation 

commenced taking steps towards its implementation: 

o	 OPPT’s Pollution Prevention Division (PPD) recently revised and updated its 

Quality Management Plan to document the procedures they have implemented, in 

part, to respond to the PART assessment’s findings. While not officially 

completed at the time of the OIG audit review and unavailable for distribution, it 

has now been officially approved by the Agency. 

o	 A number of P2 programs use and report performance results data generated by 

organizations outside EPA. To prevent inappropriate and inconsistent use of 

these data, OPPT is in the process of developing data quality criteria that will be 

provided to those organizations to ensure that future data submitted to EPA 

comply with these requirements and that the data can appropriately be used by 

EPA. The development of QAPPs for contractors and grantees which engage in 

environmental data activities has been a tenet of the OPPT Quality Assurance 

policy for a number of years and all P2 Program Centers have been in compliance 

with this policy. However, outside organizations that voluntarily have submitted 

data to the P2 Program have generally not submitted associated QAPPs, which 

document the procedures they followed to collect, use and report results, with 

these data. OPPT is currently developing procedures (see response to 

Recommendation 4-3) to ensure these organizations provide the appropriate 

documentation with the data for future submissions. For example, the FY 2009 

30
 
09-P-0088 

P2 Grant Program is establishing “appropriate means for measuring the 

effectiveness of the P2 Grant Program by promoting the use of source reduction 

techniques by businesses.” To qualify for a grant, applicants must provide 

qualitative and quantitative estimates of expected outcomes and outputs of project 

activities through an approved plan. This will comply with EPA Policy Order # 

5700.7 (Environmental Results Policy). 

o	 The Design for Environment and Green Chemistry Centers have data validation 

procedures for information submitted by partners and in nominations for program 

awards. For the Green Chemistry Awards, nominations are peer reviewed by an 

independent panel for formal judging. This panel identifies data that need further 

verification and validation by EPA prior to the award. In addition, those who 

apply for and receive awards are required to follow a set of procedures to ensure 

the quality of the information submitted by the nominees. The program will 

continue these effective procedures in future years. As noted in the response to 

Recommendation 4-3, Standard Operating Procedures (SOPs) and QAPPs as 

warranted will be developed to specifically address areas where additional data 

verification and validation are needed and will be in place for the next round of 

awards in 2009. 

4-3: Implement controls to ensure full implementation of the Pollution Prevention QMP. 

OPPTS concurs with this recommendation, having previously at our own initiation 

commenced taking steps towards its implementation: 

o	 As noted in Recommendation 4-2, the new PPD QMP is now officially approved. 

However, implementation of the QMP has been ongoing and is well underway. 

To ensure that the procedures documented in the QMP are properly followed, 

OPPT is in the process of developing a set of Standard Operating Procedures 

(SOPs). These procedures are analogous to QAPPs for routine or repetitive 

activities conducted by an organization, that will specify the data quality 

objectives, data standards, including reporting formats, and data assessment 

procedures that EPA needs for its use of data for performance measurement and 

reporting and, as warranted, for dissemination to the public. These SOPs will be 

used in developing program specific QAPPs that will include data assessment and 

validation procedures to ensure that the data that are submitted comply with these 

requirements. 

o	 OPPT will work closely with grantees and voluntary submitters of performance 

results data to ensure that these data are appropriate for their intended use and are 

accompanied by documentation that addresses the procedures followed to collect, 

use and report them. These procedures will be in place for the next round of 

awards or data submission cycle in 2009. 

31
 
 

09-P-0088 

5-1: Have the P2 Program plan and schedule periodic evaluations of its pollution 

prevention efforts to identify areas where program improvements are needed and 

implement needed corrective actions. 

OPPTS concurs with this recommendation, having previously at our own initiation 

commenced taking steps towards its implementation: 

o	 The P2 Program has conducted two extensive program evaluations addressing two 

of its Centers of Results over the past three years. Both evaluations were 

sponsored by EPA's Office of Policy, Economics and Innovation (OPEI) through 

its annual Program Evaluation Competition. The first focused on the Hospitals 

for a Healthy Environment Program, the forerunner to the Partnership for 

Sustainable Healthcare, and was completed in May 2006. The second focused on 

the Pollution Prevention Information Network Grant Program, which supports the 

Pollution Prevention Resource Exchange (P2Rx) centers, and was completed in 

August 2008. Both evaluations were completed on schedule and within budget, 

producing reports documenting program effectiveness and conveying 

recommendations for program improvements. 

o	 Fall 2008: OPPTS is currently assessing the recommendations from the report 

evaluating the P2Rx program to determine which recommendations to implement 

and in what priority. 

o	 Winter/Spring 2009: OPPTS will begin implementation of recommendations 

from the P2Rx evaluation. 

o	 Spring 2009: OPPTS will target another one of its Centers of Results to submit 

an application for consideration under OPEI's Program Evaluation Competition. 

Note: Since the OPEI-sponsored evaluations are not conducted by entities considered 

independent of EPA, OMB criteria do not allow the evaluated programs to receive credit 

for them as program evaluations in PART assessments. However, the cost of conducting 

an independent program evaluation that meets OMB's criteria is prohibitive for the P2 

program, and OPPTS believes it has obtained the full benefits offered by program 

evaluations in its two previous EPA evaluations. 

5-2: Develop actions to address the PART questions where the P2 Program was not 

awarded any points during the FY 2006 PART Assessment and include the PIP in the next 

PART update to promote continual P2 Program improvements. 

OPPTS concurs with this recommendation, having already addressed some of the 

identified weaknesses under previously completed or currently active PART Follow-Up 

Actions. Commencing with the 2009 Spring PART Update, OPPTS will ensure that all 

proposed replacements for Completed Follow-Up Actions are targeted at unaddressed 

weaknesses identified in the Pollution Prevention Program PART Assessment until such 

time that all such weaknesses have been fully addressed or, in the case of the PART 

32
 
09-P-0088 

requirements for independent program evaluations, addressed to the extent deemed 

practicable given program resource allocations and influence over decisions made by 

independent evaluating entities. 

o	 Spring, 2009: Propose replacement Follow-Up Action for current to-becompleted Action #1. 

o	 Fall, 2009: Propose replacement Follow-Up Actions for current to-be-completed 

Actions #5. 

o	 Spring, 2010: Propose replacement Follow-Up Action for future to-be-completed 

Actions. 

33
 
09-P-0088 

Appendix C 

Distribution 

Office of the Administrator 

Acting Assistant Administrator, Office of Prevention, Pesticides, and Toxic Substances 

Acting General Counsel 

Agency Follow-up Official (the CFO) 

Agency Follow-up Coordinator 

Acting Director, Office of Pollution Prevention and Toxics 

Acting Associate Administrator for Congressional and Intergovernmental Relations 

Acting Associate Administrator for Public Affairs 

Audit Follow-up Coordinator, Office of Prevention, Pesticides, and Toxic Substances 

Deputy Inspector General 

34