U.S. ENVIRONMENTAL PROTECTION AGENCY OFFICE OF INSPECTOR GENERAL

Catalyst for Improving the Environment


## Audit Report



# ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate With Additional Improvements

Project No. 10-P-0230

September 22, 2010Abbreviations

DMR Discharge Monitoring Report ECHO Enforcement Compliance and History Online EPA U.S. Environmental Protection Agency GPRA Government Performance and Results Act of 1996 ICIS-NPDES Integrated Compliance Informa tion System – National Pollutant Discharge Elimination System IDEF Interim Data Exchange Format OECA Office of Enforcement and Compliance Assurance OIG Office of Inspector General OMB Office of Management and Budget PCS Permit Compliance SystemU.S. Environmental Protection Agency 10-P-0230 September 22, 2010 Office of Inspector General 

# At a Glance

Catalyst for Improving the Environment

Why We Did This Review

We sought to assess the quality of key data elements reported through the Enforcement Compliance and History Online (ECHO) Website. KPMG, LLP, performed the review.

Background

ECHO provides a single source of detailed compliance history of U.S. Environmental Protection Agency (EPA)

- regulated facilities. EPA developed ECHO to provide the public compliance and inspection data under its environmental programs, as well as demographic data of the surrounding areas. This report focuses on the quality of data elements entered into ECHO source systems.

For further information, contact our Office of Congressional, Public Affairs and Management at (202) 566-2391.

To view the full report, click on the following link: www.epa.gov/oig/reports/2010/ 20100922-10-P-0230.pdf 

## ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate With Additional Improvements

What KPMG Found

EPA mandates that data elements re ported to the public through the ECHO Website have a 95 percent accuracy rate . KPMG found a 91.5 percent data accuracy rate for key data elements entered into two primary ECHO source systems: the legacy Permit Compliance System (PCS) and the newer Integrated Compliance Information System – National Pollutant Discharge Elimination System (ICIS-NPDES). Although the 91.5 percent data quality rate is close to EPA’s goal, EPA and the State environmen tal offices could take additional steps to increase the quality of data reported through the ECHO Website.

What KPMG Recommends

KPMG made several recommendations to the Assistant Administrator for Enforcement and Compliance Assurance. These included:

 Establishing an internal control struct ure to help manage the conversion of PCS to ICIS-NPDES.

 Including language in the National Program Manager Guidance requiring the use of the Environmental Inform ation Exchange Network for reporting data to EPA.

 Developing a plan to share data quality best practices implemented at State environmental offices with all States.

 Completing new rules requiring reporting ECHO data for minor facilities and notifying ECHO Website users that the site does not contain data on minor facilities.

 Reviewing procedures used to t est ICIS-NPDES programming code before it is placed into production.

The Agency generally agreed with the report findings. EPA felt it has extensive, documented procedures in place to test ICIS-NPDES programming code before it is placed into production. Tests disclos ed that programming errors directly resulted in incomplete data on the EC HO Website and, as such, management should review these processes to prevent future occurrences. The Agency’s response is included in Appendix A.UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460

THE INSPECTOR GENERAL

September 22, 2010

MEMORANDUM

SUBJECT: ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate With Additional Improvements Report No. 10-P-0230

FROM: Arthur A. Elkins, Jr. Inspector General

TO: Cynthia Giles Assistant Administrator for Enforcement and Compliance Assurance

This is the report on the subject audit conducte d by KPMG, LLP, on behalf of the Office of Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report contains findings KPMG LLP identified and corrective actions recommended. This report represents the conclusion s of KPMG and does not necessarily represent the fina l position of the OIG on the subjects reported. Final determinatio ns on matters in this report will be made by EPA management in accordance with established audit resolution procedures.

The estimated cost of this audit, which includ es contract costs and OI G contract management oversight, is $331,361.

Action Required

In accordance with EPA Manual 2750, you are require d to provide a written response to this report within 90 calendar days. You should incl ude a corrective actions plan for agreed-upon actions, including milestone dates. Your respon se will be posted on the OIG’s public Website, along with our comments on your response. Your response should be provided in an Adobe PDF file that complies with th e accessibility requirements of s ection 508 of the Rehabilitation Act of 1973, as amended. If your response contains data that you do not want to be released to the public, you should identify the data for redaction. We have no objections to the further release of this report to the public. This report will be available at http://www.epa.gov/oig

.
If you or your staff have any questions regarding this report, please contact Patricia H. Hill at (202) 566-0894 or hill.patricia@epa.gov , or Rudolph M. Brevard at (202) 566-0893 or brevard.rudy@epa.gov

.September 22, 2010

SUBJECT: Audit Report: ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve Data Quality Rate With Additional Improvements

THRU: Arthur A. Elkins, Jr. Inspector General

TO: Cynthia Giles Assistant Administrator Office of Enforcement and Compliance Assurance

This memorandum is to inform the U.S. Enviro nmental Protection (EPA) of critical findings by KPMG, LLP, that require management action rega rding the improvement of the data quality for the Enforcement Compliance History Online Website.

If you or your staff have any questions re garding this report, please contact the U.S. Environmental Protection Agency, Office of Inspector General, point of contact identified on the report transmittal memorandum.ECHO Data Quality Audit - Phase 2 Results: 10-P-0230 EPA Could Achieve Data Quality Rate With Additional Improvements



# Table of Contents

Purpose

........................................................................................................................ 1

Background

................................................................................................................. 1

Scope and Methodology

............................................................................................. 2

Findings

....................................................................................................................... 4

Identified Data Quality Exceptions

............................................................................ 4

Factors That Contributed to Data Quality Exceptions

............................................. 5

No Automated Data Quality Processes

................................................................... 5 No Consistent State Quality Review Processes

..................................................... 6 No Standardized DMR Form

................................................................................... 6 No Documentation of Data Entry

............................................................................ 7 No Requirements to Report Minor Facility Data

...................................................... 7

Michigan Data Processing

......................................................................................... 7

Data Quality Exceptions Affect EPA’s Representation of Data

.............................. 8

Recommendations

...................................................................................................... 8

Agency Comments and OIG Evaluation

................................................................... 9

Status of Recommendations and Potential Monetary Benefits

.............................. 10



## Appendices

A Agency Response

.............................................................................................. 11

B Distribution

......................................................................................................... 1810-P-0230



### Purpose

The U.S. Environmental Protection Agency (EPA) Office of Insp ector General (OIG) initiated this review to evaluate the quality of data reported through the EPA Enforcement Compliance and History Onlin e (ECHO) Website. KPMG LLP (KPMG) completed the review in the following two phases:

 Phase 1: Evaluate the automated process EPA uses to display ECHO data transmitted from select source systems. We reviewed the automated processes for the legacy Permit Compliance System (PCS) and the newer Integrated Compliance Information System – Nati onal Pollutant Discharge Elimination System (ICIS-NPDES). We provided the Phase 1 results in OIG report, ECHO Data Quality Audit - Phase I Results: The Integrated Compliance Information System Needs Security Controls to Protect Significant Non-Compliance Data

, Report No. 09-P-0226, August 31, 2009.

 Phase 2: Test the quality of select PCS and ICIS-NPDES data elements against Discharge Monitoring Reports (DMRs), wh ich are the primary source documents for PCS and ICIS-NPDES data.



### Background

ECHO provides a single source of detailed and up-to-date environmental compliance history of EPA-regulated facilities. EPA developed ECHO in partnership with State environmental offices to provide compliance and inspection data under air, water, and hazardous waste programs, as well as demographic data of the surrounding areas. Since ECHO’s November 2002 launch, hundreds of thousands of citizens, government officials, investors, and st aff at regulated facilities and companies have asked over 3,000,000 questions about the environmental record s of the more than 800,000 facilities. EPA and the state environmental offices need to maintain a high level of ECHO data quality to: 1) provide accurate and complete information to the public, States, facilities, and other stakeholders, and 2) maintain their reputations as effectiv e public stewards and information providers.

The Government Performance and Results Ac t of 1996 (GPRA) require s federal agencies to provide managers with accurate informati on about program results and service quality. EPA’s 2009 Performance and Accountability Report (PAR), developed in accordance with GPRA, specifically identifies data qual ity as a key element of EPA’s efforts to improve management and results.

The Office of Management and Budget (OMB) Circular A-130, Management of Federal Information Resources , authorized in part by GPRA and the Paperwork Reduction Act, guides that federal agencies should ensure that systems maximize the usefulness of information, minimize the burden on the public, and preserve the a ppropriate integrity,

110-P-0230

usability, availability, and conf identiality of information th roughout the life cycle of the information.

EPA’s Permit Compliance System (PCS) Quality Assurance Guidance Manual states:

 It is very important that PCS and NPDE S data be complete, accurate, and up-to

date, and that all users be consistent in the way they define and use various data elements because the systems play such a central role in program management.

 Quality is achieved through effective management and comm itment. Responsive management should provide the support and training that is necessary to achieve good data quality; and recognize, rewar d, and encourage quality service and performance.

 Quality must be tracked and performan ce of the program evaluated at regular intervals. Data quality must be measurable so that the causes of poor data quality can be identified and corrected.

 In order for the quality of PCS data to be “good,” the data must meet quality targets for timeliness, accuracy, completeness, and consistency. For example, a quality target for accuracy is that 95 percen t of the data elements in the system of record be identical to the source document . The failure to meet these targets is especially serious when the data elements affect public access.

 Ninety-five percent of the data elements entered into PCS should be identical with those reported on the DMR permit or other input document.

EPA’s PCS Policy Statement states that to achieve nationa l consistency and uniformity in the NPDES program, the required data in PCS must be complete and accurate.

The Office of Compliance is responsible for identifying, preventing, and reducing noncompliance and environmental risks by pr oviding leadership in program planning, priority setting, and expanding the use of comp liance data. In this regard, the office’s Enforcement Targeting and Data Division ( ETDD) supports EPA’s national enforcement and compliance information and reporting need s and manages the ECHO Internet site.


### Scope and Methodology

We coordinated with the OIG to select major high-risk ecosystems and corresponding discharge facilities as the population for our testing. We and the OIG selected the major ecosystems and facilities, in part, based on the EPA’s 2006-2011 Strategic Plan , which identifies high-risk ecosystems in the Unite d States. For the selected ecosystems and facilities, we reviewed 2 008 calendar year DMR data ex tracted from PCS and ICIS– NPDES.

210-P-0230

We performed this audit in accordance with Generally Accepted Government Auditing Standards issued by the Comptroller Genera l of the United States. These standards require that we plan and perform the audit to obtain sufficient and appropriate evidence to provide a reasonable basis for their findi ngs and conclusions based on the audit objectives. We believe the evidence obtained provides a reasonable basis for our findings and conclusions.

To assist in selecting the ecosystems, we used the EPA Website, which provided a map of watersheds and Hydrologic Unit Codes, as well as the ECHO map feature. We then selected a judgmental sample of faciliti es supporting each ecosystem. For these facilities, we compared data elements reported in ECHO to related DMR records. We included the following types of da ta elements in our review:

 Facility name

 Permit number and approval

 Pollutant levels

 Facility status (e.g., major or minor)

 Specific DMR data (e.g., types of pollutants)

We focused our review on the more critical data elements, such as pollutant levels and facility status, rather than less critical el ements, such as facility addresses and phone numbers. Our testing was based on a j udgmental sample, and the results are not statistically valid and should not be projected to other States or to all facilities in the States we visited (for the State of Florida, we did te st the entire population of 41 nonFederal managed facilities).

We conducted the following site visits:

Table 1: State Environmental Offices Visited During the Review

State Visited ECHO Source System Used Dates of Visit Ecosystem No. of Facilities Reviewed

Maryland ICIS-NPDES November 2 – 10, 2009 Chesapeake Bay 50 (sample)

Indiana ICIS-NPDES November 16 – 20, 2009 Great Lakes 50 (sample)

Michigan PCS December 7 – 11, 2009 Great Lakes 50 (sample)

Florida PCS January 11 – 15, 2010 South Florida 42 (entire population)

Source: KPMG-compiled data

310-P-0230



### Findings

EPA mandates that data elements reported to the public through the ECHO Website have a 95 percent accuracy rate. During this review, we found a 91.5 percent data accuracy rate for key data elements entered into two primary ECHO source systems - the legacy PCS and newer ICIS-NPDES. Although the 91.5 percent data quality rate is close to EPA’s goal, EPA and the State Environmental Offices can make additional improvements to further increase data quality. Specifically, we identified several types of data quality exceptions and a processing issue in the State of Michigan that affected the quality of data ultimately presented thr ough ECHO. We cite them on the following
pages.


### Identified Data Quality Exceptions

During each of our four site visits to State environmental offices, we found four types of data element exceptions, as follows:

 Data elements were reported in ECHO but not in DMRs. In these cases, we noted that data elements were reported through ECHO but the supporting source DMRs did not reflect the same data element information.

 Data elements were reported in DMRs but not in ECHO. In these cases we noted that data elements were reported in th e source DMRs (e.g., pollutant values), but the same data element informati on was not reported through ECHO.

 Data elements were inaccurately input into ECHO source systems (PCS or ICIS

NPDES) from DMRs. For example, at one facility, the pollutant phosphorous had a value of 1.7 in ECHO but a value of 0.7 on the supporting DMR. At another facility, the pollutant nitrogen had a va lue of 2.64 in ECHO but a value of 2.22 on the DMR.

 Data elements had missing DMRs. In these cases, we were not able to review the applicable source DMRs because they were not available at the state environmental offices.

410-P-0230

The following table summarizes the data quality exceptions by state.

Table 2: Data Quality Exceptions

State No. of Data Elements

Tested Data Elements in ECHO Not Supported by DMRs Data Elements on DMRs Not Entered Into ECHO Data Elements Inaccurately Entered Into Source Systems From DMRs Data Elements With Missing

DMRs Total Data Elements With Exceptions Exception % (By State)
Maryland 6,199 94 386 174 133 787 12.7%

Indiana 8,850 297 16 16 8 337 3.8%

Florida 12,590 62 501 85 565 1213 9.6%

Totals 27,639 453 903 275 706 2,337 8.5%

Source: KPMG-compiled results from documentation review

Michigan had data quality exceptions in the sa me four categories. However, Michigan’s results are not included in Table 2 because we were not able to fully quantify the exceptions. Michigan had a data processing issue that resulte d in many data elements not being displayed in ECHO. See the Michigan Data Processing section later in this report for details.

We provided detailed data quality exceptions to the visited State environmental offices, in which we identified the facility names and types of exceptions identified at each facility.



### Factors That Contributed to Data Quality Exceptions

No Automated Data Quality Processes

The DMR data entry process is a manual-orie nted process for many State environmental offices, which significantly elevates the risk of data exceptions. The EPA 2009 PAR addresses this issue in part by stating, “dat a quality has been improving and will continue to improve as existing data entry requiremen ts and procedures are being reevaluated and communicated with data entry practitioners.” The PAR further states that in FY 1999 EPA acknowledged PCS as a weakness and r ecognized that the Agency needed to revitalize or replace PCS to ensure complete and accurate water data. EPA has been continually enhancing DMR data entry and quali ty review processes, including efforts to (1) convert from PCS to ICIS -NPDES, (2) increase use of the Interim Data Exchange Format (IDEF) quality review process, and (3) implement eDMR and NetDMR automation capabilities. According to EPA officials, competing resources have limited full implementation of these enhancements.

510-P-0230

Regarding the conversion from PCS to IC IS-NPDES, the Office of Enforcement and Compliance Assurance (OECA) Office of Comp liance, with support from the Office of Water and the Office of Environmental Inform ation, has been working with States and regions to develop data require ments, design the new system, test the developed software, and prepare PCS data for migr ation. Now 28 States, 2 tribes , and 10 territories are using ICIS-NPDES instead of PCS. The full migration from PCS to ICIS-NPDES is scheduled for completion in FY 2013. These and other au tomation efforts are cr itical to improve data quality. The Environmental Information Exchange Network estimates that “electronic DMR reporting has th e highest documented return on investment of any other EPA data automation project - implementati on by all or most NPDES facilities could save industry, states, and EP A around $100 million per year.” 1

Using eDMR and NetDMR capabilities will a ssist with Full Batch State conversion. OECA can accelerate the Full Batch Stat e conversion by using the Environmental Information Exchange Network as a preferred method of transmitting data to the Agency. OECA’s FY 2010 & 2011 National Program Ma nager Guidance does not contain any language or plans for using the Environmenta l Information Exchange Network. In this regard, more liaison with resp ective EPA offices would help ensure Agency grants given to States would more effectivel y align with EPA data.

No Consistent State Qu ality Review Processes

Closely linked to the above factor, ECHO data quality e fforts are reliant upon resource capabilities in EPA and the State environmenta l offices. During our site visits, we noted that the States’ data quality resources vary. For example, the State of Maryland has one dedicated person focused on the ECHO data qua lity program, while the State of Indiana had four data entry personnel and three data qua lity reviewers. As noted earlier in Table 2, Indiana had the smallest number of exceptions.

EPA’s IDEF function provides th e State environmental office s with the capability to audit their submission of ECHO data for comp leteness and accuracy, but the States are not consistently reviewing IDEF. In fact, St ate environmental office officials confirmed that some of the data quality findings identified during this review would not have been identified through existing EPA data quality processes, such as IDEF.

No Standardized DMR Form

State environmental offices use various type s of DMR forms for data entry. Some of these offices use their own State-specific repor ting forms, and forms can vary within the same State.

EPA’s Permit Compliance System (PCS) Quality Assurance Guidance Manual states:

1 http://www.exchangenetwork.net/b enefits/NetDMR_SuccessStory.pdf

610-P-0230

All DMRs submitted to EPA Regional Offices (including DMRs submitted by NPDES States for EPA entry into PCS) must be preprinted using the Office of Management and Budget (OMB) approved DMR form. NPDES States directly using PCS are not required to use the OMB-approved fo rm; however, its use is strongly encouraged.

Although some similarity exists between the various input forms, the lack of full standardization increases the risk of data quality exceptions. Enhanced automation with ICIS-NPDES, eDMR, and NetDMR could improve this issue. There is a standard form for entering NPDES data (OMB Form Number 2040-0004, National Pollutant Discharge Elimination System (NPDES) Discharge Monitoring Report ), but we found that not all State environmental offices were consistently using this form.

No Documentation of Data Entry

State environmental office officials informed us their data quality personnel sometimes will input data into PCS or ICIS-NPDES but not record the information in supporting documentation (e.g., DMRs, permits). The PCS Quality Assurance Manual states that the regions should work closely with their NPDE S States using PCS to address similar data entry problems with State- issued NPDES permits. The primary reason for not updating the source documentation was that the State officials believed the ch anges were so minor they would not affect overall data quality.

No Requirements to Report Minor Facility Data

Existing legislation does not require regulated facilities to send EPA all minor facility data elements displayed in ECHO. Only subm ission of major facility data elements is required.

EPA’s Permit Compliance System (PCS) Quality Assurance Guidance Manual states:

If States and Regions wish to enter NPDES data beyond what has been required, they may do so. For example, if States want to enter DMR data for minor facilities, the options is available in PCS and the States may use it as their resources allow.

According to EPA officials, a proposed rule is being developed to require States and regulated facilities to provide additional minor facility data. EPA expects to have the rule finalized during FY 2012.


### Michigan Data Processing

During our site visit to Michigan, we identi fied a data processing error where DMR data for 18 facilities were transmitted to PCS but the data was not completely populated into ECHO. Specifically, ECHO did not completely receive PCS data fo r facilities with

710-P-0230

multiple discharge points. EPA had previously identified this issue, as the ECHO Data Alerts Website ( http://www.epa-echo.gov/echo/data_alerts.html ), last updated on July 6, 2009, identified the issue and stated that EPA is working with the State of Michigan to resolve the issue. ETDD determined that soft ware code used to pull data elements from PCS into ECHO was not pulling all necessa ry data. ETDD developed, tested, and implemented new software code to correct the issue. The processing error contributed to a significant number of 2008 da ta quality exceptions for Michigan. As noted in the Permit Compliance System (PCS) Quality Assurance Guidance Manual , PCS data must be complete and accurate to assist EP A with program management activities.

While these efforts correct the identified i ssues with the Michigan data, ultimately non

compliance with prescribed EPA System Management Life Cycle (SMLC) Guide contributed to the code being placed in to production without testing. EPA SMLC guidance states that the Test Subphase results in proof th at the system satisfies the requirements. As such, internal controls over system testing woul d have identified the error in the Michigan pr ogramming logic before it was placed into production.


### Data Quality Exceptions A ffect EPA’s Representation of Data

The exception rates increase the risk that ECHO data presented to internal EPA stakeholders and the public is not accurate. EPA’s Permit Compliance System (PCS) Quality Assurance Guidance Manual notes that PCS and ICIS-NPDES data must be complete and accurate to assist EPA with program management ac tivities. Further, inaccurate ECHO data could misrepresent f acility information, so data accuracy is significant for EPA and State environmenta l offices in their roles as good public stewards.

EPA officials informed us that the Agency pa rtially mitigates the risk of data exceptions by relying on the public and facilities to no tify the Agency of inaccurate data. The EPA 2009 PAR emphasizes this point by stating, “an important means fo r catching potential data errors is to obtain stakeholder feedbac k. EPA has a formal error correction process in place to investigate and react to public notif ication of errors in it s publications.” While relying on the public serves as a detective co ntrol, it is still incumbent upon EPA and the State environmental offices, as good public stew ards, to have more effective preventive controls in place to better ensure data quality.



### Recommendations

We recommend that the Director, Office of Compliance, within the Office of Enforcement and Compliance Assurance:

1. Establish a management c ontrol structure to facil itate PCS to ICIS-NPDES conversion. The management control st ructure should support plans to meet milestone dates.

810-P-0230

2. Insert language that requires the use of the Environmental Information Exchange Network in the National Program Manager (NPM) Guidance to assist with PCS to ICIS-NPDES conversion for FY 2012.

3. Gather key leading data qua lity practices used by the States and share these practices with all States.

4. Provide reminders to EPA regional offices and State environmental offices to use the standard Form 2040-0004 for data entry. Where this is not feasible (e.g., for State-specific reporting requirements), EP A should require completely reporting DMR data by using standardized Form 2040-0004 or alternative entry forms. Further, EPA should remind the States of the importance of documenting changes made to data elements after entry into the source systems.

5. Complete new rules that require States to report minor facility data.

6. Place a notice on the EPA ECHO Website st ating that ECHO does not contain all minor facility data. The wording and pl acement of the notice should be designed to help provide the public with transparency into the actual data elements maintained in ECHO.

7. Conduct a review of the procedures used to test ICIS-NPDES programming code before it is placed into production.


### Agency Comments and OIG Evaluation

EPA agreed with the report findings. EP A felt that it has extensive, documented procedures in place to test ICIS-NPDES programming code before it is placed into production. Management cited that they follo w best practices used by the information technology industry as well as EPA system lif e cycle policies and procedures and that their primary contractor is certified by the So ftware Engineering Institute to be Level 3 on the Capability Maturity Model Integrati on scale. Management also stated their procedures have been refined and improved ove r the years, and, because of this, new and revised software is typically implemented into production with few, if any, defects.

We agree that management should use industr y best practices whenever feasible and adopt an industry recognized method for c ontrolling system development activities. However, we believe as part of an effectiv e internal control structure management must continually review these practices to ensure they operate as intende d. For the State of Michigan, EPA recognized the error in the State’s ECHO data and it took the Agency close to one year to make changes to the programming code to correct the problem. As such, we believe it is incumbent upon manage ment to conduct a root cause analysis to identify the causes for the programming code error and update the testing procedures to prevent similar errors from happening in the future.

910-P-0230



## Status of Recommendations and Potential Monetary Benefits

POTENTIAL MONETARY RECOMMENDATIONS BENEFITS (in $000s)

Rec. No.

1

2

3

4

5

6

7 Page No.

8

9

9

9

9

9

9 Subject

Establish a management control structure to facilitate PCS to ICIS-NPDES conversion. The management control structure should support plans to meet milestone dates.

Insert language that requires the use of the Environmental Information Exchange Network in the National Program Manager (NPM) Guidance to assist with PCS to ICIS

NPDES conversion for FY 2012.

Gather key leading data quality practices used by the States and share these practices with all States.

Provide reminders to EPA regional offices and State environmental offices to use the standard Form 2040-0004 for data entry.Where this is not feasible (e.g., for Statespecific reporting requirements), EPA should require completely reporting DMR data by using standardized Form 2040-0004 or alternative entry forms. Further, EPA should remind the States of the importance of documenting changes made to data elements after entry into the source systems.

Complete new rules that require States to report minor facility data.

Place a notice on the EPA ECHO Website stating that ECHO does not contain all minor facility data. The wording and placement of the notice should be designed to help provide the public with transparency into the actual data elements maintained in ECHO.

Conduct a review of the procedures used to test ICIS-NPDES programming code before it is placed into production. Status 1

O

O

O

O

O

O

O Action Official

Director, Office of Compliance, Office of

Enforcement and Compliance Assurance

Director, Office of Compliance, Office of

Enforcement and Compliance Assurance

Director, Office of Compliance, Office of

Enforcement and Compliance Assurance

Director, Office of Compliance, Office of

Enforcement and Compliance Assurance

Director, Office of Compliance, Office of

Enforcement and Compliance Assurance

Director, Office of Compliance, Office of

Enforcement and Compliance Assurance

Director, Office of Compliance, Office of

Enforcement and Compliance Assurance Planned Completion Date Claimed Amount Agreed To Amount

O = recommendation is open with agreed-to corrective actions pending C = recommendation is closed with all agreed-to actions completed U = recommendation is undecided with resolution efforts in progress

1010-P-0230



### Appendix A



## Agency Response

UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460

August 20, 2010 OFFICE OF ENFORCEMENT AND COMPLIANCE ASSURANCE MEMORANDUM

SUBJECT: Response to the Office of the Inspector General Draft Evaluation Report, “ECHO Data Quality Audit – Phase 2 Results: EPA Could Achieve Data Quality Rate with Additional Improvements” Proj ect Number OMS-FY09-00011, dated July 28, 2010

FROM: Cynthia Giles /s/ Assistant Administrator

TO: Rudolph M. Brevard Director, Information Resources Management Assessments Office of Mission Systems Office of Inspector General

Thank you for the opportunity to review and comment on the draft evaluation report entitled, “ECHO Data Quality Audit – Phase 2 Results: EPA Could Achieve Data Quality Rate with Additional Improvements,” (Report) Project Number OMS-FY09-00011. This Report focuses on improving the quality and transpar ency of the enforcement and compliance information EPA provides to the public through our Enforcement and Compliance History Online (ECHO) website.

OECA agrees that providing high quality tran sparent data to the American public is a fundamental responsibility of the government. Recently, OECA has taken several steps to expand transparency of enforcement and complia nce information through ECHO with the release of the Clean Water Act Annual Noncompliance 2008 Report and the enhanced Air, Water and Waste Resource pages.

Additionally, OECA is expanding our efforts to ensure the accuracy of the information in ECHO. A key new resource we are providing the public is consolidated information by program area (Clean Water, Clean Air and Hazardous Waste) . Included on these pages are reports such as the “2009 Annual CAA Majors Report” and the “ 2009 State Review Framework Frozen National Report”. Prior to updating these a nnual reports, we provide the States an opportunity to verify the data and provide additional information. OECA is working with our State partners to ensure that the public receives the best information available.

1110-P-0230

Report Recommendation 1:

 Establish a management control structur e to facilitate PCS to ICIS-NPDES conversion. The management control st ructure should support plans to meet milestone dates.

OECA Response:

OECA agrees with the reports attention to strong management controls to ensure a continued quality conversion from PCS to ICIS-NPDES. OECA ha s strong management controls in place to facilitate the PCS to ICIS-NPDES conversion. There are three control structures used for this purpose including an In tegrated Project Team using established best practices for IT project management, the Environmental Information Exchange Network (EN) governance structure which controls elec tronic exchange of data, and the permanent EPA management team of the Office of Compliance. In addition there are state-specific Data Migration Workgroups which are formed in advance of each state’s conversion; to work through data migr ation issues and ensure readiness for the conversion process.

The ICIS-NPDES Batch Integrated Proj ect Team (IPT) was formed to bring together management and staff with the necessary business and technology skills from EPA (OECA, OEI, and EPA Regions) and our state partners thr ough the Environmental Council of States to implement the successful transition from PCS to ICIS-NPDES. The IPT serves as the primary management c ontrol structure for the conversion. The members oversee development processes a nd software, including finalization and implementation of the NPDES schemas for the electronic flow of data from state NPDES systems to ICIS-NPDES. This IPT was in itially convened in March of 2007 to move forward the electronic submi ssion of Discharge Monitori ng Report data from state systems to ICIS-NPDES. This was successf ully completed in May of 2008. The ICIS

NPDES Batch IPT was reconvened in October of 2009 to move forward with the electronic transmittal of the remaining NPDES data families and complete the conversion of the remaining 22 states to ICIS. There are currently 38 states and 6 regions participating on the IPT. Part of the responsib ility of the IPT is to set project milestones and regularly review them. To date, the work of the project remains on schedule. Pilot state(s) are drawn from the IPT (as voluntee rs) to test out the sc hemas and migrate to ICIS-NPDES per the ag reed upon schedule.

The governance structure of the Environmental Information Exchange Network (EN) requires that all software schema to be used for flowing data across the EN, including the products develope d under the leadership of the IPT, be reviewed and approved by the Network Technology Group (N TG) which is made up of information technology specialists that have expertise in data conversi on, exchange, and publishing. The NTG operates under the au spices of the Network Oper ations Board (NOB), a group that is responsible for the da y-to-day operations of the EN and its operational policies and procedures. Any issues that are not resolved at the NOB le vel may be elevated to the

1210-P-0230

higher Exchange Network Leadership Council. All three groups have EPA, state, and tribal partners and decisions are made collaboratively.

Within the Office of Compliance, the activities of the IPT and project milestones are reported to Office of Comp liance executives at biweekly management meetings. Both the Division Director and the Office Director and their deputies review accomplishments, address issues, and ensure that funding and other resources are available to assist states w ith conversion of their data.

Finally, an ICIS-NPDES Data Migration Wo rkgroup is initiated each time there is a group of states scheduled to migrate from legacy PCS to ICIS-N PDES. Participants include management and staff from each migr ating state, each state's respective region, and OECA. Work must be done by each migratin g state to ready their data in legacy PCS for the migration to ICIS-NPDES. The purpos e of holding the workgroup is to walk the state and regional participants through the data migration process, explain the data differences between legacy PCS and ICIS-N PDES, walk through the mapping rules for data being migrated from legacy PCS to ICIS -NPDES, review test da ta migration results from each test data migration run, and answer questions concerning the process, the data and ICIS-NPDES. Work groups typically last a year and th e status of activities is reported to the IPT and the management ch ain within the Office of Compliance.

Report Recommendation 2:

 Insert language that requires the use of the Environmental Information Exchange Network in the National Program Manager (NPM) Guidance to assist with PCS to ICIS-N PDES conversion for FY 2012

OECA Response:

OECA currently has guidance language re garding the use of the Environmental Information Exchange Network in its “FY-2011 OECA National Program Manager (NPM) Guidance” dated April 30, 2010 that reads:

“Implement the use of NetDMR or other e-DMR tools for the electronic transfer of Discharge Monitoring Reports (DMR) to ICIS-NPDES, supported by use of the National Environmental Information Exch ange Network (Exchange Network), by all of their NPDES permitted facilities. Administrator Jackson , in her July 7, 2009 memorandum, “Achieving the Promis e of the National Environmental Information Exchange Network,” reques ted cross-Agency commitment to make the Network the preferred way EPA, St ates, Tribes, and others share and exchange data while supporting an aggre ssive timetable to phase out other data submission and exchange methods. EPA Regi ons need to demonstrate leadership in implementing this strategic visi on for the Agency.” (See page 29.)

OECA commits to continuing to provide this information in future National Program Manager Guidance documents.

1310-P-0230

Report Recommendation 3:

 Gather key leading data quality practices used by the states and share these practices with all states.

OECA Response:

OECA agrees that sharing of best pr actices across the states will improve the overall quality of information available to EPA and the American public. OECA is committed to assuring data quality. It will , as permitted under the Paperwork Reduction Act, seek to identify a few states that are willing to share and pos t their quality control and quality assurance methods for achievi ng high quality enforcement and compliance information. OECA also commits to using National Meetings as forums for panel and other discussions to allow states to shar e their data quality best practices.

In addition, there are a number of existi ng examples of states sharing and using data quality practices for both ICIS and PC S. In 2009 at the ICIS National Meeting, regions and states shared their Discha rge Monitoring Report (DMR) data quality practices. States also shared their Qual ity Assurance and Quality Control (QA/QC) processes as they related to the standard reports available in ICIS. These processes involved quality control of da ta for limits, overdue sche dules, and violations. The Quarterly Non-Compliance Report is also used by states as a data quality tool. On occasion, data quality issues are addresse d on the ICIS monthly user calls.

For PCS, states use the "Dummy" audit capability to review data and reject transactions before transfe rring the data into the system . They follow up by reviewing the rejected transactions from the "Live" audit reports.

Report Recommendation 4:

 Provide reminders to EPA regional offi ces and state environmental offices to use the standard Form 2040-0004 for data entry. Where this is not feasible (e.g., for state-specific reporting requirements), EPA should require completely reporting DMR data by using standardized Form 2040-0004 or alternative entry forms. Further, EPA should remind the states of the importance of documenting changes made to data elements after entry into the source systems.

OECA Response:

OECA agrees with the report that we shoul d strive to better ensure states and regions are well informed on the proper fo rms and procedures for collecting and submitting data to EPA. OECA will take every available opportunity to remind EPA regional offices and states to use the St andard Form 2040-0004. This will include mechanisms such as the quarterly ICIS news letter, monthly user ca lls, and electronic mail

1410-P-0230

messages sent to individuals on the current ICIS list serve. Please note, however, that this form is also called Standard Form 3320-1 and both numbers appear on the physical document. Because our current user community and documentation refer to SF-3320-1 and we do not wish to confuse submitters, we will continue to refer to SF-3320-1 in our communications.

Additionally, all the Stat e and Federal electronic DMR systems are 3320-1 compliant.

Report Recommendation 5:

 Complete new rules that require st ates to report minor facility data.

OECA Response:

OECA strongly agrees with the reco mmendation to complete the rulemaking process requiring the submission of minor fac ility data. OECA is currently developing a rule which will significantly improve the NPDE S data quantity and quality nationally. A key component of this rulemaking is the requ irement for electronic reporting of certain NPDES information from the permittees, incl uding non-major facilities, to states and EPA. This information is currently required to be reported by the permittees to the states, but there is no regulatory require ment for states to provide th at specific information to EPA. This NPDES information is genera lly obtained on paper forms rather than electronically, although some states have de veloped electronic reporting tools. Therefore, this rule may likely change the mode by which the permittees report, without requiring additional information to be collected and submitted by the permittees, and reducing the burden on state programs cause d by the handling of paper forms..

Key EPA management decisions regardi ng the specifics of the rulemaking are still pending, awaiting ongoing detailed anal yses. However, this proposed requirement for electronic reporting of NPDES informati on from the permittees may likely include information such as discharge monitoring re ports (DMRs), program reports, notices of intent to discharge, etc. As a supplemen t to that information from permittees, other information will likely be required to be reported by the states to EPA regarding the states’ implementation activities (such as inspections, viola tion determinations, enforcement actions, and permit issuance) for major and non-major facilities as well as for NPDES subprograms (e.g., pretreatme nt, biosolids, CAFOs, etc.).

EPA currently projects issuance of this proposed rule in April 2011. Following a comment period, EPA expects to issue a final rule regarding this matter circa September 2012.

Report Recommendation 6:

 Place a notice on the EPA ECHO website stating that ECHO does not contain all minor facility data. The wording and placement of the notice should be

1510-P-0230

designed to help provide the public w ith transparency into the actual data elements maintained in ECHO.

OECA Response:

OECA agrees that the public needs to be aware that not all minor facility data is currently represented in ECHO. OECA does provide several notices to the public on the limitations of the data provided by ECHO. OECA has recently added several new features that allow ECHO users to find addi tional detailed informa tion about the quality and completeness of information, including an interactive map that shows the data entry percentage for minor discharge monitoring data in each state. However, this information can be difficult to find, and OECA agrees this information can be better organized. For example we currently use grey text to indicate that there are data caveats. We will enhance this grey text so it is more rec ognizable to ECHO users. OECA will evaluate each area where the existing notice is displayed and determine if it needs to be enhanced. OECA will also evaluate its ECHO site as a whole and determine if additional notices need to be added. OECA regularly enhan ces the ECHO website, and will continue its practice of providing the public wi th caveats concerning the data.

Report Recommendation 7:

 Develop and implement procedures to test ICIS-NPDES programming code before it is placed into production.

OECA Response:

OECA has extensive, documented pro cedures in place to test ICIS-NPDES programming code before it is placed into pr oduction. It follows best practices used by the information technology industry as we ll as EPA system life cycle policies and procedures. The prime contractor for the ICIS system is certified by the Software Engineering Institute (SEI) to be Level 3 on the Capability Maturity Model Integration (CMMI) scale. This is the same high level that is required for Department of Defense contractors. This certificati on requires the contractor to follow repeatable, standard processes for verification of all products. In addition, OECA completes and documents all items that are required by the EPA National Computing Center’s Automated Deployment Checklist prior to each software release.

As part of the standard life cycle pro cess, procedures for thoroughly testing the software for ICIS-NPDES were implemente d during the 2004/2005 timeframe. This was the period during which the software for ICIS -NPDES was being initi ally developed and tested prior to its release into production. These proce dures have been refined and improved over the years, and, because of this , new and revised software is typically implemented into production with few, if any, defects. The software goes through Unit Testing, Functional Testing, and End-to-End or System Integration Testing by the ICIS contractor to ensure it both meets stated re quirements and executes correctly before it is turned over to EPA and the user community for User Acceptance Testing. For new

1610-P-0230

software relating to major new capability, the software also goe s though Beta Testing with experts on the ICIS Project Team prior to User Acceptance Testing. This ensures that as many defects as possible are found and fixed prior to user co mmunity testing. All testing is conducted according to Test Plan s which include documenting test cases and, upon execution, the results of ex ecuting the test cases. Pr oblems identified during each test phase are corrected and retested as necessa ry prior to the software being released into Production. Testing results are reviewed during the Bi-weekly Project Management meetings held between EPA and the ICIS contractor.

1710-P-0230

Appendix B



## Distribution

Office of the Administrator Assistant Administrator for Enforcement and Compliance Assurance Assistant Administrator for Environmental Information and Chief Information Officer Regional Administrator, Region 3 Regional Administrator, Region 4 Regional Administrator, Region 5 Director, Office of Compliance, Office of Enforcement and Compliance Assurance Acting Director, Office of Information Collect ion, Office of Environmental Information Agency Follow-up Official (the CFO) Agency Follow-up Coordinator General Counsel Associate Administrator for Congressi onal and Intergovernmental Relations Associate Administrator for External Affairs and Environmental Education Agency Follow-up Coordinator, Office of Enforcement and Compliance Assurance Inspector General

18