U.S. ENVIRONMENTAL PROTECTION AGENCY 

OFFICE OF INSPECTOR GENERAL 

Catalyst for Improving the Environment 

Audit Report 

ECHO Data Quality Audit -

Phase 2 Results: EPA Could 

Achieve Data Quality Rate With 

Additional Improvements 

 Project No. 10-P-0230 

 September 22, 2010 
 

 

 

 

 

 

 

 

Abbreviations 

DMR Discharge Monitoring Report 

ECHO Enforcement Compliance and History Online 

EPA U.S. Environmental Protection Agency 

GPRA Government Performance and Results Act of 1996 

ICIS-NPDES Integrated Compliance Information System – National Pollutant Discharge 

 Elimination System 

IDEF Interim Data Exchange Format 

OECA Office of Enforcement and Compliance Assurance 

OIG Office of Inspector General 

OMB Office of Management and Budget 

PCS Permit Compliance System 
 

 

U.S. Environmental Protection Agency 10-P-0230 

September 22, 2010 Office of Inspector General 

At a Glance 

Catalyst for Improving the Environment 

Why We Did This Review 

We sought to assess the 

quality of key data elements 

reported through the 

Enforcement Compliance and 

History Online (ECHO) 

Website. KPMG, LLP, 

performed the review. 

Background 

ECHO provides a single 

source of detailed compliance 

history of U.S. Environmental 

Protection Agency (EPA)-

regulated facilities. EPA 

developed ECHO to provide 

the public compliance and 

inspection data under its 

environmental programs, as 

well as demographic data of 

the surrounding areas. This 

report focuses on the quality 

of data elements entered into 

ECHO source systems. 

For further information, 

contact our Office of 

Congressional, Public Affairs 

and Management at 

(202) 566-2391. 

To view the full report, 

click on the following link: 

www.epa.gov/oig/reports/2010/ 

20100922-10-P-0230.pdf 

ECHO Data Quality Audit - Phase 2 Results: 

EPA Could Achieve Data Quality Rate With 

Additional Improvements 

What KPMG Found 

EPA mandates that data elements reported to the public through the ECHO 

Website have a 95 percent accuracy rate. KPMG found a 91.5 percent data 

accuracy rate for key data elements entered into two primary ECHO source 

systems: the legacy Permit Compliance System (PCS) and the newer Integrated 

Compliance Information System – National Pollutant Discharge Elimination 

System (ICIS-NPDES). Although the 91.5 percent data quality rate is close to 

EPA’s goal, EPA and the State environmental offices could take additional steps 

to increase the quality of data reported through the ECHO Website. 

 What KPMG Recommends 

KPMG made several recommendations to the Assistant Administrator for 

Enforcement and Compliance Assurance. These included: 

	 Establishing an internal control structure to help manage the conversion of 

PCS to ICIS-NPDES. 

	 Including language in the National Program Manager Guidance requiring 

the use of the Environmental Information Exchange Network for reporting 

data to EPA. 

	 Developing a plan to share data quality best practices implemented at 

State environmental offices with all States. 

	 Completing new rules requiring reporting ECHO data for minor facilities 

and notifying ECHO Website users that the site does not contain data on 

minor facilities. 

	 Reviewing procedures used to test ICIS-NPDES programming code 

before it is placed into production. 

The Agency generally agreed with the report findings. EPA felt it has extensive, 

documented procedures in place to test ICIS-NPDES programming code before it 

is placed into production. Tests disclosed that programming errors directly 

resulted in incomplete data on the ECHO Website and, as such, management 

should review these processes to prevent future occurrences. The Agency’s 

response is included in Appendix A. 
 

 

UNITED STATES ENVIRONMENTAL PROTECTION AGENCY 

WASHINGTON, D.C. 20460 

THE INSPECTOR GENERAL 

September 22, 2010 

MEMORANDUM 

SUBJECT:	 ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve 

Data Quality Rate With Additional Improvements 

Report No. 10-P-0230 

FROM: Arthur A. Elkins, Jr. 

 Inspector General 

TO:	 Cynthia Giles 

Assistant Administrator for Enforcement and Compliance Assurance 

This is the report on the subject audit conducted by KPMG, LLP, on behalf of the Office of 

Inspector General (OIG) of the U.S. Environmental Protection Agency (EPA). This report 

contains findings KPMG LLP identified and corrective actions recommended. This report 

represents the conclusions of KPMG and does not necessarily represent the final position of the 

OIG on the subjects reported. Final determinations on matters in this report will be made by 

EPA management in accordance with established audit resolution procedures. 

The estimated cost of this audit, which includes contract costs and OIG contract management 

oversight, is $331,361. 

Action Required 

In accordance with EPA Manual 2750, you are required to provide a written response to this 

report within 90 calendar days. You should include a corrective actions plan for agreed-upon 

actions, including milestone dates. Your response will be posted on the OIG’s public Website, 

along with our comments on your response. Your response should be provided in an Adobe 

PDF file that complies with the accessibility requirements of section 508 of the Rehabilitation 

Act of 1973, as amended. If your response contains data that you do not want to be released to 

the public, you should identify the data for redaction. We have no objections to the further 

release of this report to the public. This report will be available at http://www.epa.gov/oig. 

If you or your staff have any questions regarding this report, please contact 

Patricia H. Hill at (202) 566-0894 or hill.patricia@epa.gov, or Rudolph M. Brevard at 

(202) 566-0893 or brevard.rudy@epa.gov. 
 

 

 

September 22, 2010 

SUBJECT: Audit Report: 

ECHO Data Quality Audit - Phase 2 Results: EPA Could Achieve 

Data Quality Rate With Additional Improvements 

THRU:	 Arthur A. Elkins, Jr. 

 Inspector General 

TO:	 Cynthia Giles 

 Assistant Administrator 

Office of Enforcement and Compliance Assurance 

This memorandum is to inform the U.S. Environmental Protection (EPA) of critical findings by 

KPMG, LLP, that require management action regarding the improvement of the data quality for 

the Enforcement Compliance History Online Website. 

If you or your staff have any questions regarding this report, please contact the 

U.S. Environmental Protection Agency, Office of Inspector General, point of contact 

identified on the report transmittal memorandum. 
ECHO Data Quality Audit - Phase 2 Results: 10-P-0230 

EPA Could Achieve Data Quality Rate With 

Additional Improvements 

Table of Contents 

Purpose........................................................................................................................ 1 

Background ................................................................................................................. 1 

Scope and Methodology............................................................................................. 2 

Findings ....................................................................................................................... 4 

Identified Data Quality Exceptions ............................................................................ 4 

Factors That Contributed to Data Quality Exceptions............................................. 5 

No Automated Data Quality Processes................................................................... 5 

No Consistent State Quality Review Processes ..................................................... 6 

No Standardized DMR Form................................................................................... 6 

No Documentation of Data Entry ............................................................................ 7 

No Requirements to Report Minor Facility Data...................................................... 7 

Michigan Data Processing ......................................................................................... 7 

Data Quality Exceptions Affect EPA’s Representation of Data .............................. 8 

Recommendations...................................................................................................... 8 

Agency Comments and OIG Evaluation ................................................................... 9
 

Status of Recommendations and Potential Monetary Benefits.............................. 10 

Appendices 

A Agency Response .............................................................................................. 11 

B Distribution ......................................................................................................... 18 
10-P-0230 

Purpose 

The U.S. Environmental Protection Agency (EPA) Office of Inspector General (OIG) 

initiated this review to evaluate the quality of data reported through the EPA 

Enforcement Compliance and History Online (ECHO) Website. KPMG LLP (KPMG) 

completed the review in the following two phases: 

	 Phase 1: Evaluate the automated process EPA uses to display ECHO data 

transmitted from select source systems. We reviewed the automated processes for 

the legacy Permit Compliance System (PCS) and the newer Integrated 

Compliance Information System – National Pollutant Discharge Elimination 

System (ICIS-NPDES). We provided the Phase 1 results in OIG report, ECHO 

Data Quality Audit - Phase I Results: The Integrated Compliance Information 

System Needs Security Controls to Protect Significant Non-Compliance Data, 

Report No. 09-P-0226, August 31, 2009. 

	 Phase 2: Test the quality of select PCS and ICIS-NPDES data elements against 

Discharge Monitoring Reports (DMRs), which are the primary source documents 

for PCS and ICIS-NPDES data. 

Background 

ECHO provides a single source of detailed and up-to-date environmental compliance 

history of EPA-regulated facilities. EPA developed ECHO in partnership with State 

environmental offices to provide compliance and inspection data under air, water, and 

hazardous waste programs, as well as demographic data of the surrounding areas. Since 

ECHO’s November 2002 launch, hundreds of thousands of citizens, government 

officials, investors, and staff at regulated facilities and companies have asked over 

3,000,000 questions about the environmental records of the more than 800,000 facilities. 

EPA and the state environmental offices need to maintain a high level of ECHO data 

quality to: 1) provide accurate and complete information to the public, States, facilities, 

and other stakeholders, and 2) maintain their reputations as effective public stewards and 

information providers. 

The Government Performance and Results Act of 1996 (GPRA) requires federal agencies 

to provide managers with accurate information about program results and service quality. 

EPA’s 2009 Performance and Accountability Report (PAR), developed in accordance 

with GPRA, specifically identifies data quality as a key element of EPA’s efforts to 

improve management and results. 

The Office of Management and Budget (OMB) Circular A-130, Management of Federal 

Information Resources, authorized in part by GPRA and the Paperwork Reduction Act, 

guides that federal agencies should ensure that systems maximize the usefulness of 

information, minimize the burden on the public, and preserve the appropriate integrity, 

1
 
10-P-0230 

usability, availability, and confidentiality of information throughout the life cycle of the 

information. 

EPA’s Permit Compliance System (PCS) Quality Assurance Guidance Manual states: 

	 It is very important that PCS and NPDES data be complete, accurate, and up-todate, and that all users be consistent in the way they define and use various data 

elements because the systems play such a central role in program management. 

	 Quality is achieved through effective management and commitment. Responsive 

management should provide the support and training that is necessary to achieve 

good data quality; and recognize, reward, and encourage quality service and 

performance. 

	 Quality must be tracked and performance of the program evaluated at regular 

intervals. Data quality must be measurable so that the causes of poor data quality 

can be identified and corrected. 

	 In order for the quality of PCS data to be “good,” the data must meet quality 

targets for timeliness, accuracy, completeness, and consistency. For example, a 

quality target for accuracy is that 95 percent of the data elements in the system of 

record be identical to the source document. The failure to meet these targets is 

especially serious when the data elements affect public access. 

	 Ninety-five percent of the data elements entered into PCS should be identical with 

those reported on the DMR permit or other input document. 

EPA’s PCS Policy Statement states that to achieve national consistency and uniformity in 

the NPDES program, the required data in PCS must be complete and accurate. 

The Office of Compliance is responsible for identifying, preventing, and reducing 

noncompliance and environmental risks by providing leadership in program planning, 

priority setting, and expanding the use of compliance data. In this regard, the office’s 

Enforcement Targeting and Data Division (ETDD) supports EPA’s national enforcement 

and compliance information and reporting needs and manages the ECHO Internet site. 

Scope and Methodology 

We coordinated with the OIG to select major high-risk ecosystems and corresponding 

discharge facilities as the population for our testing. We and the OIG selected the major 

ecosystems and facilities, in part, based on the EPA’s 2006-2011 Strategic Plan, which 

identifies high-risk ecosystems in the United States. For the selected ecosystems and 

facilities, we reviewed 2008 calendar year DMR data extracted from PCS and ICIS– 

NPDES. 

2
 
 

10-P-0230 

We performed this audit in accordance with Generally Accepted Government Auditing 

Standards issued by the Comptroller General of the United States. These standards 

require that we plan and perform the audit to obtain sufficient and appropriate evidence to 

provide a reasonable basis for their findings and conclusions based on the audit 

objectives. We believe the evidence obtained provides a reasonable basis for our findings 

and conclusions. 

To assist in selecting the ecosystems, we used the EPA Website, which provided a map 

of watersheds and Hydrologic Unit Codes, as well as the ECHO map feature. We then 

selected a judgmental sample of facilities supporting each ecosystem. For these 

facilities, we compared data elements reported in ECHO to related DMR records. We 

included the following types of data elements in our review: 

 Facility name 

 Permit number and approval 

 Pollutant levels 

 Facility status (e.g., major or minor) 

 Specific DMR data (e.g., types of pollutants) 

We focused our review on the more critical data elements, such as pollutant levels and 

facility status, rather than less critical elements, such as facility addresses and phone 

numbers. Our testing was based on a judgmental sample, and the results are not 

statistically valid and should not be projected to other States or to all facilities in the 

States we visited (for the State of Florida, we did test the entire population of 41 nonFederal managed facilities). 

We conducted the following site visits: 

Table 1: State Environmental Offices Visited During the Review 

State 

Visited 

ECHO Source 

System Used Dates of Visit Ecosystem 

No. of Facilities 

Reviewed 

Maryland ICIS-NPDES November 2 – 10, 2009 Chesapeake Bay 50 (sample) 

Indiana ICIS-NPDES November 16 – 20, 2009 Great Lakes 50 (sample) 

Michigan PCS December 7 – 11, 2009 Great Lakes 50 (sample) 

Florida PCS January 11 – 15, 2010 South Florida 42 (entire population) 

Source: KPMG-compiled data 

3
 
10-P-0230 

Findings 

EPA mandates that data elements reported to the public through the ECHO Website have 

a 95 percent accuracy rate. During this review, we found a 91.5 percent data accuracy 

rate for key data elements entered into two primary ECHO source systems - the legacy 

PCS and newer ICIS-NPDES. Although the 91.5 percent data quality rate is close to 

EPA’s goal, EPA and the State Environmental Offices can make additional 

improvements to further increase data quality. Specifically, we identified several types of 

data quality exceptions and a processing issue in the State of Michigan that affected the 

quality of data ultimately presented through ECHO. We cite them on the following 

pages. 

Identified Data Quality Exceptions 

During each of our four site visits to State environmental offices, we found four types of 

data element exceptions, as follows: 

	 Data elements were reported in ECHO but not in DMRs. In these cases, we noted 

that data elements were reported through ECHO but the supporting source DMRs 

did not reflect the same data element information. 

	 Data elements were reported in DMRs but not in ECHO. In these cases we noted 

that data elements were reported in the source DMRs (e.g., pollutant values), but 

the same data element information was not reported through ECHO. 

	 Data elements were inaccurately input into ECHO source systems (PCS or ICISNPDES) from DMRs. For example, at one facility, the pollutant phosphorous had 

a value of 1.7 in ECHO but a value of 0.7 on the supporting DMR. At another 

facility, the pollutant nitrogen had a value of 2.64 in ECHO but a value of 2.22 on 

the DMR. 

	 Data elements had missing DMRs. In these cases, we were not able to review the 

applicable source DMRs because they were not available at the state 

environmental offices. 

4
 
 

 

10-P-0230 

The following table summarizes the data quality exceptions by state. 

Table 2: Data Quality Exceptions 

State 

No. of 

Data 

Elements 

Tested 

Data 

Elements 

in ECHO 

Not 

Supported 

by DMRs 

Data 

Elements 

on DMRs 

Not 

Entered 

Into ECHO 

Data 

Elements 

Inaccurately 

Entered Into 

Source 

Systems 

From DMRs 

Data 

Elements 

With 

Missing 

DMRs 

Total Data 

Elements 

With 

Exceptions 

Exception 

% 

 (By State) 

Maryland 6,199 94 386 174 133 787 12.7% 

Indiana 8,850 297 16 16 8 337 3.8% 

Florida 12,590 62 501 85 565 1213 9.6% 

Totals 27,639 453 903 275 706 2,337 8.5% 

Source: KPMG-compiled results from documentation review 

Michigan had data quality exceptions in the same four categories. However, Michigan’s 

results are not included in Table 2 because we were not able to fully quantify the 

exceptions. Michigan had a data processing issue that resulted in many data elements not 

being displayed in ECHO. See the Michigan Data Processing section later in this report 

for details. 

We provided detailed data quality exceptions to the visited State environmental offices, 

in which we identified the facility names and types of exceptions identified at each 

facility. 

Factors That Contributed to Data Quality Exceptions 

No Automated Data Quality Processes 

The DMR data entry process is a manual-oriented process for many State environmental 

offices, which significantly elevates the risk of data exceptions. The EPA 2009 PAR 

addresses this issue in part by stating, “data quality has been improving and will continue 

to improve as existing data entry requirements and procedures are being reevaluated and 

communicated with data entry practitioners.” The PAR further states that in FY 1999 

EPA acknowledged PCS as a weakness and recognized that the Agency needed to 

revitalize or replace PCS to ensure complete and accurate water data. EPA has been 

continually enhancing DMR data entry and quality review processes, including efforts to 

(1) convert from PCS to ICIS-NPDES, (2) increase use of the Interim Data Exchange 

Format (IDEF) quality review process, and (3) implement eDMR and NetDMR 

automation capabilities. According to EPA officials, competing resources have limited 

full implementation of these enhancements. 

5
 
 

 

 

10-P-0230 

Regarding the conversion from PCS to ICIS-NPDES, the Office of Enforcement and 

Compliance Assurance (OECA) Office of Compliance, with support from the Office of 

Water and the Office of Environmental Information, has been working with States and 

regions to develop data requirements, design the new system, test the developed software, 

and prepare PCS data for migration. Now 28 States, 2 tribes, and 10 territories are using 

ICIS-NPDES instead of PCS. The full migration from PCS to ICIS-NPDES is scheduled 

for completion in FY 2013. These and other automation efforts are critical to improve 

data quality. The Environmental Information Exchange Network estimates that 

“electronic DMR reporting has the highest documented return on investment of any other 

EPA data automation project - implementation by all or most NPDES facilities could 

save industry, states, and EPA around $100 million per year.”1 

Using eDMR and NetDMR capabilities will assist with Full Batch State conversion. 

OECA can accelerate the Full Batch State conversion by using the Environmental 

Information Exchange Network as a preferred method of transmitting data to the Agency. 

OECA’s FY 2010 & 2011 National Program Manager Guidance does not contain any 

language or plans for using the Environmental Information Exchange Network. In this 

regard, more liaison with respective EPA offices would help ensure Agency grants given 

to States would more effectively align with EPA data. 

No Consistent State Quality Review Processes 

Closely linked to the above factor, ECHO data quality efforts are reliant upon resource 

capabilities in EPA and the State environmental offices. During our site visits, we noted 

that the States’ data quality resources vary. For example, the State of Maryland has one 

dedicated person focused on the ECHO data quality program, while the State of Indiana 

had four data entry personnel and three data quality reviewers. As noted earlier in Table 

2, Indiana had the smallest number of exceptions. 

EPA’s IDEF function provides the State environmental offices with the capability to 

audit their submission of ECHO data for completeness and accuracy, but the States are 

not consistently reviewing IDEF. In fact, State environmental office officials confirmed 

that some of the data quality findings identified during this review would not have been 

identified through existing EPA data quality processes, such as IDEF. 

No Standardized DMR Form 

State environmental offices use various types of DMR forms for data entry. Some of 

these offices use their own State-specific reporting forms, and forms can vary within the 

same State. 

EPA’s Permit Compliance System (PCS) Quality Assurance Guidance Manual states: 

1 http://www.exchangenetwork.net/benefits/NetDMR_SuccessStory.pdf 

6
 
10-P-0230 

All DMRs submitted to EPA Regional Offices (including DMRs 

submitted by NPDES States for EPA entry into PCS) must be 

preprinted using the Office of Management and Budget (OMB) 

approved DMR form. NPDES States directly using PCS are not 

required to use the OMB-approved form; however, its use is strongly 

encouraged. 

Although some similarity exists between the various input forms, the lack of full 

standardization increases the risk of data quality exceptions. Enhanced automation with 

ICIS-NPDES, eDMR, and NetDMR could improve this issue. There is a standard form 

for entering NPDES data (OMB Form Number 2040-0004, National Pollutant Discharge 

Elimination System (NPDES) Discharge Monitoring Report), but we found that not all 

State environmental offices were consistently using this form. 

No Documentation of Data Entry 

State environmental office officials informed us their data quality personnel sometimes 

will input data into PCS or ICIS-NPDES but not record the information in supporting 

documentation (e.g., DMRs, permits). The PCS Quality Assurance Manual states that the 

regions should work closely with their NPDES States using PCS to address similar data 

entry problems with State-issued NPDES permits. The primary reason for not updating 

the source documentation was that the State officials believed the changes were so minor 

they would not affect overall data quality. 

No Requirements to Report Minor Facility Data 

Existing legislation does not require regulated facilities to send EPA all minor facility 

data elements displayed in ECHO. Only submission of major facility data elements is 

required. 

EPA’s Permit Compliance System (PCS) Quality Assurance Guidance Manual states: 

If States and Regions wish to enter NPDES data beyond what has been required, 

they may do so. For example, if States want to enter DMR data for minor 

facilities, the options is available in PCS and the States may use it as their 

resources allow. 

According to EPA officials, a proposed rule is being developed to require States and 

regulated facilities to provide additional minor facility data. EPA expects to have the rule 

finalized during FY 2012. 

Michigan Data Processing 

During our site visit to Michigan, we identified a data processing error where DMR data 

for 18 facilities were transmitted to PCS but the data was not completely populated into 

ECHO. Specifically, ECHO did not completely receive PCS data for facilities with 

7
 
10-P-0230 

multiple discharge points. EPA had previously identified this issue, as the ECHO Data 

Alerts Website (http://www.epa-echo.gov/echo/data_alerts.html), last updated on July 6, 

2009, identified the issue and stated that EPA is working with the State of Michigan to 

resolve the issue. ETDD determined that software code used to pull data elements from 

PCS into ECHO was not pulling all necessary data. ETDD developed, tested, and 

implemented new software code to correct the issue. The processing error contributed to 

a significant number of 2008 data quality exceptions for Michigan. As noted in the 

Permit Compliance System (PCS) Quality Assurance Guidance Manual, PCS data must 

be complete and accurate to assist EPA with program management activities. 

While these efforts correct the identified issues with the Michigan data, ultimately noncompliance with prescribed EPA System Management Life Cycle (SMLC) Guide 

contributed to the code being placed into production without testing. EPA SMLC 

guidance states that the Test Subphase results in proof that the system satisfies the 

requirements. As such, internal controls over system testing would have identified the 

error in the Michigan programming logic before it was placed into production. 

Data Quality Exceptions Affect EPA’s Representation of Data 

The exception rates increase the risk that ECHO data presented to internal EPA 

stakeholders and the public is not accurate. EPA’s Permit Compliance System (PCS) 

Quality Assurance Guidance Manual notes that PCS and ICIS-NPDES data must be 

complete and accurate to assist EPA with program management activities. Further, 

inaccurate ECHO data could misrepresent facility information, so data accuracy is 

significant for EPA and State environmental offices in their roles as good public 

stewards. 

EPA officials informed us that the Agency partially mitigates the risk of data exceptions 

by relying on the public and facilities to notify the Agency of inaccurate data. The EPA 

2009 PAR emphasizes this point by stating, “an important means for catching potential 

data errors is to obtain stakeholder feedback. EPA has a formal error correction process 

in place to investigate and react to public notification of errors in its publications.” While 

relying on the public serves as a detective control, it is still incumbent upon EPA and the 

State environmental offices, as good public stewards, to have more effective preventive 

controls in place to better ensure data quality. 

Recommendations 

We recommend that the Director, Office of Compliance, within the Office of 

Enforcement and Compliance Assurance: 

1.	 Establish a management control structure to facilitate PCS to ICIS-NPDES 

conversion. The management control structure should support plans to meet 

milestone dates. 

8
 
10-P-0230 

2.	 Insert language that requires the use of the Environmental Information Exchange 

Network in the National Program Manager (NPM) Guidance to assist with PCS to 

ICIS-NPDES conversion for FY 2012. 

3.	 Gather key leading data quality practices used by the States and share these 

practices with all States. 

4.	 Provide reminders to EPA regional offices and State environmental offices to use 

the standard Form 2040-0004 for data entry. Where this is not feasible (e.g., for 

State-specific reporting requirements), EPA should require completely reporting 

DMR data by using standardized Form 2040-0004 or alternative entry forms. 

Further, EPA should remind the States of the importance of documenting changes 

made to data elements after entry into the source systems. 

5.	 Complete new rules that require States to report minor facility data. 

6.	 Place a notice on the EPA ECHO Website stating that ECHO does not contain all 

minor facility data. The wording and placement of the notice should be designed 

to help provide the public with transparency into the actual data elements 

maintained in ECHO. 

7.	 Conduct a review of the procedures used to test ICIS-NPDES programming code 

before it is placed into production. 

Agency Comments and OIG Evaluation 

EPA agreed with the report findings. EPA felt that it has extensive, documented 

procedures in place to test ICIS-NPDES programming code before it is placed into 

production. Management cited that they follow best practices used by the information 

technology industry as well as EPA system life cycle policies and procedures and that 

their primary contractor is certified by the Software Engineering Institute to be Level 3 

on the Capability Maturity Model Integration scale. Management also stated their 

procedures have been refined and improved over the years, and, because of this, new and 

revised software is typically implemented into production with few, if any, defects. 

We agree that management should use industry best practices whenever feasible and 

adopt an industry recognized method for controlling system development activities. 

However, we believe as part of an effective internal control structure management must 

continually review these practices to ensure they operate as intended. For the State of 

Michigan, EPA recognized the error in the State’s ECHO data and it took the Agency 

close to one year to make changes to the programming code to correct the problem. As 

such, we believe it is incumbent upon management to conduct a root cause analysis to 

identify the causes for the programming code error and update the testing procedures to 

prevent similar errors from happening in the future. 

9
 
 

 

 

 

 

 

 

 

 

 

10-P-0230 

Status of Recommendations and 

Potential Monetary Benefits 

POTENTIAL MONETARY 

RECOMMENDATIONS BENEFITS (in $000s) 

Rec. 

No. 

1 

2 

3 

4 

5 

6 

7 

Page 

No. 

8 

9 

9 

9 

9 

9 

9 

Subject 

Establish a management control structure to 

facilitate PCS to ICIS-NPDES conversion. 

The management control structure should 

support plans to meet milestone dates. 

Insert language that requires the use of the 

Environmental Information Exchange 

Network in the National Program Manager 

(NPM) Guidance to assist with PCS to ICISNPDES conversion for FY 2012. 

Gather key leading data quality practices 

used by the States and share these practices 

with all States. 

Provide reminders to EPA regional offices 

and State environmental offices to use the 

standard Form 2040-0004 for data entry. 

Where this is not feasible (e.g., for Statespecific reporting requirements), EPA should 

require completely reporting DMR data by 

using standardized Form 2040-0004 or 

alternative entry forms. Further, EPA should 

remind the States of the importance of 

documenting changes made to data elements 

after entry into the source systems. 

Complete new rules that require States to 

report minor facility data. 

Place a notice on the EPA ECHO Website 

stating that ECHO does not contain all minor 

facility data. The wording and placement of 

the notice should be designed to help provide 

the public with transparency into the actual 

data elements maintained in ECHO. 

Conduct a review of the procedures used to test 

ICIS-NPDES programming code before it is placed 

into production. 

Status1 

O 

O 

O 

O 

O 

O 

O 

Action Official 

Director, Office of 

Compliance, Office of 

Enforcement and 

Compliance Assurance 

Director, Office of 

Compliance, Office of 

Enforcement and 

Compliance Assurance 

Director, Office of 

Compliance, Office of 

Enforcement and 

Compliance Assurance 

Director, Office of 

Compliance, Office of 

Enforcement and 

Compliance Assurance 

Director, Office of 

Compliance, Office of 

Enforcement and 

Compliance Assurance 

Director, Office of 

Compliance, Office of 

Enforcement and 

Compliance Assurance 

Director, Office of 

Compliance, Office of 

Enforcement and 

Compliance Assurance 

Planned 

Completion 

Date 

Claimed 

Amount 

Agreed To 

Amount 

O = recommendation is open with agreed-to corrective actions pending 

C = recommendation is closed with all agreed-to actions completed 

U = recommendation is undecided with resolution efforts in progress 

10
 
 

 

 

 

 

10-P-0230 

Appendix A 

Agency Response 

UNITED STATES ENVIRONMENTAL PROTECTION AGENCY 

WASHINGTON, D.C. 20460 

August 20, 2010 OFFICE OF 

ENFORCEMENT AND 

COMPLIANCE ASSURANCE 

MEMORANDUM 

SUBJECT:	 Response to the Office of the Inspector General Draft Evaluation Report, “ECHO 

Data Quality Audit – Phase 2 Results: EPA Could Achieve Data Quality Rate 

with Additional Improvements” Project Number OMS-FY09-00011, dated 

July 28, 2010 

FROM:	 Cynthia Giles /s/ 

 Assistant Administrator 

TO: Rudolph M. Brevard 

Director, Information Resources Management Assessments 

Office of Mission Systems 

Office of Inspector General 

Thank you for the opportunity to review and comment on the draft evaluation report 

entitled, “ECHO Data Quality Audit – Phase 2 Results: EPA Could Achieve Data Quality Rate 

with Additional Improvements,” (Report) Project Number OMS-FY09-00011. This Report 

focuses on improving the quality and transparency of the enforcement and compliance 

information EPA provides to the public through our Enforcement and Compliance History Online 

(ECHO) website. 

OECA agrees that providing high quality transparent data to the American public is a 

fundamental responsibility of the government. Recently, OECA has taken several steps to 

expand transparency of enforcement and compliance information through ECHO with the release 

of the Clean Water Act Annual Noncompliance 2008 Report and the enhanced Air, Water and 

Waste Resource pages. 

Additionally, OECA is expanding our efforts to ensure the accuracy of the information in 

ECHO. A key new resource we are providing the public is consolidated information by program 

area (Clean Water, Clean Air and Hazardous Waste). Included on these pages are reports such as 

the “2009 Annual CAA Majors Report” and the “2009 State Review Framework Frozen National 

Report”. Prior to updating these annual reports, we provide the States an opportunity to verify the 

data and provide additional information. OECA is working with our State partners to ensure that 

the public receives the best information available. 

11
 
 

10-P-0230 

Report Recommendation 1: 

	 Establish a management control structure to facilitate PCS to ICIS-NPDES 

conversion. The management control structure should support plans to meet 

milestone dates. 

OECA Response: 

OECA agrees with the reports attention to strong management controls to ensure a 

continued quality conversion from PCS to ICIS-NPDES. OECA has strong management 

controls in place to facilitate the PCS to ICIS-NPDES conversion. There are three 

control structures used for this purpose including an Integrated Project Team using 

established best practices for IT project management, the Environmental Information 

Exchange Network (EN) governance structure which controls electronic exchange of 

data, and the permanent EPA management team of the Office of Compliance. In addition 

there are state-specific Data Migration Workgroups which are formed in advance of each 

state’s conversion; to work through data migration issues and ensure readiness for the 

conversion process. 

The ICIS-NPDES Batch Integrated Project Team (IPT) was formed to bring 

together management and staff with the necessary business and technology skills from 

EPA (OECA, OEI, and EPA Regions) and our state partners through the Environmental 

Council of States to implement the successful transition from PCS to ICIS-NPDES. The 

IPT serves as the primary management control structure for the conversion. The 

members oversee development processes and software, including finalization and 

implementation of the NPDES schemas for the electronic flow of data from state NPDES 

systems to ICIS-NPDES. This IPT was initially convened in March of 2007 to move 

forward the electronic submission of Discharge Monitoring Report data from state 

systems to ICIS-NPDES. This was successfully completed in May of 2008. The ICISNPDES Batch IPT was reconvened in October of 2009 to move forward with the 

electronic transmittal of the remaining NPDES data families and complete the conversion 

of the remaining 22 states to ICIS. There are currently 38 states and 6 regions 

participating on the IPT. Part of the responsibility of the IPT is to set project milestones 

and regularly review them. To date, the work of the project remains on schedule. Pilot 

state(s) are drawn from the IPT (as volunteers) to test out the schemas and migrate to 

ICIS-NPDES per the agreed upon schedule. 

The governance structure of the Environmental Information Exchange Network 

(EN) requires that all software schema to be used for flowing data across the EN, 

including the products developed under the leadership of the IPT, be reviewed and 

approved by the Network Technology Group (NTG) which is made up of information 

technology specialists that have expertise in data conversion, exchange, and publishing. 

The NTG operates under the auspices of the Network Operations Board (NOB), a group 

that is responsible for the day-to-day operations of the EN and its operational policies and 

procedures. Any issues that are not resolved at the NOB level may be elevated to the 

12
 
10-P-0230 

higher Exchange Network Leadership Council. All three groups have EPA, state, and 

tribal partners and decisions are made collaboratively. 

Within the Office of Compliance, the activities of the IPT and project milestones 

are reported to Office of Compliance executives at biweekly management meetings. 

Both the Division Director and the Office Director and their deputies review 

accomplishments, address issues, and ensure that funding and other resources are 

available to assist states with conversion of their data. 

Finally, an ICIS-NPDES Data Migration Workgroup is initiated each time there is 

a group of states scheduled to migrate from legacy PCS to ICIS-NPDES. Participants 

include management and staff from each migrating state, each state's respective region, 

and OECA. Work must be done by each migrating state to ready their data in legacy PCS 

for the migration to ICIS-NPDES. The purpose of holding the workgroup is to walk the 

state and regional participants through the data migration process, explain the data 

differences between legacy PCS and ICIS-NPDES, walk through the mapping rules for 

data being migrated from legacy PCS to ICIS-NPDES, review test data migration results 

from each test data migration run, and answer questions concerning the process, the data 

and ICIS-NPDES. Work groups typically last a year and the status of activities is 

reported to the IPT and the management chain within the Office of Compliance. 

Report Recommendation 2: 

	 Insert language that requires the use of the Environmental Information 

Exchange Network in the National Program Manager (NPM) Guidance to 

assist with PCS to ICIS-NPDES conversion for FY 2012 

OECA Response: 

OECA currently has guidance language regarding the use of the Environmental 

Information Exchange Network in its “FY-2011 OECA National Program Manager 

(NPM) Guidance” dated April 30, 2010 that reads: 

“Implement the use of NetDMR or other e-DMR tools for the electronic transfer 

of Discharge Monitoring Reports (DMR) to ICIS-NPDES, supported by use of the 

National Environmental Information Exchange Network (Exchange Network), by 

all of their NPDES permitted facilities. Administrator Jackson , in her July 7, 

2009 memorandum, “Achieving the Promise of the National Environmental 

Information Exchange Network,” requested cross-Agency commitment to make 

the Network the preferred way EPA, States, Tribes, and others share and 

exchange data while supporting an aggressive timetable to phase out other data 

submission and exchange methods. EPA Regions need to demonstrate leadership 

in implementing this strategic vision for the Agency.” (See page 29.) 

OECA commits to continuing to provide this information in future National 

Program Manager Guidance documents. 

13
 
 

 

 

10-P-0230 

Report Recommendation 3: 

	 Gather key leading data quality practices used by the states and share these 

practices with all states. 

OECA Response: 

OECA agrees that sharing of best practices across the states will improve the 

overall quality of information available to EPA and the American public. OECA is 

committed to assuring data quality. It will, as permitted under the Paperwork Reduction 

Act, seek to identify a few states that are willing to share and post their quality control 

and quality assurance methods for achieving high quality enforcement and compliance 

information. OECA also commits to using National Meetings as forums for panel and 

other discussions to allow states to share their data quality best practices. 

In addition, there are a number of existing examples of states sharing and using 

data quality practices for both ICIS and PCS. In 2009 at the ICIS National Meeting, 

regions and states shared their Discharge Monitoring Report (DMR) data quality 

practices. States also shared their Quality Assurance and Quality Control (QA/QC) 

processes as they related to the standard reports available in ICIS. These processes 

involved quality control of data for limits, overdue schedules, and violations. The 

Quarterly Non-Compliance Report is also used by states as a data quality tool. On 

occasion, data quality issues are addressed on the ICIS monthly user calls. 

For PCS, states use the "Dummy" audit capability to review data and reject 

transactions before transferring the data into the system. They follow up by reviewing 

the rejected transactions from the "Live" audit reports. 

Report Recommendation 4: 

	 Provide reminders to EPA regional offices and state environmental offices to 

use the standard Form 2040-0004 for data entry. Where this is not feasible (e.g., 

for state-specific reporting requirements), EPA should require completely 

reporting DMR data by using standardized Form 2040-0004 or alternative entry 

forms. Further, EPA should remind the states of the importance of 

documenting changes made to data elements after entry into the source systems. 

OECA Response: 

OECA agrees with the report that we should strive to better ensure states and 

regions are well informed on the proper forms and procedures for collecting and 

submitting data to EPA. OECA will take every available opportunity to remind EPA 

regional offices and states to use the Standard Form 2040-0004. This will include 

mechanisms such as the quarterly ICIS newsletter, monthly user calls, and electronic mail 

14
 
 

10-P-0230 

messages sent to individuals on the current ICIS list serve. Please note, however, that 

this form is also called Standard Form 3320-1 and both numbers appear on the physical 

document. Because our current user community and documentation refer to SF-3320-1 

and we do not wish to confuse submitters, we will continue to refer to SF-3320-1 in our 

communications. 

Additionally, all the State and Federal electronic DMR systems are 3320-1 

compliant. 

Report Recommendation 5: 

 Complete new rules that require states to report minor facility data. 

OECA Response: 

OECA strongly agrees with the recommendation to complete the rulemaking 

process requiring the submission of minor facility data. OECA is currently developing a 

rule which will significantly improve the NPDES data quantity and quality nationally. A 

key component of this rulemaking is the requirement for electronic reporting of certain 

NPDES information from the permittees, including non-major facilities, to states and 

EPA. This information is currently required to be reported by the permittees to the states, 

but there is no regulatory requirement for states to provide that specific information to 

EPA. This NPDES information is generally obtained on paper forms rather than 

electronically, although some states have developed electronic reporting tools. 

Therefore, this rule may likely change the mode by which the permittees report, without 

requiring additional information to be collected and submitted by the permittees, and 

reducing the burden on state programs caused by the handling of paper forms.. 

Key EPA management decisions regarding the specifics of the rulemaking are 

still pending, awaiting ongoing detailed analyses. However, this proposed requirement 

for electronic reporting of NPDES information from the permittees may likely include 

information such as discharge monitoring reports (DMRs), program reports, notices of 

intent to discharge, etc. As a supplement to that information from permittees, other 

information will likely be required to be reported by the states to EPA regarding the 

states’ implementation activities (such as inspections, violation determinations, 

enforcement actions, and permit issuance) for major and non-major facilities as well as 

for NPDES subprograms (e.g., pretreatment, biosolids, CAFOs, etc.). 

EPA currently projects issuance of this proposed rule in April 2011. Following a 

comment period, EPA expects to issue a final rule regarding this matter circa September 

2012. 

Report Recommendation 6: 

 Place a notice on the EPA ECHO website stating that ECHO does not contain 

all minor facility data. The wording and placement of the notice should be 

15
 
10-P-0230 

designed to help provide the public with transparency into the actual data 

elements maintained in ECHO. 

OECA Response: 

OECA agrees that the public needs to be aware that not all minor facility data is 

currently represented in ECHO. OECA does provide several notices to the public on the 

limitations of the data provided by ECHO. OECA has recently added several new 

features that allow ECHO users to find additional detailed information about the quality 

and completeness of information, including an interactive map that shows the data entry 

percentage for minor discharge monitoring data in each state. However, this information 

can be difficult to find, and OECA agrees this information can be better organized. For 

example we currently use grey text to indicate that there are data caveats. We will 

enhance this grey text so it is more recognizable to ECHO users. OECA will evaluate 

each area where the existing notice is displayed and determine if it needs to be enhanced. 

OECA will also evaluate its ECHO site as a whole and determine if additional notices 

need to be added. OECA regularly enhances the ECHO website, and will continue its 

practice of providing the public with caveats concerning the data. 

Report Recommendation 7: 

	 Develop and implement procedures to test ICIS-NPDES programming code 

before it is placed into production. 

OECA Response: 

OECA has extensive, documented procedures in place to test ICIS-NPDES 

programming code before it is placed into production. It follows best practices used by 

the information technology industry as well as EPA system life cycle policies and 

procedures. The prime contractor for the ICIS system is certified by the Software 

Engineering Institute (SEI) to be Level 3 on the Capability Maturity Model Integration 

(CMMI) scale. This is the same high level that is required for Department of Defense 

contractors. This certification requires the contractor to follow repeatable, standard 

processes for verification of all products. In addition, OECA completes and documents 

all items that are required by the EPA National Computing Center’s Automated 

Deployment Checklist prior to each software release. 

As part of the standard life cycle process, procedures for thoroughly testing the 

software for ICIS-NPDES were implemented during the 2004/2005 timeframe. This was 

the period during which the software for ICIS-NPDES was being initially developed and 

tested prior to its release into production. These procedures have been refined and 

improved over the years, and, because of this, new and revised software is typically 

implemented into production with few, if any, defects. The software goes through Unit 

Testing, Functional Testing, and End-to-End or System Integration Testing by the ICIS 

contractor to ensure it both meets stated requirements and executes correctly before it is 

turned over to EPA and the user community for User Acceptance Testing. For new 

16
 
10-P-0230 

software relating to major new capability, the software also goes though Beta Testing 

with experts on the ICIS Project Team prior to User Acceptance Testing. This ensures 

that as many defects as possible are found and fixed prior to user community testing. All 

testing is conducted according to Test Plans which include documenting test cases and, 

upon execution, the results of executing the test cases. Problems identified during each 

test phase are corrected and retested as necessary prior to the software being released into 

Production. Testing results are reviewed during the Bi-weekly Project Management 

meetings held between EPA and the ICIS contractor. 

17
 
10-P-0230 

Appendix B 

Distribution 

Office of the Administrator 

Assistant Administrator for Enforcement and Compliance Assurance 

Assistant Administrator for Environmental Information and Chief Information Officer 

Regional Administrator, Region 3 

Regional Administrator, Region 4 

Regional Administrator, Region 5 

Director, Office of Compliance, Office of Enforcement and Compliance Assurance 

Acting Director, Office of Information Collection, Office of Environmental Information 

Agency Follow-up Official (the CFO) 

Agency Follow-up Coordinator 

General Counsel 

Associate Administrator for Congressional and Intergovernmental Relations 

Associate Administrator for External Affairs and Environmental Education 

Agency Follow-up Coordinator, Office of Enforcement and Compliance Assurance 

Inspector General 

18