Drinking Water: Unreliable State Data Limit EPA's Ability to Target Enforcement Priorities and Communicate Water Systems' Performance
Released: 2011-07-19T00:00:00

Letter

Background

This section provides information on the risks posed by unsafe drinking water, the authority EPA gives to states under SDWA, differences among community water systems, EPA and the states’ processes for entering water systems’ data into SDWIS/Fed, EPA’s Enforcement Response Policy and Enforcement Targeting Tool, and EPA’s strategic targets for compliance with SDWA.

Risks of Unsafe Drinking Water

While the nation’s public drinking water supplies are much less prone to outbreaks of waterborne diseases such as cholera and typhoid than they were in the 19th and early 20th centuries, waterborne-disease outbreaks caused by microorganisms do still occur. For example, according to a 2006 study, an estimated 4.3 million to 11.7 million annual cases of acute gastrointestinal illnesses in the United States are attributable to drinking water from community drinking water systems supplied by surface-water and ground-water sources.

Other contaminants found in drinking water may also pose a threat to human health from long-term exposure at certain levels. For example: Long-term exposure to disinfectants—such as chlorine—that are added to water to control microorganisms and the byproducts of disinfectants may cause anemia, stomach discomfort, and eye or nose irritations. In small children and infants, inappropriate exposure to disinfectants could lead to nervous system problems. In addition, long-term ingestion of water with disinfection byproducts may increase the risk of cancer and may affect the nervous system, liver, and kidneys.

Arsenic, which occurs naturally and in industrial waste, may cause skin damage and circulatory system problems and increase the risk of cancer if it is not treated.

Lead and copper introduced into drinking water from the corrosion of household plumbing systems or the erosion of natural deposits may cause liver or kidney damage. Long-term exposure to lead may delay physical or mental development in infants and children.

Nitrate, which comes from fertilizer runoff, septic tanks, and erosion of natural deposits, is especially harmful to infants below the age of 6 months, and exposure may cause shortness of breathe, a serious illness known as blue-baby syndrome, and, if left untreated, death.

EPA Provides Authority to State Primacy Agencies

Under SDWA, EPA may authorize states meeting specified requirements to implement the PWSS program—referred to as primacy authority. For example, states must have regulations for contaminants that are no less stringent than those promulgated by EPA, adequate record keeping and reporting requirements, and adequate enforcement authority to compel water systems to comply with drinking water requirements. EPA has approved primacy authority for 49 states, 5 territories, and the Navajo Nation. EPA’s regions administer the programs in Wyoming and the District of Columbia, and for most tribes. EPA provides annual grants through the PWSS program to the states using a formula that takes into account population, geographical area, and the number of water systems covered. (EPA may also consider other relevant factors in its allocation formula.) In recent years, total EPA allocations to these grants have averaged about $100 million per year. States must provide matching funds; under the act, the PWSS grant can provide no more than 75 percent of the costs expended by a state to carry out its PWSS program. EPA’s drinking water program guidance instructs EPA regions to work with the states to develop grant workplans that include the states’ commitments to report key activities. For example, the State of Washington’s workplan includes a commitment to assure complete and accurate identification and reporting of public water system compliance.

Differences among Community Water Systems

As of July 2009, more than 51,000 community water systems supplied water to the same populations year-round. Community water systems vary widely in the number of people they serve, from 25 to over a million. As figure 1 shows, small systems are the most common. However, the 8 percent of community water systems that serve more than 10,000 people supply approximately 82 percent of all community water system users. Figure 1 shows the number of community water systems in 2009 categorized by the number of people they serve.

Community water systems obtain their water from groundwater reserves or from surface water sources. They may obtain, treat, and distribute their water entirely on their own, or they may purchase treated water from another system. Treatment generally consists of filtration, sedimentation, and other processes to remove impurities and harmful agents, and disinfection processes such as chlorination to eliminate biological contaminants.

Community water systems must meet a variety of health-based requirements under SDWA. These include providing drinking water that meets numerical limits for some contaminants, using treatment techniques for other contaminants, and using laboratory testing to monitor and report on the quality of the drinking water that they provide. Under SDWA, EPA may establish an enforceable standard—called a maximum contaminant level, or MCL—that limits the amount of a contaminant that may be present in drinking water. If EPA determines it is not economically or technically feasible to ascertain the level of a contaminant, the agency may instead establish a treatment technique to prevent known or anticipated health effects. In total, EPA has set MCL or treatment technique standards—known as the National Primary Drinking Water Regulations—for 89 regulated contaminants. We refer to violations of these standards as health-based violations.

EPA has also established monitoring, reporting, and other requirements for each of the 89 regulated contaminants. In this report we refer to these requirements collectively as monitoring requirements and refer to violations of these requirements as monitoring violations. These requirements may vary depending on several factors. For example, the frequency of monitoring may depend on whether the system obtains its water from ground water or surface water sources or upon the size of the water system. Additionally, if the water system detects certain contaminants above a specified amount, it may need to increase the frequency of its monitoring. Community water systems must also notify the public within specified times about the occurrence of health-based or monitoring violations and provide their customers with an annual Consumer Confidence Report containing data on the presence and concentrations of the 89 regulated contaminants.

The States’ Processes for Entering Water Systems’ Data into SDWIS

Most of the states enter data they collect and generate on community water systems into a version of SDWIS designed for use by the states known as SDWIS/State. As EPA promulgates new or revised regulations for particular contaminants, it develops new SDWIS versions to capture data associated with those regulations. EPA encourages the states to place their water systems’ data into SDWIS/State but the states may choose not to if they have an alternative database that meets their needs while also complying with EPA recordkeeping requirements. The data include inventory information about each system, such as its name, owner, address, and the size of the population it serves. The data also include the results of the water monitoring conducted according to contaminant- specific schedules by each system, the state’s determination of whether the system has committed violations, and a record of enforcement actions taken.

According to EPA, every 3 months, the states must transfer certain information from either SDWIS/State or their alternative data system to SDWIS/Fed. Specifically, the states transfer to SDWIS/Fed data on the health and monitoring violations identified and enforcement actions taken, and whether the state has determined that the system has returned to compliance. SDWIS/Fed is the data system EPA uses to gauge community water systems’ compliance with SDWA. In 2006, EPA and the Association of State Drinking Water Administrators set a goal that 90 percent of health- based drinking water violations be completely and accurately reported to SDWIS/Fed. EPA, however, does not have a goal for the completeness and accuracy of data on monitoring violations.

EPA’s Drinking Water Enforcement Response Policy and Enforcement Targeting Tool

Under its 2009 Drinking Water Enforcement Response Policy, EPA is to identify water systems with the most serious compliance problems and direct enforcement resources to these systems. An important component of EPA’s enforcement policy is its Enforcement Targeting Tool for identifying water systems with the most serious compliance problems. The Enforcement Targeting Tool assigns a score to each water system based on an accounting of unresolved violations over a 5-year period. Because some violations may have more serious health consequences than others, the tool assigns each violation a “weight” or number of points based on the potential threat to public health: acute health violations are worth 10 points, other health violations and some major monitoring violations are worth 5 points, and all other monitoring violations are worth 1 point. Additional points are added for each year a violation remains unresolved. Points for each violation at a water system are summed to generate the system’s score. Water systems whose scores meet or exceed a certain threshold—EPA has set the threshold at 11 points—are considered to have serious compliance problems and are placed on a priority list of water systems that the states and EPA are to target for enforcement. Using this approach, EPA and the states target resources to address those water systems that EPA determines have the most significant problems complying with SDWA’s requirements.

EPA’s Enforcement Response Policy also provides guidance on the amount of time in which states and EPA regions should address violations at priority water systems. Once systems have been targeted as a priority, states and the regions have 6 months to work with them in whatever manner they deem appropriate to resolve violations and return the system to compliance. Enforcement and compliance assistance may include a range of actions such as providing violation notification letters to systems, offering them technical assistance, conducting site visits to resolve violations, entering into compliance agreements such as consent orders, and additional formal actions such as issuing administrative orders, assessing fines or penalties, and filing or referring judicial cases. In situations where the system is unlikely to return to compliance within the 6-month time frame, EPA’s policy calls for states or the regions to take a formal enforcement action within 6 months that will put the system “on the path” to compliance by laying out future actions and time frames the system needs to follow. According to EPA, states and the regions are required to enter information on enforcement actions, including violation resolution, into SDWIS/State or the equivalent system as they occur and send those data to SDWIS/Fed every 3 months.

EPA’s Strategic Targets for Compliance with SDWA

EPA’s two most recent strategic plans issued in 2006 and 2010 have included the strategic objective of protecting human health by reducing exposure to contaminants in drinking water. These strategic plans are required by the Government Performance and Results Act (GPRA), which calls for related annual performance plans to outline the process for communicating goals and strategies throughout the agency, and for assigning accountability to managers and staff for goal achievement. As we have previously reported, a clear relationship should exist between an agency’s long-term strategic goals and the performance goals in the annual performance plan. Successful organizations try to link performance measures to the organization’s strategic goals and, to the extent possible, have performance measures that will show annual progress toward achieving their long-term strategic goals. GPRA also requires that the agency publish an annual performance report communicating to managers, policymakers, and the public what was actually accomplished for the resources expended.

To help gauge its progress relative to its objective of reducing exposure to contaminants, EPA uses annual performance measures and strategic targets to track national rates of drinking water compliance, including the: percentage of community water systems that meet all applicable health-based standards—the strategic target for 2009 was 90 percent; percentage of population served by community water systems that will receive drinking water that meet all applicable health-based drinking water standards—the strategic target for 2009 was 90 percent; and percentage of person months during which community water systems provide drinking water that meets all applicable health-based standards—the strategic target for 2009 was 95 percent.

EPA uses the data on violations that the states report to SDWIS/Fed to gauge the performance of community water systems in relation to these GPRA strategic targets. According to EPA, the SDWIS/Fed data indicated that community water systems either met, or came close to meeting, these strategic targets in 2007 through 2009. As part of its effort to achieve the objective in its 2010 strategic plan to reduce exposure to contaminants in drinking water, EPA has also adopted performance indicators that it will use to track the number and percentage of small water systems with repeat health-based violations and the average time for those systems to return to compliance.

SDWIS Data from States Did Not Reliably Reflect Community Water Systems’ Violations of SDWA or the Status of Enforcement Actions

The data that states provided to EPA did not reliably reflect the frequency of community water systems’ violations of SDWA’s health-based standards, according to our analysis of EPA’s audit data for 2007 and 2009 and past EPA audit reports. In addition, the data did not reliably reflect the frequency of monitoring violations, which are a predictor of health-based violations. Survey respondents support the concept of EPA setting a numerical goal for the percentage of monitoring violations accurately reported in order to increase the reliability of data in SDWIS/Fed. Furthermore, data provided by the states on the status of enforcement actions taken against systems with violations were incomplete, according to EPA and state officials we interviewed. Officials identified several factors, such as inadequate training, staffing, and guidance, as contributing to errors in data on violations and enforcement.

States’ Data Did Not Reliably Report the Frequency of Health-based Violations

Using EPA’s 2007 and 2009 audits of the data that the states provided to SDWIS/Fed, we found that the states did not completely and accurately report health-based violations committed by community water systems. For example, we estimate that the 19 states EPA audited in 2007 did not report or reported inaccurately 20 percent, or 543, of the health-based violations that EPA determined should have been reported. For 2009, we estimate that the 14 states EPA audited in that year did not report or reported inaccurately 26 percent, or 778, of the health-based violations that EPA determined should have been reported. Figure 2 shows our estimates of the percentage of health-based violations the states did not report or inaccurately reported.

EPA’s audits from 1996 through 2004 also found that violations had been unreported. For example, on the basis of its 2002 through 2004 audits, EPA reported that the 37 states it audited did not report or inaccurately reported about 49 percent of health-based violations committed by community water systems to SDWIS/Fed, as shown in figure 3. It is not possible to infer a trend between 2002 and 2009 because EPA’s 2002-2004 results are not directly comparable to the results of our analysis of 2007 and 2009 audit data. That is because, among other things, EPA’s analysis combined 3 years of audits and because the audits were of different states than were audited in 2007 and 2009. In its analysis of the completeness and accuracy of state data for all types of public water systems, EPA found that the reliability of state data on violations varied for different health- based standards. For example, data on violations of the total coliform rule and surface water treatment rules were more reliable than data on violations of the lead and copper treatment technique standards.

EPA’s audits have shown two types of errors in the data the states submitted. The first type, known as a compliance determination error, occurs when a violation occurs but the state (or EPA region acting as a primacy agency) does not issue a violation notice to the water system and does not report that violation to SDWIS/Fed. The second type, known as a data flow error, occurs when the state or region issues a violation notice to the water system and is reported to the state data system but information about the violation is not correctly transferred to SDWIS/Fed. Compliance determination errors, according to our analysis of EPA’s data, are much more common than data flow errors. For example, using EPA’s audit data from 2009, we estimate that 91 percent of the errors in health- based violations between the audited data and SDWIS/Fed were compliance determination errors and 9 percent were data flow errors. Among these errors were some state-reported violations to SDWIS/Fed that EPA determined had not occurred (e.g., false positives).

States’ Data Did Not Reliably Reflect the Frequency of Monitoring Violations, Which Are a Predictor of Health-based Violations

According to our analysis of EPA’s audit data from 2007 through 2009, the states did not report or inaccurately reported the number of monitoring violations. For example, we estimate that the 14 states audited in 2009 did not report or inaccurately reported about 54,600—or 84 percent—of the monitoring violations committed by community water systems to SDWIS/Fed. On the basis of these audit results, we conclude that the total nationwide number of actual monitoring violations had to have been considerably higher than the 82,000 reported in SDWIS/Fed. Monitoring violations, as we have defined them in this report, include a variety of situations, ranging from instances in which a water system did not do required monitoring, did not report the results to the state on time, or did not issue public notice of a health-based violation in a timely fashion. It is important to note that the underreporting of monitoring violations may affect what is known about health-based violations. Some unknown percentage of both reported and unreported monitoring violations may have hidden the presence of a health violation, particularly when the violation was that required monitoring was not done at all.

Our analysis found that having a monitoring violation was a strong and statistically significant predictor of whether a system had a health-based violation, among systems sampled in EPA’s audit data for 2007 to 2009. Furthermore, the number of monitoring violations was positively and statistically significantly related to the rate of health-based violations. In its 2010 report on 2007 and 2008 national drinking water compliance rates, EPA noted that monitoring violations were a concern because “if a water system does not monitor and report on the quality of its water it is impossible to know if there are health-based violations.” Therefore, the presence of monitoring violations may “mask” the presence of health- based violations. The total number of “masked” health-based violations is unknown, but may be affected by the total number of monitoring violations. As we have shown, the total number of monitoring violations is much higher than indicated by the SDWIS/Fed data, suggesting that the total number of health-based violations is also larger than indicated.

Majority of Survey Respondents Support a Goal for the Quality of Data on Monitoring Violations

Regarding the low quality of SDWIS/Fed data on monitoring violations, a majority of survey respondents who expressed an opinion (20 of 34) indicated they thought EPA should—as it has for health-based violations— establish a numerical data quality goal for the percentage of monitoring violations that are completely and accurately reported. These respondents had a range of views on what a numerical goal should be, from a low of 41 percent to a high of 100 percent; the average was about 83 percent. When asked to describe the actions they thought EPA and the states need to take to achieve their preferred goal, the most common responses focused on increasing management prioritization, training, and information system technology. When respondents who indicated they thought EPA should not establish a data quality goal were asked to explain their answers, the most common response was that the quality of data on monitoring violations was not a high priority. For example, one respondent said that unless the states and EPA can address inadequate staffing, monitoring violations will continue to be the lowest priority.

State Data on Enforcement Actions Were Incomplete, According to State and EPA Officials

EPA’s data verification audits in 2005 through 2009 did not include any analysis of the accuracy of the data the states reported to SDWIS/Fed on their enforcement actions. According to EPA’s audits for 2002 through 2004, the audited states did not accurately report to EPA 27 percent of the enforcement actions they took against community water systems. However, EPA arrived at this estimate by comparing data in SDWIS/Fed with the data in SDWIS/State to determine whether they matched. EPA did not examine original source documents that could have shown whether the data in SDWIS/State accurately represented the status of the states’ enforcement actions. The approach EPA used in its audits to estimate the accuracy of the reporting of enforcement actions by states differed from the approach it took in its audits of violations data, in which EPA examined the sampling data that community water systems provided to the states. Consequently, EPA’s estimates of the completeness and accuracy of enforcement data were less likely to be as reliable as its audits of violations data.

EPA has not conducted recent audits of enforcement data, but officials we spoke with from EPA’s drinking water and enforcement offices and three regions—as well as survey respondents—stated that current SDWIS/Fed data underreport the percentage of water systems where enforcement actions have been taken. They also indicated that the SDWIS/Fed data do not accurately report the percentage of water systems that have returned to compliance. For example, state officials told us that when they have quarterly discussions with the regions about the status of enforcement actions as shown in SDWIS/Fed they discover that the database is not accurate because the states have not consistently entered the data on enforcement actions into SDWIS/State.

We examined violations that occurred from 2005 through 2009 to determine what the states have reported to EPA. According to our analysis of SDWIS/Fed data on enforcement, the states reported that less than half of these health-based and monitoring violations were resolved as of March 31, 2010 (see fig. 4). Specifically, we found that about 59 percent of health- based violations and about 49 percent of monitoring violations committed by community water systems had not been resolved. However, given that the enforcement data have not been audited for several years, as well as the concerns of officials we spoke with, we cannot be certain that the results of our analysis accurately reflect the status of enforcement actions.

EPA and State Officials Indicated That Violation and Enforcement Data Are Unreliable for Several Reasons

EPA and state officials responding to our survey or in interviews cited several factors as contributing to inaccuracies in SDWIS/Fed data on health-based and monitoring violations and the status of enforcement. For the violations data, some factors were cited as contributing to both compliance determination and data flow errors—such as inadequate training and guidance—but the importance of these factors varied by the type of error. For enforcement data problems, other factors were often cited, such as higher priorities, inadequate guidance, and information system flaws.

Incorrect Compliance Determination Data and Data Flow Errors Occur for Several Reasons, According to EPA and State Officials

We asked survey respondents whether they thought any of five factors (information system structure; training by state or federal agencies; funding from state or federal agencies; state staffing levels; or guidance from EPA) contributed to incorrect compliance determinations and to the less common data flow errors; we also asked them to indicate if other factors were important. As figure 5 shows, at least half of the 41 respondents identified each of the factors as contributing to compliance determination errors, with training and staffing cited most often. And, as the figure shows, more than two-thirds of the 41 respondents cited the information system structure as contributing to data flow errors and more than half of respondents cited state staffing and training as contributing factors.

Respondents also provided more detailed information on the factors they identified as contributing to incorrect compliance determinations and data flow errors. For example, with regard to incorrect compliance determinations, one respondent said that training for new drinking water rules was limited and training for old drinking water rules was virtually nonexistent. Another respondent said that staffing levels were at an all- time low while another said that states had always experienced a revolving door for compliance staff. An EPA official responded that a state might not issue a violation because of “sympathy” for a water system if the state viewed the violation as not being a major health problem. With regard to data flow errors, several respondents said that SDWIS does not have adequate quality control features to clearly identify errors that might occur when the states transfer violations data from SDWIS/State to SDWIS/Fed.

When we asked survey respondents to identify the most important steps they believe that EPA could take to address compliance determination errors, the most frequent suggestions—from 18 of the 41 respondents— concerned training and guidance. For example, 9 respondents said that EPA needs to improve the timeliness of guidance on how to make compliance determinations to ensure that the guidance does not come after the date that a drinking water rule takes effect. With respect to actions states should take, the most frequent comments related to management and training, from 25 and 22 respondents, respectively. Many of the comments regarding management called for states to conduct more thorough oversight or to hold their staff accountable.

Survey respondents also identified the most important steps they believe that EPA could take to address data flow errors. The most frequent suggestions for lowering the error rate concerned information system structures and management, from 16 and 15 respondents, respectively. With respect to information systems, respondents said that EPA should take action to address the quality, complexity, or ease of use of SDWIS. Most who commented on management called for more oversight and accountability.

Unreliable Data on Enforcement Actions Are Attributed to Higher Priorities, Inadequate Guidance, and Information System Flaws

According to EPA and state officials we interviewed, as well as survey respondents, the factors that contributed to concerns about incomplete data on enforcement actions and water systems’ return to compliance are similar to those that contributed to unreliable data on violations. For example, EPA and state officials told us that some state agencies have not routinely and thoroughly entered data on enforcement actions or returns to compliance into SDWIS/Fed because it is a low priority for their limited staff. Officials from EPA regions said this is particularly the case for monitoring violations that states may have considered less serious than violations of health standards.

State and EPA officials also cited a lack of guidance from EPA on what conditions must exist for a system with a violation to be recorded as returned to compliance as having been a factor contributing to incomplete data on enforcement. Recognizing the importance of these definitions, EPA collaborated with states and the Association of State Drinking Water Administrators on guidance it issued in 2010. EPA regional officials told us the new definitions would likely lead to improvements in the states’ reporting on returns to compliance. In addition, officials we spoke with stated that SDWIS/Fed used to have an automated function that would categorize some common violations as returned to compliance if certain subsequent conditions existed. However, that function is no longer available, meaning that state officials need to enter the information manually. In its comments on a draft of this report, EPA said that the function was removed because it did not work correctly.

See appendix II for more details on the results of our survey.

Incomplete and Inaccurate SDWIS/Fed Data Hamper EPA’s Ability to Manage the PWSS Program and Communicate Progress toward Its Strategic Objective

Incomplete and inaccurate data on violations and enforcement actions limit EPA’s ability to identify water systems with the most serious compliance problems and ensure its enforcement goals are met. Unreported violations and unreliable enforcement data also impede EPA’s ability to monitor and fully communicate to Congress and the public the agency’s progress toward its strategic objective of reducing the public’s exposure to contaminants in drinking water.

Incomplete and Inaccurate Data on Violations and Enforcement Actions Reduce EPA’s Ability to Ensure Its Enforcement Goals Are Met

Incomplete and inaccurate data on violations and enforcement actions reduce EPA’s ability to ensure that it is achieving its goal of targeting for enforcement those systems with the most serious compliance problems. Specifically, the lack of reliable data in SDWIS/Fed reduces the usefulness of EPA’s Enforcement Targeting Tool for identifying water systems with the most serious compliance problems. That is, water systems without a complete violations record in SDWIS/Fed could receive a lower enforcement targeting score indicating a higher level of compliance than other systems whose violation record is complete. Conversely, systems whose return to compliance has not been recorded in SDWIS/Fed could receive a score that is higher, or worse, than warranted. According to EPA’s current enforcement policy, water systems whose scores equal or exceed 11 points are considered to have serious compliance problems and are targeted for enforcement actions.

To demonstrate the effect that unreported health and monitoring violations have on the implementation of the Enforcement Targeting Tool, we calculated two scores for each community water system audited by EPA in 2007, 2008, and 2009—a total of 1,225 systems over the period. One score was based on more complete data incorporating the violations found in the data verification audits, and the other score was based on violations in SDWIS/Fed. Because the audited data are a more complete dataset, we expected to see, and indeed found, differences between the two scores for each system. For 16 percent of the systems, the point difference between the two scores alone equaled or exceeded the 11-point threshold. Another 14 percent had scores that were 6 to 10 points higher, which would increase the likelihood that these systems would have been prioritized for enforcement under EPA’s targeting tool. The results of our analysis are shown in figure 6.

Overall, according to our analysis, 73 percent of the water systems (or 892) had a different score using the two sets of data. Twenty-seven percent (333) of the water systems showed no difference between the scores calculated using the two sets of data. We found that the majority of score differences were the result of unreported monitoring violations. The Enforcement Targeting Tool assigns a much lower weight to monitoring violations than to health-based violations, but, as previously discussed, the number of monitoring violations plays an important role in limiting EPA’s ability to identify systems with serious compliance problems. While most of the 1,225 water systems had a higher score with audited data than with SDWIS/Fed data, 2 percent (21 systems) had lower scores because EPA found in its audit that the violations had not occurred.

When the SDWIS/Fed data are incomplete, EPA’s ability to identify and set priorities for enforcement in water systems is compromised. For example, because the Enforcement Targeting Tool uses SDWIS/Fed data that may be missing violations, some systems may not be assigned enough points to exceed EPA’s threshold of 10 points for priority enforcement action. For some of these systems, one or two additional points may be all that are needed to exceed the threshold and in other cases, as described below, the point difference for a particular system exceeded the threshold by an extraordinary amount. For example, we calculated the following for three systems: A 170 point difference: The score we calculated was 3 points for one water system in Vermont using SDWIS/Fed data, but 173 when we accounted for unreported health and monitoring violations that EPA found in its 2009 audit.

A 138 point difference: The score we calculated was 0 using SDWIS/Fed data for a tribal system in New York that EPA’s Region 2 office oversees as the primacy agency, but 138 when we accounted for unreported health and monitoring violations found during EPA’s 2009 audit.

A 95 point difference: The score we calculated was 0 using SDWIS/Fed data for a system in Utah, but 95 when we used data from EPA’s audit. The difference was entirely attributable to unreported monitoring violations.

Our analysis echoes concerns voiced by respondents to our survey; 22 of 41 respondents indicated that the usefulness of EPA’s Enforcement Targeting Tool is affected by limitations in the SDWIS/Fed database. One respondent said “missing data will significantly affect the usefulness of the results.” Another respondent said the tool “is hinging on the information recorded in SDWIS/Fed” and that “the tool is as good as the data provided.”

Survey respondents and state and EPA officials also reported that incomplete or inaccurate data on the resolution of violations could result in a water system receiving a higher score for enforcement priority than it merits. EPA and state officials told us that states do not always indicate in SDWIS/Fed that a violation is resolved, perhaps causing the Enforcement Targeting Tool to mistakenly place the system on the targeted enforcement list. According to one survey respondent, this condition will “confuse states and lead to continued poor quality data.” Another respondent said that use of the Enforcement Targeting Tool “is a waste of time” without steps taken to fix this issue. State officials told us that in their regular review of the targeted list with EPA regional officials, they can recognize when a system has been erroneously included on the targeting list because resolved violations were not recorded and they can correct the discrepancy. However, EPA officials have told us this data correction process is a time-consuming one that places additional demands on limited state and EPA enforcement staff.

Incomplete SDWIS/Fed data can also limit EPA’s ability to ensure that the states meet the agency’s enforcement goal that targeted systems have returned, or are returning, to compliance in a timely fashion. EPA’s Enforcement Response Policy calls for states to work with systems to resolve violations or put the system on a “path to compliance” within 6 months of when the system becomes a priority system on an Enforcement Targeting Tool list. However, unreported data on enforcement actions can hamper EPA’s ability to determine whether states have met that goal. For instance, while states might take an enforcement response that leads, or will lead, a water system to resolve the violation, states frequently do not enter this information into the SDWIS/Fed database or enter the information months or years later, according to EPA and state officials we spoke with. Either situation hampers EPA’s ability to track the timeliness of enforcement responses.

Unreported Violations and Enforcement Data Impede EPA’s Ability to Monitor and Report Progress Toward Its Strategic Objective of Reducing Exposure to Contaminants in Drinking Water

Unreported violations and enforcement data impede EPA’s ability to fully measure and communicate its progress toward meeting the strategic objective of reducing human exposure to contaminants in drinking water. The agency has established a number of indicators and targets that it uses to measure its progress toward meeting that objective. However, the unreliable quality of the violations data and concerns about the accuracy of enforcement data in SDWIS/Fed make it difficult for EPA to reliably communicate the relative public health risk posed by community water systems’ noncompliance with SDWA and the progress made in resolving noncompliance in a timely manner. For example: EPA’s 2011 national water program guidance contains an indicator for the number and percentage of systems serving less than 10,000 people with certain repeated health-based violations. EPA’s ability to set and reliably use this type of indicator requires complete and accurate data on violations, but as we have shown, the SDWIS/Fed data on violations are not reliable.

EPA’s 2011 national water program guidance also contains an indicator for the average time taken for systems serving less than 10,000 people to return to compliance after committing certain health-based violations. However, the ability to set and reliably use an indicator of this type requires complete and accurate data on enforcement actions. As we have previously indicated, EPA and state officials we interviewed told us the enforcement data in SDWIS/Fed are not reliable.

Unreliable data quality also limits EPA’s ability to introduce or modify targets to manage its program and communicate progress in meeting the program’s goals. Quality data are necessary to accurately measure performance relative to strategic targets. Two key EPA strategic targets associated with the agency’s strategic objective of reducing exposure to contaminants in drinking water—the percentage of community water systems that met all health-based standards and the percentage of the population served by community water systems that received drinking water that met all applicable health-based drinking water standards—are broad measures of compliance. However, these measures do not provide information on the relative severity of the violations or account for systems that have multiple health-based violations, offering the public a narrow view of the quality of the nation’s water systems and not clearly communicating the public health risk posed by these systems’ noncompliance with SDWA. For example, a water system with multiple health-based violations is effectively “counted” the same as a system with one health-based violation. Thus, the relative health risk posed by different systems’ noncompliance is not apparent. Without complete and accurate SDWIS/Fed data it is difficult to develop a new measure or modify these strategic targets. Similarly, without complete and accurate data from the states, EPA will be unable to establish reliable measures or targets regarding the rate of reduction in health-based violations or compliance with monitoring requirements or further EPA’s core value of transparency.

Actions EPA and States Are Taking to Improve the Quality of Data in SDWIS Have Not Been Fully Successful and More Actions Are Planned

EPA and the states have taken actions over many years to identify and address the causes of incomplete and inaccurate violations data, but those efforts have not been fully successful, according to those we surveyed. EPA has conducted audits to assess the quality of state violation data in SDWIS/Fed and developed recommendations for improving data quality. Survey respondents generally reported that those audits have contributed to improvement, but EPA has discontinued them. EPA and the states also established work groups to address data management and quality. In addition EPA has emphasized the importance of specific data quality management tools, although it has not required states or regions to use them. More recent EPA initiatives include a new strategy for data sharing, plans to redesign SDWIS, and a new tool to help the states make and report compliance determinations and enforcement actions.

According to Survey Respondents, EPA’s Audit Recommendations Contributed to Improving Data Quality, but the Agency Has Discontinued Them, at Least Temporarily

As described earlier, EPA used its data verification audits to assess the quality of the violations data and, to a lesser extent, the enforcement data the states have submitted to SDWIS/Fed. The agency also used the audits it conducted from 1996 through 2004 to develop state-specific and national recommendations for improving data quality. EPA and state officials we surveyed had mixed, but generally favorable, views about the value of the audits’ recommendations with regard to improving data quality. Eight respondents said the recommendations were very effective in improving data quality, while most respondents (26 of 39) said the recommendations were only slightly or moderately effective.

According to respondents, the audits pointed out states’ inefficiencies and poor practices. For example, one respondent said that states are able to use the results as a guide to improve training for staff and improve data quality. Despite the recommendations offered to help states, six respondents indicated that the states or regions did not adequately change their practices in response to the audit findings. For example, one EPA headquarters manager commented that states may incorrectly interpret systemic problems identified through the audit as isolated problems to be corrected only at the water systems covered by the individual audits. Nonetheless, seven respondents stated that the audits’ scope or methodology was not adequate to determine data quality.

EPA discontinued the audits of violations data in 2010 due to funding constraints. According to the Director of the Office of Ground Water and Drinking Water, EPA may be able to resume the audits in 2011, but at a much reduced number. EPA conducted an average of about 17 audits of states, regions, and other primacy agencies in 2007 through 2009, but the director told us in December 2010 the agency may be able to do 4 or 5 in 2011. EPA had not done any 2011 audits as of June 2011. In its comments on a draft of this report, EPA said that the Office of Water will conduct six to eight audits in 2011.

EPA and the States Have Established Joint Work Groups to Address Data Issues

In the 1990s, EPA and the states jointly established two work groups charged with providing analysis and recommendations on various aspects of data management and formed a third such group in 2010. According to most of their members, the two older groups—the Data Management Steering Committee and Data Technical Advisory Committee—were effective at helping EPA and the states improve data quality. On the other hand, the members of the newer Data Quality Work Group were almost evenly divided on whether this new group has been effective or not effective.

The Data Management Steering Committee is charged with supporting EPA and the states in their cooperative efforts to enhance management of drinking water data. In explaining their answers to a question about the group’s effectiveness, one-third of the members said the committee had helped EPA and the states understand the nature of the data quality problem and about one-half said it provided direction. However, one-third of the members commented on the lack of implementation of the committee’s recommendations.

The Data Technical Advisory Committee is responsible for recommending ways to obtain the data EPA needs to carry out its PWSS program responsibilities. About three-fifths of the members commented that the committee had helped EPA and the states understand data problems and had provided direction. However, similar to the steering committee, advisory committee members had concerns about EPA’s implementation of recommendations, with close to half saying that the agency’s implementation had been inadequate.

The Data Quality Work Group met several times in 2010 and outlined draft recommendations for improving data quality, including additional training, standard operating procedures for staff managing any new drinking water rules, checklists of rule milestone dates for states, and quality assurance/quality control checks for SDWIS data. However, the Office of Ground Water and Drinking Water diverted the work group staff in 2010 to focus their attention on implementing the Administrator’s Drinking Water Strategy before the group could issue final recommendations, according to senior office staff. Perhaps in light of that, among the most common comments from members of the group was that the work group was too new to evaluate or that its activity level had been inadequate. The Office of Ground Water and Drinking Water staff noted they would consider the work group’s draft recommendations in its redesign of SDWIS.

The Office of Ground Water and Drinking Water Has Emphasized the Importance of Data Quality but Not Required States or Regions to Take Actions

In March 2008, EPA reported the results of the audits it conducted of state data in 2002 through 2004. As it had done in its prior reports on state audits, EPA included recommendations for improving data quality in this report; these recommendations took the form of an action plan in its 2008 report. The 2008 action plan was a joint effort of EPA and the Association of State Drinking Water Administrators to provide recommendations for achieving the goal set in 2006 of 90 percent complete and accurate data for health-based violations, as well as improving the quality of monitoring violations data. According to the action plan, the largest challenge was ensuring that all data reflecting determinations of violations were entered into SDWIS/Fed. The plan called for, among other things, the development of new data management tools as well as the implementation of tools that EPA has developed over the years to improve compliance determinations. However, as discussed below, widespread implementation of those tools has not yet occurred.

In April 2009, the Director of EPA’s Office of Ground Water and Drinking Water issued a memorandum to EPA’s regional water management directors calling attention to (1) the incomplete implementation of the 2008 action plan and (2) the importance of increasing oversight and accountability of the states for the quality of drinking water data. Noting that the quality of drinking water data in SDWIS/Fed was called into question in the media in the late 1990s and was the subject of a 2004 report by EPA’s Office of Inspector General, the director stated that the quality of data continued to be too low. She also cited the 2006 agreement between EPA and the Association of State Drinking Water Administrators to set a goal that 90 percent of health-based drinking water violations be completely and accurately reported to SDWIS/Fed and said that more than 10 states had met the goal, but the overall goal had not been met.

The Director of the Office of Ground Water and Drinking Water Requested Help in Implementing the 2008 Action Plan

In her memorandum, the director noted that one of the conditions of primacy the states must meet is to report all violations. To improve data quality, the director called upon the regions to increase their efforts to implement the 2008 action plan. Her memorandum and the action plan call for the states to increase their use of several data management tools that EPA has developed over the years to improve accuracy, including SDWIS/State, electronic data verification (eDV), and electronic reporting from laboratories to states. Although EPA has developed these tools to improve compliance determinations and data flow, the states are not required to use them as a condition of primacy or their PWSS grant agreements. We gathered information from EPA regarding the current status of these tools and asked survey respondents to comment on the factors that have limited the use of these tools and the steps they believed should be taken to increase the tools’ use.

EPA reported to us in March 2011 that eight states, one tribe, and one territory were not using SDWIS/State at all. Of the states that do use SDWIS/State, some report using the database only for particular drinking water rules. For example, an EPA survey of states in mid-2010 found that 20 of the 55 primacy agencies were using SDWIS/State to make compliance determination decisions for the surface water treatment rule and 35 were using it for the total coliform rule.

Due to limitations in the data available to us and inherent difficulties in establishing a cause-and-effect relationship, we could not determine whether a state’s use of SDWIS/State leads to more reliable data on violations of particular SDWA rules. However, 26 of the 41 respondents indicated that more widespread use of SDWIS/State would improve data quality; 5 respondents indicated it would not; and 10 indicated they did not know. Of the 26 respondents who provided detail on why they thought more widespread use of SDWIS/State would improve data quality, 17 indicated it would promote more consistent and accurate compliance determinations through automation. The most common theme among the respondents’ suggestions for what EPA should do to address the factors that have prevented full use of SDWIS related to the quality, complexity, and ease of use of the system. For example, one respondent said that EPA should be aware of the needs of drinking water managers, who are the principal users of SDWIS/State, rather than database managers.

Electronic Data Verification

According to EPA officials, as of March 2011, only seven states had done pilot tests of EPA’s electronic quality control tool for SDWIS/State, known as electronic data verification, or eDV. EPA officials told us that the eDV tool needs additional refinement to be fully compatible with SDWIS/State. eDV could assist states in making compliance determinations according to 18 of the 24 survey respondents who were familiar with it. They said that using the tool would improve data quality by improving states’ oversight or auditing capability. However, EPA and state officials told us this tool can be used only by states that use SDWIS/State to manage all of the drinking water rules, and survey respondents noted that the need to fully use SDWIS/State was a factor that prevented more states from using eDV. The survey respondents’ most common recommendation for EPA was to improve the quality and ease the use of the tool. For example, one respondent said that EPA needs to update the tool to keep pace with changes in regulations. At the same time, many respondents suggested that the states need to make a greater commitment to using SDWIS/State and eDV more.

Electronic Reporting from Laboratories to States

EPA has developed a tool that testing laboratories can use to electronically transmit the results of community and other public water systems’ monitoring directly to the state. However, according to EPA’s mid-2010 survey, only 19 states were using this tool. Of the 40 survey respondents who expressed an opinion, 39 believed that more widespread use of this tool would improve data quality, and 25 of these respondents stated the tool would reduce data entry errors and increase accuracy. Another 11 respondents said the tool would increase the speed of data exchange between the states and EPA, and 10 said it would free up state resources that could be used to improve data quality in other ways. For example, one respondent said that, in the long run, electronic reporting should save state resources by reducing the need for data entry staff but that in the short run, switching to electronic reporting requires technical support for the laboratories and additional resources.

According to our survey respondents, the two leading barriers to having more states require electronic reporting are (1) laboratories’ inadequate capability to implement the reporting technology and (2) the states’ lack of legal authority to make the tool’s use a requirement. Specifically, many survey respondents said that laboratories, particularly small ones, are not always adequately equipped or staffed to adopt electronic reporting. Several survey respondents said that EPA needs to provide support to laboratories to make it easier to adopt the tool. Some respondents also said that as EPA’s current SDWA regulations do not require electronic reporting and some state laws prohibit state agencies from including requirements in their PWSS programs that are more stringent than what is required by SDWA, the state agencies are unable to require electronic reporting. Nine respondents said that EPA should require electronic reporting.

Several respondents also identified EPA’s Cross-Media Electronic Reporting Rule as a regulatory barrier to more widespread use of electronic reporting. This rule provides the legal framework for electronic reporting under all of the agency’s environmental regulations.

The rule requires states, tribes, and local governments that wish to use electronic reporting for implementing authorized federal environmental programs to obtain EPA approval, which may require modifications to electronic reporting systems to meet EPA requirements. One state respondent said that many states do not have the financial resources to build a system to receive data electronically because of the constraints of this rule, noting that the rule places very tight requirements on the security required for receiving data electronically. Several respondents called on EPA to review the need for the rule or abolish it.

The Director of the Office of Ground Water and Drinking Water also Called for Increased Regional Oversight and Accountability

Citing the need for strong regional oversight of the states’ PWSS programs, the director also requested in her April 2009 memorandum that the regions take specific actions over and above those identified in the 2008 action plan. Specifically, she called for the regions to provide documentation of good standard operating procedures and lessons learned that may enable EPA to improve SDWIS/Fed data quality, among other things; discuss with states annually (or more frequently) the completeness and accuracy of the violations data reported to SDWIS/Fed, including a review of the state’s implementation of recommendations contained in previous data verification reports; and include language in future PWSS grant agreements indicating that the state must make compliance determinations that are consistent with applicable drinking water regulations, report all violations and enforcement actions to SDWIS/Fed in a timely fashion, and otherwise comply with 40 CFR §142.15. Also include any corrective action steps identified by data verification audits or program reviews in the state’s annual work plan.

Although the Director of the Office of Ground Water and Drinking Water requested these actions, she told us her office does not have the authority to require the regions—which report directly to the EPA Administrator— to do so. The director and other drinking water program managers we interviewed told us the regions had responded to her request, but the office had not assessed and could not document the extent to which the regions had complied with the requests to discuss data quality with the states or add relevant language to grant agreements. In its comments on a draft of this report, EPA stated that all of the regions have incorporated data quality into their discussions with states and data quality has been incorporated into grant agreements or state workplans. However, we were unable to verify these statements; in response to our request during the comment process, EPA said that documentation was not available. According to EPA, it made the statements on the basis of e-mail communications and discussions between the managers and staff in the Office of Ground Water and Drinking Water and regional management and staff.

The director also requested comments from the regions’ water management directors on four proposed measures that would assist the office in monitoring the regions’ oversight of the states’ performance, including several directly related to the steps discussed above: the percentage of states within a region with which the region has an annual discussion regarding data quality; the percentage of states that have an action plan to correct deficiencies relating to drinking water compliance determinations or data reporting that were noted in the most recent EPA data verification audit; the percentage of a region’s annual PWSS grants that include grant conditions requiring the states to make compliance determinations that are consistent with drinking water regulations; and the extent to which a region is achieving EPA’s goal that 90 percent of health-based violations are completely and accurately reported to SDWIS/Fed, when a region is acting as the primacy authority for a particular rule in a state.

According to Office of Ground Water and Drinking Water officials, the regions responded to the director’s request for comments but did not fully support the proposed performance measures, and none of these have been implemented as of March 2011.

EPA Administrator Issued New Strategy for Data Sharing and EPA Announced Plans to Redesign SDWIS

In March 2010, the Administrator of EPA issued a drinking water strategy that called for, among other goals, the agency and the states to increase data sharing on water systems. As part of this strategy, EPA announced it will redesign SDWIS to help to meet the Administrator’s goals for data sharing. According to the director of the Office of Ground Water and Drinking Water, software for the next generation of SDWIS is only at the beginning stages of development; she anticipates it will be ready by 2014, depending upon the availability of funding. To help achieve the goals of the strategy, EPA signed a memorandum of understanding in November 2010 with three associations that represent state agencies and officials. EPA also formed an Implementation Work Group comprising agency and state officials to further the data sharing goals spelled out in the memorandum.

The November 2010 memorandum of understanding on data sharing— which is a voluntary agreement among the parties—outlines the vision, goals, terms, and conditions under which drinking water monitoring data are to be exchanged between the states and EPA. The anticipated benefits of data sharing include allowing states to more readily compare their water system monitoring results with EPA regulations and with the SDWIS/Fed data before the states submit the data to EPA. EPA intends to use the shared data for a variety of purposes, including calculating national and state data completeness, accuracy, and timeliness; evaluating differences in state interpretations of EPA regulations; and conducting national program oversight. The mission of the Implementation Working Group is to recommend ways for states to share appropriate compliance monitoring data that eventually will be housed in the next generation of SDWIS.

EPA officials said they expect this redesign of SDWIS—and accompanying revisions to state data submission requirements—to expand the amount of data that EPA receives electronically from the states. With SDWIS/Fed, EPA generally only receives data from the states on inventories, violations, and enforcement actions. According to the Director of EPA’s Office of Ground Water and Drinking Water, the next generation of SDWIS would give EPA access to the compliance monitoring and enforcement data now collected by the states. She told us that having direct access to the states’ raw monitoring data would improve EPA’s ability to better understand national patterns of compliance and to diagnose problems faced by states. For instance, according to the director, the data could reveal that particular rules are hampered by a misunderstanding of the requirements, and EPA could use that information to write regulations that are easier to understand and report. The director said that a redesigned SDWIS could also reveal and address instances when a state has made a compliance determination error. However, she also said that some compliance determination errors can be addressed with a redesigned SDWIS, but others might need to be addressed through better training or writing regulations more clearly so that state staff understand what constitutes a violation.

According to the director, under the agency’s current position, states will continue to have the option to use the next generation of SDWIS. EPA will continue to provide ways for states that do not use SDWIS to transfer their data to the agency. However, EPA will expect those states, like those using SDWIS, to share their compliance monitoring data with EPA.

EPA Is Developing a New Compliance Determination and Violations/Enforcement Reporting Tool

In November 2010, the Office of Ground Water and Drinking Water unveiled an initial version of another tool for improving compliance determinations and data quality—the Compliance Determination and Violation/Enforcement Reporting Tool. According to the office, the tool was designed by Region 5 staff to consolidate, update, and supplement EPA guidance on SDWA violations of specific requirements in one electronic document. The tool includes violation descriptions, compliance determinations, violation reporting instructions, common discrepancies from data verifications, related EPA memos, enforcement tracking instructions, and return to compliance definitions. The target audience for the tool is state and regional compliance, enforcement, and data staff. The tool is being developed in modules for each National Primary Drinking Water Regulation. The first module of the tool was for the lead and copper rule. EPA officials said they anticipate all modules will be developed by the end of fiscal year 2011 if funding is available.

Conclusions

EPA relies on the soundness of state-reported data to ensure that community water systems are complying with SDWA and that the states and regions are taking appropriate enforcement actions against noncompliant water systems. As our analysis of EPA’s audit data for 2007 through 2009 shows, however, states continue to fall short in providing accurate and complete data on health-based and monitoring violations. EPA’s data verification audits after 2004 did not examine the quality of data on enforcement actions and systems’ return to compliance, but EPA officials and survey respondents told us that current state data on such actions are also incomplete or inaccurate. In addition to discontinuing its audits of enforcement data after 2004, EPA also discontinued its audits of violation data after 2009 because of budget constraints. The agency hopes to resume these audits of violation data in 2011, but the number of states to be audited would be greatly reduced from an annual average of about 17 from 2007 through 2009 to 6 to 8. Conducting fewer audits of state- reported data—both violations and enforcement data—will hamper the effectiveness of EPA’s oversight of the states and its ability to assess its efforts to improve data quality.

EPA and the Association of State Drinking Water Administrators established a goal of completely and accurately reporting 90 percent of health-based violations, but EPA has not set a similar goal for monitoring violations. Monitoring violations may reflect a wide range of circumstances, such as instances in which monitoring was done but was not reported to the state in a timely fashion or the potentially more serious situation in which required monitoring was not done at all. We found, however, that the number of monitoring violations was positively and statistically significantly related to the rate of health-based violations. Recognizing the importance of having complete and accurate data on monitoring violations, a majority of those state and EPA officials we surveyed who voiced an opinion supported the idea of having a goal for the quality of data on monitoring violations.

As called for by GPRA, EPA has established several performance measures with associated targets and indicators for community water systems to assess progress toward the agency’s strategic objective of reducing exposure to contaminants in drinking water. Each year, EPA publicly reports systems’ performance levels relative to the targets using data from SDWIS/Fed. To be useful and appropriate, these performance measures should clearly reflect conditions that directly relate to human exposure to contaminants. However, we found that some of the measures that EPA relies upon to gauge national compliance levels measure how many systems are out of compliance but not the extent to which they are out of compliance, which does not clearly communicate the public health risk posed by these systems’ noncompliance with SDWA. In addition, some measures that EPA uses depend on data on violations and the status of enforcement actions that are unreliable. We found that incomplete and inaccurate data could impede EPA’s ability to monitor and report progress toward its strategic targets for those measures, including having those systems return to compliance in a timely manner.

Recognizing its long-standing problem of receiving incomplete and inaccurate state data on violations, EPA has made efforts to improve the quality of data reported to SDWIS/Fed. However, many of those efforts, including those to implement EPA’s 2008 action plan, have not been fully successful. In light of the need for more improvement, the Director of the Office of Ground Water and Drinking Water requested in her 2009 memorandum that the regional water managers take numerous steps to implement the action plan—including encouraging the states to increase their use of SDWIS/State, electronic data verification, and electronic reporting from laboratories to states. Both the action plan and additional steps requested in the memorandum have the potential to address the factors EPA and state officials identified as contributing to unreliable data quality. According to the director, however, the states currently are not required to use the data management tools that EPA has developed as a condition of primacy or their PWSS grant agreements, and many have chosen not to do so. In response to our survey, EPA and state drinking water officials generally said these tools would help improve data quality but noted barriers they believe have prevented more widespread use of them. For example, the most common theme among the survey respondents’ suggestions for increasing the use of the SDWIS/State and electronic data verification tools was to address their quality, complexity, and ease of use. The Director of the Office of Ground Water and Drinking Water also asked the regional water management directors in her memorandum to increase their oversight of state programs. For example, she asked the regions to include language in future grant agreements indicating that the state, among other things, must make compliance determinations that are consistent with applicable drinking water regulations. However, the director also indicated that her office does not have the authority to require the regions to take the actions requested in the memorandum, and could not document the extent to which the regions had done so. EPA’s plan to develop a next generation of SDWIS and to increase its access to state data might help the agency ensure that it receives higher quality violations and enforcement data from the states. However, it is uncertain if and when the new system or increased access to data will be available. In the meantime, further efforts to overcome the barriers to implementation of the 2008 action plan and the director’s 2009 memorandum are needed to improve state data.

Recommendations for Executive Action

To improve EPA’s ability to oversee the states’ implementation of the Safe Drinking Water Act and provide Congress and the public with more complete and accurate information on compliance, we recommend that the Administrator of EPA take the following four actions: Resume data verification audits to routinely evaluate the quality of selected drinking water data on health-based and monitoring violations that the states provide to EPA. These audits should also evaluate the quality of data on the enforcement actions that states and other primacy agencies have taken to correct violations.

Work with the states to establish a goal, or goals, for the completeness and accuracy of data on monitoring violations. In setting these goals, EPA may want to consider whether certain types of monitoring violations merit specific targets. For example, the agency may decide that a goal for the states to completely and accurately report when required monitoring was not done should differ from a goal for reporting when monitoring was done but not reported on time.

Consider whether EPA’s performance measures for community water systems could be constructed to more clearly communicate the aggregate public health risk posed by these systems’ noncompliance with SDWA and progress in having those systems return to compliance in a timely manner.

Work with the EPA regions and states to assess the progress made in implementing the steps called for by the 2008 action plan and the Director of the Office of Ground Water and Drinking Water’s 2009 memorandum; identify the barriers that have prevented more widespread implementation of the action plan and memorandum; and develop and publish a strategy for overcoming those barriers.

Agency Comments and Our Evaluation

We provided a draft of this report to EPA for review and comment. The agency provided written comments, which are reproduced in appendix III. EPA partially agreed with two of our recommendations, disagreed with one, and neither agreed nor disagreed with another. Our responses to EPA’s comments on our recommendations follow, and our responses to EPA’s attachment of substantive comments are in appendix III. EPA also provided technical comments that we have incorporated as appropriate.

In its overall comments, EPA said that it recognizes the importance and value of high quality data to complement the activities that comprise its oversight of primacy agencies. EPA also acknowledged that GAO found data quality problems similar to those previously found by the agency during data verification audits and that underreporting violations data and enforcement actions may limit the public’s full knowledge of the status of public water system compliance. EPA noted that it has implemented a number of activities to improve data quality and its ability to oversee the drinking water program. We agree that EPA has taken steps to improve data quality and describe many of them in our report.

EPA also noted that complete and accurate data are important in order to effectively target enforcement to those systems with the most serious compliance problems. The agency added that its 2009 Enforcement Targeting Tool provides an incentive to the states to keep their enforcement data current to ensure that the tool yields accurate scores. We agree that the tool underscores the importance to the states of keeping enforcement data current. However, the scores generated by the tool will also be incorrect if data on the existence of violations are incomplete or inaccurate. We believe that whereas the use of the tool provides an incentive to the states to improve the accuracy of their enforcement data, it does not necessarily provide them an incentive to improve the accuracy of their violations data.

EPA partially agreed with our first recommendation that it resume data verification audits of violations and enforcement actions. The agency stated that it has found that data verification audits provide valuable information on data completeness and accuracy and that it plans to conduct six to eight audits during calendar year 2011. However, EPA did not commit to conducting data verification audits beyond 2011. Instead, EPA said that until the next generation of SDWIS is deployed, thus enabling the agency to view compliance monitoring data and compliance determinations directly, it will consider using data verification audits to evaluate data quality. EPA did not comment on how it would evaluate data on enforcement actions taken to correct violations. We understand that the next generation of SDWIS may enable EPA to more directly monitor water systems data and oversee the states’ compliance determinations. We note that the Director of the Office of Ground Water and Drinking Water said that the new system would not be available until 2014, depending on the availability of funding. We continue to believe that EPA should commit to, not merely consider, conducting data verification audits until the new system is available, and that those audits should also evaluate the completeness and accuracy of enforcement data.

EPA did not clearly indicate its agreement or disagreement with our second recommendation that it work with the states to establish a goal, or goals, for the completeness and accuracy of data on monitoring violations. The agency stated that it appreciates the need for improved data quality for those types of violations. However, EPA neither indicated that it would adopt a goal nor offered any reasons for why a goal—such as the one it has for the quality of data on health-based violations—would be inappropriate. Instead, the agency suggested that along with technology enhancements as part of the next generation of SDWIS, (1) it will consider changes to its approach to reporting violations data, and (2) will explore the possibility of revising the Enforcement Targeting Tool, which could improve its oversight capabilities. We are not able to evaluate these changes given their speculative nature, and it is not clear how they might be relevant to achieving a higher degree of data quality. EPA also stated that the regions’ annual incorporation of data quality in state grants and workplans will improve EPA’s oversight capabilities. We agree that increased emphasis from the regions is necessary and could lead to improved data quality. However, we continue to believe that setting a goal for the quality of data on monitoring violations would emphasize its importance and encourage the states to make and report correct compliance determinations.

EPA disagreed with our third recommendation that it consider whether its performance measures could be constructed to more clearly communicate the aggregate public health risk posed by systems’ noncompliance with SDWA. The agency noted that its program guidance currently includes a measure that attempts to address the duration (in “person months”) of time consumers may be exposed to health-based violations. We describe this measure in the background section of our report. However, we believe this measure has the same limitation as other EPA strategic targets, in that it does not distinguish between water systems with multiple health-based violations in a particular month and those systems with a single violation in that month. EPA also stated that it uses a variety of tools that may convey information on risks associated with noncompliance and show progress toward returning systems to compliance better than a new performance measure would. Among the tools EPA identified is a Web site containing detailed information about the violations for individual water systems. EPA also said that it recently posted drinking water data to its Enforcement and Compliance History Online tool. Similarly, EPA said water systems directly convey to their customers information on public health risks associated with violations through Public Notifications and Consumer Confidence Reports. We acknowledge these tools provide the public with details on the violations that states and water systems have reported for individual water systems. However, our recommendation encourages EPA to consider changes to its performance measures to provide Congress and the public a clearer understanding of noncompliance with SDWA at a national level and not elicit more information about the performance of individual water systems. We continue to believe that it is important for EPA to develop a national performance measure that helps gauge EPA’s overall management of the drinking water program.

Regarding our fourth recommendation that EPA work with the regions and states to assess progress in, and develop a strategy for overcoming barriers to implementing the 2008 action plan and the Director’s 2009 memorandum, the agency expressed partial agreement by saying it will continue to assess the progress of improving data quality. EPA also noted that since these documents were issued, the office has worked with state and regional staff to understand data quality challenges and opportunities for improvement. Specifically, EPA commented that all of the regions have incorporated data quality into their discussions with states and that data quality has been incorporated into grant agreements or state workplans. We agree that those actions would signal progress toward implementing the Director’s memorandum. However, we were unable to verify these statements because EPA told us that documentation was not available.

EPA told us it made the statements on the basis of e-mail communications and discussions between the managers and staff in the Office of Ground Water and Drinking Water and regional management and staff. EPA also stated that the Data Quality Work Group formed by the Director developed a list of recommendations to address underlying data quality problems and that it will continue to evaluate those recommendations. As we note in the report, the recommendations were in draft, and EPA’s comments did not provide evidence that they have been adopted. EPA identified other future actions that it believes will lead to improved data quality. For example, EPA emphasized the effect that a next generation of SDWIS could have on improving data quality. We do not disagree that these actions, if taken, may contribute to improved data quality. However, we point out that EPA has already developed tools that states could use to improve the quality of their data on violations but that those tools have not been widely used. There is no requirement that the states use the next generation of SDWIS, if and when it is available. Furthermore, EPA did not directly address the recommendation to identify the barriers that have prevented more widespread implementation of the action plan and memorandum and develop and publish a strategy for overcoming those barriers.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution for 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Administrator of EPA, and other interested parties. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

To examine the quality of the Safe Drinking Water Information System/Federal (SDWIS/Fed) data that the Environmental Protection Agency (EPA) uses to measure community water systems’ compliance with the health-based and monitoring requirements in the Safe Drinking Water Act (SDWA), we examined the results of audits EPA conducted from 1996 through 2009 in which it assessed—for a sample of states—the completeness and accuracy of violations data those states submitted to SDWIS/Fed. We evaluated the methods that EPA used to conduct those audits to test the methods’ validity and determined that, while limited, these methods were sufficiently reliable for the purposes of our review. We also conducted our own analysis of EPA’s audit data from 2007 through 2009 in order to arrive at estimates of the quality of the data that states reported to SDWIS/Fed. We focused our analysis on community water systems only.

The sample design for the EPA data verification audits consists of a nonprobability sample of primacy agencies within a given year and a probability sample of community water systems within each selected primacy agency. Based on our review of the sample design, we determined that it is not appropriate for our purposes to make quantitative statements or inferences about the entire nation from the selected primacy agencies or comparisons with sampled primacy agency data quality results from the previous years. As such, we only generated estimates to the states audited within a given year. Table 1 provides a description of the number of primacy agencies that were included in the sample for each year.

Classification of Violation Types

We classified violations into one of two types: health-based violations and monitoring violations. By definition, monitoring violations include “other” violations such as Public Notification and Consumer Confidence Report violations. We reviewed and decided to use definitions of violation types provided by EPA to make these classifications. We included lead and copper treatment technology violations as health-based violations in our analysis.

Data Quality Estimates

We defined three separate measures of data quality: accuracy, completeness, and overall quality (a combination of accuracy and completeness). These measures are consistent with the measures used by EPA in previous years. We reviewed and decided to use definitions provided by EPA to calculate these measures.

To estimate the percentage of violations that were accurate and complete, we first created a data set with one observation per violation, and then we used a procedure in statistical software that appropriately accounts for the stratified cluster sample design. We calculated point estimates and 95 percent confidence intervals. We did not report estimates that have margins of error that exceed plus or minus 20 percentage points at the 95 percent confidence level.

To determine whether there was a relationship between monitoring violations and health-based violations, we used audit data to estimate regression models controlling for size, source and administrative control. We tested various specifications of three types of statistical models to ensure that the significance and magnitude of our estimates were consistent across statistical models.

Estimates of Compliance Determination and Data Flow Errors

We conducted a subpopulation analysis to estimate the percentage of incomplete (not reported) violations that were either compliance determination or data flow reporting discrepancies. We calculated point estimates and 95 percent confidence intervals. We did not report estimates that have margins of error that exceed plus or minus 20 percentage points at the 95 percent confidence level.

The second component of our first objective was to examine the quality of SDWIS/Fed data on the status of the states’ and EPA regions’ enforcement actions. Because EPA’s recent audits of state data did not assess the completeness and accuracy of enforcement data in SDWIS/Fed, we examined EPA’s national SDWIS/Fed data from 2005 through 2009 to determine the percentage of violations the states have identified as either resolved (known as returned to compliance), addressed through an enforcement action but not yet resolved, or not addressed. We then interviewed EPA officials to obtain their views on the completeness and accuracy of those data, and also analyzed relevant comments from survey respondents. The survey was of EPA and state drinking water officials to obtain their views on a range of issues related to data quality. (See app. II for more details on our survey methodology).

To identify factors that have affected data quality, we analyzed 41 survey responses representing the views of all 44 members of three joint EPA- state work groups that were created to address various aspects of data management.

To examine the ways in which SDWIS data quality could affect EPA’s management of the Public Water System Supervision (PWSS) program, we examined the importance of data quality for two aspects of EPA’s management of the PWSS program. First, we examined the potential impact data quality could have on EPA’s Enforcement Response Policy. This policy uses a targeting tool that assigns scores to community water systems that are a high priority for enforcement action because of unresolved violations. To demonstrate the effect that underreported health and monitoring violations can have on the Enforcement Targeting Tool, we calculated two scores for each of the approximately 1,200 water systems audited by EPA in 2007, 2008, and 2009. One score was based on violations found in the data verification audits, and the second score was based on violations found in SDWIS/Fed. We then subtracted the two scores for each system to obtain a point difference, a result that we used to illustrate the impact of incomplete data on the scoring process. We used the same methodology EPA uses to create the enforcement score: We assigned point values to unresolved violations (acute health violations are worth 10 points, nonacute health violations and some monitoring violations are worth 5 points, and 1 point for all other monitoring and reporting violations) and then added these points together to produce an overall score for each system. However, the enforcement scores we calculated cannot be considered a water system’s actual score for three reasons: 1. EPA’s targeting tool scores 5 years of violation data, whereas the audits that EPA conducted reviewed state files to identify violations that had occurred in shorter periods of time. Those time periods typically ranged from 1 to 3 years, depending on the drinking water regulation. 2. Due to limited data in EPA’s audit database, our enforcement scores only include underreported violation discrepancies and do not include any discrepancies related to accuracy. However, underreported violations accounted for nearly 97 percent of the discrepancies. 3. EPA’s enforcement score includes an additional penalty that is tied to the year of the oldest unaddressed violation. For instance, if a violation is 5 years old, EPA adds an additional 5 points to the score. Because our analysis considered violations from a shorter period of time, we could not duplicate this additional penalty.

While these three limitations prevent us from duplicating EPA’s exact targeting tool, our analysis presents a conservative estimate of the effect that poor data quality has on the enforcement scoring process. It is likely that additional years of underreported violations, plus other enforcement penalties, would reveal further distortions of the scoring process.

In addition, we examined survey respondents’ views on the impact that data quality may have on implementation of the Enforcement Response Policy. We also interviewed EPA and state officials to obtain their views on the matter.

Second, we examined the impact data quality could have on the agency’s ability to inform the public and Congress about water systems’ compliance with drinking water standards. In particular, we examined whether EPA’s claims about community water systems’ performance relative to Government Performance and Reporting Act (GPRA) goals were affected by the use of incomplete and inaccurate SDWIS/Fed data. Two key GPRA measures are the percentage of community water systems and the percentage of population served by those systems that meet all health- based drinking water standards (i.e., had no health-based violations) in a fiscal year. We used EPA’s data verification audit data to estimate the percentage of community water systems that met all health-based drinking water standards for selected states within each audit year. Based on our review of EPA’s data verification audit sample design, we determined that the sample was not designed to produce reliable estimates of the percentage of the population served by systems with health-based violations. Therefore, we focused our analysis on the percentage of community water systems that met all health-based drinking water standards.

To estimate the percentage of community water systems that met all health-based drinking water standards from the data verification audit data, we counted the number of health-based violations for each community water system in the sample and calculated point estimates and 95 percent confidence intervals of the percentage of systems that met all health-based drinking water standards after accounting for the results of the data validation audits.

To examine the actions EPA and the states have been taking to improve the quality of data in SDWIS/Fed, we interviewed EPA officials and obtained and reviewed documentation on steps the agency has taken, or is considering, to modernize SDWIS and improve data quality. We also examined survey respondents’ views on steps that EPA and the states could take to address data quality—including the adoption of particular data management tools—and ways in which the three EPA-state work groups could be more effective. For more information on our survey and content analysis, see appendix II.

We conducted this performance audit from February 2010 through June 2011 in accordance with generally accepted auditing standards. Those standards require that we plan and perform the audit to obtain sufficient and appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Survey Methodology and Analysis

Selection of Survey Respondents

To obtain the views of knowledgeable EPA and state drinking water officials about SDWIS data management and data quality, we surveyed the members of three EPA-state work groups that were formed to address various aspects of drinking water data: the Data Management Steering Committee, the Data Technical Advisory Committee, and the Data Quality Work Group. EPA and the Association of State Drinking Water Administrators provided us with the names and e-mail addresses of the 46 members of these groups. The work group members come from EPA headquarters, EPA regions, states, and the Association of State Drinking Water Administrators. The Data Management Steering Committee had 18 members, the Data Technical Advisory Committee had 15 members, and the Data Quality Work Group had 25 members. Nine officials served on more than one of the committees.

Survey Design and Pretesting

Our survey asked a range of questions related to the drinking water violations data that states and other primacy agencies provide to EPA. We asked the respondents to comment on the factors that have contributed to data errors, the steps that should be taken to correct those errors, the impact that data errors could have on EPA’s Enforcement Response Policy, and various data management tools. We also asked the respondents to evaluate the effectiveness of the work group or groups on which they served.

The practical difficulties of conducting any survey may introduce errors, commonly referred to as nonsampling errors. For example, respondents may have difficulty in interpreting a particular question or may lack information necessary to provide valid and reliable responses. In order to minimize these errors, we conducted pretests of the draft survey with one EPA headquarters official, one EPA regional official, and one state official by telephone. The Chief of the Infrastructure Branch of EPA’s Office of Ground Water and Drinking Water suggested that we conduct the pretest with those individuals. During these pretests, we checked whether (1) questions were clear and unambiguous, (2) terminology was used correctly, (3) the questionnaire did not place undue burden on respondents, (4) the information could feasibly be obtained, and (5) the survey was comprehensive and unbiased. In addition, the survey was peer reviewed by a GAO senior survey methodologist. We made changes to the content and the format of the survey based on the feedback we received.

Survey Administration We administered our survey in August and September of 2010. We first phoned each work group member to alert them to our plan to send the survey and to request their participation. Through those phone calls, we learned that one of the work group members had retired and another had transferred to a different position and was no longer a member of a work group. As a result, our survey population decreased to 44.

Prior to fielding the survey, we sent an e-mail to each member of the work groups to further explain its purpose. We notified work group members electronically when the survey was available, and sent e-mail reminders prior to our requested deadline of September 13, 2010. We also made phone calls to several survey recipients during the extension period to request their participation. In total we received 41 completed surveys. However, the three work group members from the Association of State Drinking Water Administrators collaborated to prepare one response and two EPA regional officials collaborated to prepare one response. Therefore, the 41 completed surveys represent the views of all 44 members of the survey population.

Data Analysis

The survey contained closed-ended questions that asked respondents to select from a finite number of options. For example, some questions asked respondents to select “Yes,” “No,” or “Don’t Know.” Others asked respondents to select from a list of factors that may have contributed to drinking water violation data errors. Our analysis of the responses to these questions simply involved counting the number of responses for each option. In the report, there are instances in which we identify all of the responses and other instances in which we identify the most common response.

We also asked respondents to evaluate the effectiveness of certain data management tools or the EPA-state work groups. For these questions, we offered the respondents a four-point range of answers: very effective, moderately effective, slightly effective, and not effective. Respondents could also answer “Don’t Know” to these questions. In the report, we sometimes identified the most common response while in other instances we combined the number of “moderately effective” and “slightly effective” responses because each was relatively common.

Several survey questions asked for opinions on the creation of a goal for the percentage of monitoring violations that are completely and accurately reported to SDWIS/Fed. We asked those respondents who supported the idea of having such a goal to state what they think the percentage should be. We summed those percentages and divided by the number of respondents who answered that question to arrive at an average.

The responses to the closed-ended questions are provided in this appendix.

Content Analysis

The survey also contained open-ended questions that asked respondents to provide a narrative response. In order to succinctly summarize the open-ended responses, we performed a content analysis in which we grouped the responses into a coding structure that represented common themes. We decided that the responses to each open-ended question would have a coding structure with two dimensions. To explain this, it is useful to discuss the link between closed-ended and open-ended questions. For example, one closed-ended question asked the respondents to select a factor—such as training by EPA or the states or guidance from EPA—that they believe has contributed to compliance determination errors. The subsequent open-ended question asked them to elaborate on why they thought the factor or factors they selected have contributed to compliance determination errors. Note that the respondents did not necessarily elaborate on each of the factors they selected in the prior question. As part of our content analysis, we sought to first identify the first dimension code for the response. In this example, the first dimension codes mirrored the factors (e.g., training or guidance) that the respondent selected to write about. A second dimension code provided a more detailed description of what the respondent said about the first order code. For example, second dimension codes for that question included amount or quality, timing, and targeting.

A team of three GAO analysts jointly reviewed several completed surveys to develop an initial draft of a structure for coding the open-ended responses. To further identify meaningful first and second dimension codes for the coding structure, the three GAO analysts independently reviewed the open-ended responses for four completed surveys. Each analyst made a judgment about appropriate codes that described the themes in the open-ended responses. The analysts compared their decisions and reconciled any disagreements regarding appropriate codes by refining the criteria used to categorize the responses.

After the team agreed upon the coding structure, it continued its analysis of the responses to the closed-ended questions. The three GAO analysts were each assigned to independently review the responses to specific sets of questions. For example, Analyst A and Analyst B independently reviewed and coded the open-ended answers to questions 1 through 4. Analysts A and B then compared their coding decisions and reconciled any disagreements. If they could not reconcile a disagreement, Analyst C was consulted to achieve agreement. The three analysts rotated assignments so that each performed the role of “tiebreaker” when the other two could not agree on a coding decision.

Respondents’ Answers to Closed-Ended Survey Questions

Our survey of EPA and state drinking water officials contained numerous closed-ended questions about various data management issues. Table 2 presents those questions and the respondents’ answers.

Coding Decisions for Responses to Open-Ended Survey Questions

Many of the questions in our survey of EPA and state drinking water officials asked for open-ended responses. We developed several “coding schemes” to categorize the responses to those questions. Our schemes generally had first and second dimension codes. For example, question 2 asked the respondents to explain how factors they selected contributed to compliance determination errors. The first dimension codes for that question included information system structure, training, funding, staffing, and guidance, among others. The second dimension codes included amount; targeting; timeliness; quality, complexity, or ease of use; automation; and others. A response to question 2 might have included a comment that related to information system structures and, more specifically, a comment about the quality, complexity, or ease of use of information systems. In that situation, we would have coded the response as falling into those first and second dimension codes. Because some sets of questions generated answers that could be similarly coded, we used some coding schemes for multiple questions. On the other hand, we used some schemes for only one question.

Appendix III: Comments from the Environmental Protection Agency

The following are GAO’s comments responding to the comments in Appendix A of the Environmental Protection Agency’s letter dated June 8, 2011.

GAO Comments

1. EPA commented that there are over 152,000 public water systems in the United States that are also subject to the requirements of the National Primary Drinking Water Regulations. We do not disagree, but did not modify the report in response to EPA’s comment. Footnote 11 of the report describes the universe of public water systems. 2. EPA said that violations that are reported as “other” should not be included with monitoring and reporting violations and that it would be beneficial to explain how reporting violations differ from monitoring violations. Our analysis is consistent with EPA’s 2008 analysis of data quality which also combined “other” violations, such as violations of consumer confidence reporting and public notification requirements— with monitoring and reporting violations. Therefore, we did not modify our analysis or the report in response to EPA’s comment. We also did not modify the background section to further explain the difference between monitoring and reporting violations because we provide examples of different violations in a subsequent section of the report. 3. EPA requested that we provide a footnote in the report describing new performance indicators for small water systems. Specifically, EPA requested that we explain the agency’s intent behind the indicators and its plan to evaluate their utility. We did not modify the report in response to this comment since we describe two of these indicators in the section of the report that addresses EPA’s ability to monitor and report progress toward its strategic objective of reducing exposure to contaminants in drinking water. 4. EPA said it was unsure of the basis for our statement that monitoring violations are predictors of health-based violations. EPA also noted some of the variations between monitoring and reporting violations and asked for more details regarding our analysis. Our statement was based on aggregate regression analyses (negative binomial and zero- inflated Poisson models) with limited controls. It does not take into account which type of monitoring and reporting violation occurred, and cannot differentiate between lack of monitoring and monitoring that was not reported or was delivered late. The regression was intended to illustrate the link between overall counts of monitoring and reporting violations and counts of health-based violations, and does not provide insight into the nature of the link or the reasons that monitoring and reporting violations might condition the number of health-based violations. We realize the implications of our statement are limited, but believe the correlation between overall counts of monitoring violations and health-based violations offers useful insight. 5. EPA requested that we present an analysis of the quality of violations data for the Lead and Copper Rule separately from other National Primary Drinking Water Regulations, as it has done. We acknowledge that there may be value in conducting data quality analyses for specific drinking water regulations, as EPA did, for example, in its 2008 report on data quality. That report showed that the data quality for the Lead and Copper Rule was lower than for other types of drinking water regulation. However, because the data verification audit data we analyzed was from a sample of community water systems from a sample of states, our results included margins of error. Analyzing data quality for particular drinking water regulations results in larger margins of error than analyzing date quality for all health-based violations. In light of that circumstance, we decided to conduct our analysis of all health-based violations. We did not modify our analysis or report in response to this comment. 6. EPA commented that it intends to conduct six to eight data verification audits in calendar year 2011. We have modified the report to reflect that comment, but note that EPA’s statement concerns audits it has yet to conduct. EPA also said that it would appreciate our including any specific suggestions made by survey respondents on how the data verification audits can be improved. We have added a footnote with examples of comments from survey respondents. 7. EPA requested that we update the report to reflect the status of work done by the Office of Ground Water and Drinking Water to assess regional responses and document the extent to which the regions had complied with requests to discuss data quality with states. EPA went on to say it understands that all the regions include data quality as an issue for discussions with their states and as part of their grant agreements or state work plans. We have modified the report to include EPA’s statements. However, we were not able to verify the accuracy of those statements because EPA did not have supporting documentation. 8. EPA said that it agrees that the regions did not fully support new measures for tracking regional oversight suggested by headquarters but that the lack of those measures has not prevented the regions or headquarters from continuing to work with the states to address data quality challenges. We note that the proposed performance measures would assist the Office of Ground Water and Drinking Water to monitor the regions’ oversight of the states, not the states’ performance. We assume that EPA headquarters proposed these performance measures because it thought they would help encourage the regions to increase their oversight. Without them, EPA headquarters may find it more difficult to oversee the regions. We did not modify the report in response to this comment.

Appendix IV: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the individual named above, Diane B. Raynes, Assistant Director; James Ashley; Elizabeth Beardsley; Mark Braza; Ross Campbell; Anna Maria Ortiz; Carla D. Rojas Paz; Kelly Rubin; Jerome Sandau; Jeffrey Sanders; Carol Herrnstadt Shulman; and Vasiliki Theodoropoulos made significant contributions to this report. Robert Alarapon, Mick Ray, and Lisa Vojta also made important contributions to this report.