Drinking Water: Unreliable State Data Limit EPA's Ability to Target Enforcement Priorities and Communicate Water Systems' Performance
Published: 2011-06-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The nation's drinking water is among the safest in the world, but contamination has occurred, causing illnesses and even deaths. Under the Safe Drinking Water Act (SDWA), the Environmental Protection Agency (EPA) has authorized most states, territories, and tribes to take primary responsibility for ensuring that community water systems provide safe water. EPA needs complete and accurate data on systems' compliance with SDWA to conduct oversight. GAO was asked to assess the (1) quality of the state data EPA uses to measure compliance with health and monitoring requirements of the act and the status of enforcement efforts, (2) ways in which data quality could affect EPA's management of the drinking water program, and (3) actions EPA and the states have been taking to improve data quality. GAO analyzed EPA audits of state data done in 2007, 2008, and 2009, and surveyed EPA and state officials to obtain their views on factors that have affected data quality and steps that could improve it.
section_title: What GAO Found
paragraphs: The data states reported to EPA for measuring compliance with health and monitoring requirements of SDWA did not reliably reflect the number of health-based and monitoring violations that community water systems have committed or the status of enforcement actions. Using data from the 14 states EPA audited in 2009, GAO estimates that those 14 states did not report or inaccurately reported 26 percent of the health-based violations that should have been reported and 84 percent of the monitoring violations that should have been reported. GAO's findings were consistent with the results of prior EPA audits. In addition, according to EPA headquarters and regional officials GAO interviewed and surveyed, state-reported data underreported the percentage of water systems with violations against which the states have taken enforcement actions. Survey respondents and other officials reported that numerous factors contribute to errors in reported data on violations and enforcement, including inadequate training, staffing, and guidance, and inadequate funding to conduct those activities. Unreported health-based and monitoring violations and incomplete enforcement data limit EPA's ability to identify water systems with the most serious compliance problems and ensure that it is achieving its goal of targeting for enforcement those systems with the most serious compliance problems. Specifically, incomplete and inaccurate data on both violations and enforcement actions affect a scoring tool EPA and the states are using to rank systems for enforcement actions. In addition, unreliable data quality impedes EPA's ability to monitor and report progress toward a strategic objective of reducing exposure to contaminants in drinking water. For example, EPA's 2011 national program guidance contains a performance measure for the number and percentage of systems with certain repeated health-based violations, but EPA's ability to reliably use this type of measure requires complete and accurate data on violations. Because of unreported violations data, EPA may not be able to report accurate performance information on systems with these violations. EPA and the states have collaborated over many years to identify and address the causes of incomplete and inaccurate violations data, but those efforts have not been fully successful, according to EPA and state officials GAO surveyed. EPA's efforts have included (1) conducting audits--discontinued in 2010 because of funding constraints--to determine the completeness and accuracy of the violations data states reported to EPA, (2) establishing three work groups to address data management and quality, and (3) urging EPA regions and states to use data management tools the agency has developed. However, EPA has encouraged but not required that its regions or the states take specific actions that could improve data quality. EPA's 2010 drinking water strategy calls for, among other things, an increase in shared data between the agency and the states. EPA also plans to redesign its drinking water data system to provide it with greater access to, and oversight of, the states' determinations of SDWA violations.
section_title: What GAO Recommends
paragraphs: GAO is making recommendations to improve EPA's ability to oversee the states' implementation of SDWA and provide Congress and the public with more complete and accurate data on compliance and enforcement. EPA partially agreed with two of the recommendations, disagreed with one, and neither agreed nor disagreed with one. GAO believes that EPA needs to implement all of the recommendations to improve its ability to oversee SDWA.

section_title: Letter
paragraphs: Americans rely on more than 51,000 community water systems for safe  drinking water. Even though this drinking water supply is generally  considered among the safest in the world, 11 states had 20 outbreaks of  illness associated with drinking water in 2005 and 2006 that resulted in 612  illnesses and 4 deaths, according to data published by the Centers for  Disease Control and Prevention (CDC) in 2008. In part to safeguard  against such outbreaks, the Environmental Protection Agency (EPA),  which is generally responsible for the regulation of the nation’s drinking  water, requires public water systems to comply with regulations it  established under the Safe Drinking Water Act (SDWA). Among other  things, these regulations establish (1) health-based requirements, including  limitations, and treatment techniques for controlling contaminants that  could harm human health and (2) monitoring requirements to determine  whether drinking water meets the health-based requirements.
EPA authorizes and assists state, territorial, and tribal regulatory  agencies—referred to as states in this report—to administer SDWA  through EPA’s Public Water System Supervision (PWSS) program. States  that have accepted “primacy” responsibility for the PWSS program collect  and review data from community water systems to determine their  compliance with SDWA; all states except Wyoming and the District of  Columbia have received primacy. With the exception of the Navajo  Nation, EPA maintains primacy for community water systems in Indian  Country. Primacy states are responsible for, among other things,  determining when systems have violated SDWA, taking timely and  appropriate enforcement action, and reporting those actions to EPA.  EPA’s regions and headquarters oversee the states to ensure they meet  their primacy responsibilities; the EPA regions also act as the primacy  agency in nonprimacy states, where a state has not yet received primacy  for a particular drinking water regulation, and on tribal lands where the  tribe has not assumed primacy. To determine water systems’ compliance  with federal standards for safe drinking water, EPA must have access to  reliable data on the inventory of community water systems, which, along  with other public water systems are subject to these standards; the quality  of drinking water; and violations of SDWA’s requirements including those  to monitor drinking water to ensure the water meets health standards.  EPA also needs reliable data regarding the status of enforcement actions  to inform its oversight role. These data play a critical role in helping EPA  manage the PWSS program by identifying, for example, systems’ return to  compliance after committing violations of the safe drinking water  standards for microbiological and chemical contaminants.
The states collect and manage relevant data (including violations and  enforcement information) in either a database provided by EPA—known  as the Safe Drinking Water Information System/State (SDWIS/State)—or in  a data system of their own design. The states also periodically transfer  from their database information on violations and enforcement actions to  the EPA headquarters version of SDWIS known as SDWIS/Fed. EPA  generally uses the data in SDWIS/Fed—along with other documentation  provided on request—to review state determinations of when water  systems are complying with the act. EPA also uses these data to determine  whether water systems, in the aggregate, are achieving the agency’s  national targets for compliance. Additionally, EPA can use enforcement  data to determine whether the states or EPA regions have taken actions  consistent with EPA’s Drinking Water Enforcement Response Policy. The  policy calls for states or EPA regions to take enforcement actions that are  timely and appropriate for returning the water system to compliance with  safe drinking water standards. The quality of drinking water data in  SDWIS/Fed was called into question in the late 1990s and was the subject  of a 2004 report by EPA’s Office of Inspector General.
In this context, you asked us to review the SDWIS/Fed data. Our objectives  were to examine the (1) quality of the SDWIS/Fed data that EPA uses to  measure community water systems’ compliance with the health-based and  monitoring requirements in SDWA and the status of the states’ and EPA  regions’ enforcement actions, (2) ways in which SDWIS data quality could  affect EPA’s management of the PWSS program, and (3) actions EPA and  the states have been taking to improve the quality of data in SDWIS/Fed.
To address the first objective, we examined the results of audits EPA  conducted from 1996 through 2009 to assess the completeness and accuracy  of the data that states submitted to SDWIS/Fed (data verification audits).  EPA’s most recent published analysis of its audits was released in 2008 and  covered audits done in 2002 through 2004. We evaluated the methods that  EPA used to conduct those audits to test the methods’ validity and  determined that these methods produced audit data that were sufficiently  reliable for the purposes of our review. EPA also conducted audits in 2005  through 2009, but it had not published its analysis of those audits at the time  of our review. We therefore obtained the results of the 2007, 2008, and 2009  audits from EPA and conducted our own analysis of data quality using the  methods that the agency used in its 2008 report. To identify factors that  affected the quality of the data, we surveyed all 44 members of three joint  EPA-state work groups that were created to address various aspects of data  management; we received the views of all of the members. We examined  EPA’s national SDWIS/Fed data from 2005 through 2009 to determine the  percentage of violations that the states identified as returned to compliance,  addressed through an enforcement action, or not addressed. Because EPA’s  recent audits of state data did not assess the completeness and accuracy of  these data, we interviewed EPA and state officials to obtain their views on  the completeness and accuracy of those data and analyzed relevant  comments from our survey respondents.
To address the second objective, we examined the potential impact data  quality could have on EPA’s Drinking Water Enforcement Response  Policy, which uses a scoring system that identifies community water  systems that are a high priority for enforcement action because of  unresolved violations. We examined the impact that using data from the  data verification audits could have on the scoring system compared with  using data from SDWIS/Fed. We also examined the views of the survey  respondents on the impact that data quality may have on implementation  of the Enforcement Response Policy. Further, we examined the impact  data quality could have on the agency’s ability to inform the public and  Congress about water systems’ compliance with drinking water standards  relative to strategic targets it has set under the Government Performance  and Results Act. To address the third objective, we examined the survey  respondents’ views on steps that EPA and the states could take to address  data reliability—including the adoption of particular data management  tools—and ways in which the three EPA-state work groups could be more  effective. We also examined information EPA provided on recent actions it  has taken to improve data quality, including its current proposal for  modifying the SDWIS data management system. We did not evaluate the  merits of that proposal. A more detailed description of our scope and  methodology can be found in appendixes I and II.
We conducted this performance audit from February 2010 through June  2011 in accordance with generally accepted auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient  and appropriate evidence to provide a reasonable basis for our findings  and conclusions based on our audit objectives. We believe that the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: This section provides information on the risks posed by unsafe drinking  water, the authority EPA gives to states under SDWA, differences among  community water systems, EPA and the states’ processes for entering  water systems’ data into SDWIS/Fed, EPA’s Enforcement Response Policy  and Enforcement Targeting Tool, and EPA’s strategic targets for  compliance with SDWA.
subsections: section_title: Risks of Unsafe Drinking Water
paragraphs: While the nation’s public drinking water supplies are much less prone to  outbreaks of waterborne diseases such as cholera and typhoid than they  were in the 19th and early 20th centuries, waterborne-disease outbreaks  caused by microorganisms do still occur. For example, according to a  2006 study, an estimated 4.3 million to 11.7 million annual cases of acute  gastrointestinal illnesses in the United States are attributable to drinking  water from community drinking water systems supplied by surface-water  and ground-water sources.
Other contaminants found in drinking water may also pose a threat to  human health from long-term exposure at certain levels. For example:    Long-term exposure to disinfectants—such as chlorine—that are added  to water to control microorganisms and the byproducts of disinfectants  may cause anemia, stomach discomfort, and eye or nose irritations. In  small children and infants, inappropriate exposure to disinfectants  could lead to nervous system problems. In addition, long-term  ingestion of water with disinfection byproducts may increase the risk  of cancer and may affect the nervous system, liver, and kidneys.
Arsenic, which occurs naturally and in industrial waste, may cause skin  damage and circulatory system problems and increase the risk of  cancer if it is not treated.
Lead and copper introduced into drinking water from the corrosion of  household plumbing systems or the erosion of natural deposits may  cause liver or kidney damage. Long-term exposure to lead may delay  physical or mental development in infants and children.
Nitrate, which comes from fertilizer runoff, septic tanks, and erosion of  natural deposits, is especially harmful to infants below the age of 6  months, and exposure may cause shortness of breathe, a serious illness  known as blue-baby syndrome, and, if left untreated, death.
section_title: EPA Provides Authority to State Primacy Agencies
paragraphs: Under SDWA, EPA may authorize states meeting specified requirements to  implement the PWSS program—referred to as primacy authority. For  example, states must have regulations for contaminants that are no less  stringent than those promulgated by EPA, adequate record keeping and  reporting requirements, and adequate enforcement authority to compel  water systems to comply with drinking water requirements. EPA has  approved primacy authority for 49 states, 5 territories, and the Navajo  Nation. EPA’s regions administer the programs in Wyoming and the  District of Columbia, and for most tribes. EPA provides annual grants  through the PWSS program to the states using a formula that takes into  account population, geographical area, and the number of water systems  covered. (EPA may also consider other relevant factors in its allocation  formula.) In recent years, total EPA allocations to these grants have  averaged about $100 million per year. States must provide matching funds;  under the act, the PWSS grant can provide no more than 75 percent of the  costs expended by a state to carry out its PWSS program. EPA’s drinking  water program guidance instructs EPA regions to work with the states to  develop grant workplans that include the states’ commitments to report  key activities. For example, the State of Washington’s workplan includes a  commitment to assure complete and accurate identification and reporting  of public water system compliance.
section_title: Differences among Community Water Systems
paragraphs: As of July 2009, more than 51,000 community water systems supplied  water to the same populations year-round. Community water systems  vary widely in the number of people they serve, from 25 to over a million.  As figure 1 shows, small systems are the most common. However, the 8  percent of community water systems that serve more than 10,000 people  supply approximately 82 percent of all community water system users.  Figure 1 shows the number of community water systems in 2009  categorized by the number of people they serve.
Community water systems obtain their water from groundwater reserves  or from surface water sources. They may obtain, treat, and distribute their  water entirely on their own, or they may purchase treated water from  another system. Treatment generally consists of filtration, sedimentation,  and other processes to remove impurities and harmful agents, and  disinfection processes such as chlorination to eliminate biological  contaminants.
Community water systems must meet a variety of health-based  requirements under SDWA. These include providing drinking water that  meets numerical limits for some contaminants, using treatment  techniques for other contaminants, and using laboratory testing to monitor  and report on the quality of the drinking water that they provide. Under  SDWA, EPA may establish an enforceable standard—called a maximum  contaminant level, or MCL—that limits the amount of a contaminant that  may be present in drinking water. If EPA determines it is not economically  or technically feasible to ascertain the level of a contaminant, the agency  may instead establish a treatment technique to prevent known or  anticipated health effects. In total, EPA has set MCL or treatment  technique standards—known as the National Primary Drinking Water  Regulations—for 89 regulated contaminants. We refer to violations of  these standards as health-based violations.
EPA has also established monitoring, reporting, and other requirements  for each of the 89 regulated contaminants. In this report we refer to these  requirements collectively as monitoring requirements and refer to  violations of these requirements as monitoring violations. These  requirements may vary depending on several factors. For example, the  frequency of monitoring may depend on whether the system obtains its  water from ground water or surface water sources or upon the size of the  water system. Additionally, if the water system detects certain  contaminants above a specified amount, it may need to increase the  frequency of its monitoring. Community water systems must also notify  the public within specified times about the occurrence of health-based or  monitoring violations and provide their customers with an annual  Consumer Confidence Report containing data on the presence and  concentrations of the 89 regulated contaminants.
section_title: The States’ Processes for Entering Water Systems’ Data into SDWIS
paragraphs: Most of the states enter data they collect and generate on community  water systems into a version of SDWIS designed for use by the states  known as SDWIS/State. As EPA promulgates new or revised regulations  for particular contaminants, it develops new SDWIS versions to capture  data associated with those regulations. EPA encourages the states to place  their water systems’ data into SDWIS/State but the states may choose not  to if they have an alternative database that meets their needs while also  complying with EPA recordkeeping requirements. The data include  inventory information about each system, such as its name, owner,  address, and the size of the population it serves. The data also include the  results of the water monitoring conducted according to contaminant- specific schedules by each system, the state’s determination of whether  the system has committed violations, and a record of enforcement actions  taken.
According to EPA, every 3 months, the states must transfer certain  information from either SDWIS/State or their alternative data system to  SDWIS/Fed. Specifically, the states transfer to SDWIS/Fed data on the  health and monitoring violations identified and enforcement actions taken,  and whether the state has determined that the system has returned to  compliance. SDWIS/Fed is the data system EPA uses to gauge community  water systems’ compliance with SDWA. In 2006, EPA and the Association  of State Drinking Water Administrators set a goal that 90 percent of health- based drinking water violations be completely and accurately reported to  SDWIS/Fed. EPA, however, does not have a goal for the completeness and  accuracy of data on monitoring violations.
section_title: EPA’s Drinking Water Enforcement Response Policy and Enforcement Targeting Tool
paragraphs: Under its 2009 Drinking Water Enforcement Response Policy, EPA is to  identify water systems with the most serious compliance problems and  direct enforcement resources to these systems. An important component  of EPA’s enforcement policy is its Enforcement Targeting Tool for  identifying water systems with the most serious compliance problems. The  Enforcement Targeting Tool assigns a score to each water system based  on an accounting of unresolved violations over a 5-year period. Because  some violations may have more serious health consequences than others,  the tool assigns each violation a “weight” or number of points based on the  potential threat to public health: acute health violations are worth 10  points, other health violations and some major monitoring violations are  worth 5 points, and all other monitoring violations are worth 1 point.  Additional points are added for each year a violation remains unresolved.  Points for each violation at a water system are summed to generate the  system’s score. Water systems whose scores meet or exceed a certain  threshold—EPA has set the threshold at 11 points—are considered to have  serious compliance problems and are placed on a priority list of water  systems that the states and EPA are to target for enforcement. Using this  approach, EPA and the states target resources to address those water  systems that EPA determines have the most significant problems  complying with SDWA’s requirements.
EPA’s Enforcement Response Policy also provides guidance on the  amount of time in which states and EPA regions should address violations  at priority water systems. Once systems have been targeted as a priority,  states and the regions have 6 months to work with them in whatever  manner they deem appropriate to resolve violations and return the system  to compliance. Enforcement and compliance assistance may include a  range of actions such as providing violation notification letters to systems,  offering them technical assistance, conducting site visits to resolve  violations, entering into compliance agreements such as consent orders,  and additional formal actions such as issuing administrative orders,  assessing fines or penalties, and filing or referring judicial cases. In  situations where the system is unlikely to return to compliance within the  6-month time frame, EPA’s policy calls for states or the regions to take a  formal enforcement action within 6 months that will put the system “on  the path” to compliance by laying out future actions and time frames the  system needs to follow. According to EPA, states and the regions are  required to enter information on enforcement actions, including violation  resolution, into SDWIS/State or the equivalent system as they occur and  send those data to SDWIS/Fed every 3 months.
section_title: EPA’s Strategic Targets for Compliance with SDWA
paragraphs: EPA’s two most recent strategic plans issued in 2006 and 2010 have  included the strategic objective of protecting human health by reducing  exposure to contaminants in drinking water. These strategic plans are  required by the Government Performance and Results Act (GPRA), which  calls for related annual performance plans to outline the process for  communicating goals and strategies throughout the agency, and for  assigning accountability to managers and staff for goal achievement. As  we have previously reported, a clear relationship should exist between an  agency’s long-term strategic goals and the performance goals in the annual  performance plan. Successful organizations try to link performance  measures to the organization’s strategic goals and, to the extent possible,  have performance measures that will show annual progress toward  achieving their long-term strategic goals. GPRA also requires that the  agency publish an annual performance report communicating to  managers, policymakers, and the public what was actually accomplished  for the resources expended.
To help gauge its progress relative to its objective of reducing exposure to  contaminants, EPA uses annual performance measures and strategic  targets to track national rates of drinking water compliance, including the:    percentage of community water systems that meet all applicable  health-based standards—the strategic target for 2009 was 90 percent;    percentage of population served by community water systems that will  receive drinking water that meet all applicable health-based drinking  water standards—the strategic target for 2009 was 90 percent; and    percentage of person months during which community water systems  provide drinking water that meets all applicable health-based  standards—the strategic target for 2009 was 95 percent.
EPA uses the data on violations that the states report to SDWIS/Fed to  gauge the performance of community water systems in relation to these  GPRA strategic targets. According to EPA, the SDWIS/Fed data indicated  that community water systems either met, or came close to meeting, these  strategic targets in 2007 through 2009. As part of its effort to achieve the  objective in its 2010 strategic plan to reduce exposure to contaminants in  drinking water, EPA has also adopted performance indicators that it will  use to track the number and percentage of small water systems with  repeat health-based violations and the average time for those systems to  return to compliance.
section_title: SDWIS Data from States Did Not Reliably Reflect Community Water Systems’ Violations of SDWA or the Status of Enforcement Actions
paragraphs: The data that states provided to EPA did not reliably reflect the frequency  of community water systems’ violations of SDWA’s health-based  standards, according to our analysis of EPA’s audit data for 2007 and 2009  and past EPA audit reports. In addition, the data did not reliably reflect the  frequency of monitoring violations, which are a predictor of health-based  violations. Survey respondents support the concept of EPA setting a  numerical goal for the percentage of monitoring violations accurately  reported in order to increase the reliability of data in SDWIS/Fed.  Furthermore, data provided by the states on the status of enforcement  actions taken against systems with violations were incomplete, according  to EPA and state officials we interviewed. Officials identified several  factors, such as inadequate training, staffing, and guidance, as contributing  to errors in data on violations and enforcement.
subsections: section_title: States’ Data Did Not Reliably Report the Frequency of Health-based Violations
paragraphs: Using EPA’s 2007 and 2009 audits of the data that the states provided to  SDWIS/Fed, we found that the states did not completely and accurately  report health-based violations committed by community water systems.  For example, we estimate that the 19 states EPA audited in 2007 did not  report or reported inaccurately 20 percent, or 543, of the health-based  violations that EPA determined should have been reported. For 2009,  we estimate that the 14 states EPA audited in that year did not report or  reported inaccurately 26 percent, or 778, of the health-based violations  that EPA determined should have been reported. Figure 2 shows our  estimates of the percentage of health-based violations the states did not  report or inaccurately reported.
EPA’s audits from 1996 through 2004 also found that violations had been  unreported. For example, on the basis of its 2002 through 2004 audits,  EPA reported that the 37 states it audited did not report or inaccurately  reported about 49 percent of health-based violations committed by  community water systems to SDWIS/Fed, as shown in figure 3. It is not  possible to infer a trend between 2002 and 2009 because EPA’s 2002-2004  results are not directly comparable to the results of our analysis of 2007  and 2009 audit data. That is because, among other things, EPA’s analysis  combined 3 years of audits and because the audits were of different states  than were audited in 2007 and 2009. In its analysis of the completeness and  accuracy of state data for all types of public water systems, EPA found  that the reliability of state data on violations varied for different health- based standards. For example, data on violations of the total coliform rule  and surface water treatment rules were more reliable than data on  violations of the lead and copper treatment technique standards.
EPA’s audits have shown two types of errors in the data the states  submitted. The first type, known as a compliance determination error,  occurs when a violation occurs but the state (or EPA region acting as a  primacy agency) does not issue a violation notice to the water system and  does not report that violation to SDWIS/Fed. The second type, known as a  data flow error, occurs when the state or region issues a violation notice  to the water system and is reported to the state data system but  information about the violation is not correctly transferred to SDWIS/Fed.  Compliance determination errors, according to our analysis of EPA’s data,  are much more common than data flow errors. For example, using EPA’s  audit data from 2009, we estimate that 91 percent of the errors in health- based violations between the audited data and SDWIS/Fed were  compliance determination errors and 9 percent were data flow errors.  Among these errors were some state-reported violations to SDWIS/Fed  that EPA determined had not occurred (e.g., false positives).
section_title: States’ Data Did Not Reliably Reflect the Frequency of Monitoring Violations, Which Are a Predictor of Health-based Violations
paragraphs: According to our analysis of EPA’s audit data from 2007 through 2009, the  states did not report or inaccurately reported the number of monitoring  violations. For example, we estimate that the 14 states audited in 2009 did  not report or inaccurately reported about 54,600—or 84 percent—of the  monitoring violations committed by community water systems to  SDWIS/Fed. On the basis of these audit results, we conclude that the total  nationwide number of actual monitoring violations had to have been  considerably higher than the 82,000 reported in SDWIS/Fed. Monitoring  violations, as we have defined them in this report, include a variety of  situations, ranging from instances in which a water system did not do  required monitoring, did not report the results to the state on time, or did  not issue public notice of a health-based violation in a timely fashion. It is  important to note that the underreporting of monitoring violations may  affect what is known about health-based violations. Some unknown  percentage of both reported and unreported monitoring violations may  have hidden the presence of a health violation, particularly when the  violation was that required monitoring was not done at all.
Our analysis found that having a monitoring violation was a strong and  statistically significant predictor of whether a system had a health-based  violation, among systems sampled in EPA’s audit data for 2007 to 2009.  Furthermore, the number of monitoring violations was positively and  statistically significantly related to the rate of health-based violations. In  its 2010 report on 2007 and 2008 national drinking water compliance rates,  EPA noted that monitoring violations were a concern because “if a water  system does not monitor and report on the quality of its water it is  impossible to know if there are health-based violations.” Therefore, the  presence of monitoring violations may “mask” the presence of health- based violations. The total number of “masked” health-based violations is  unknown, but may be affected by the total number of monitoring  violations. As we have shown, the total number of monitoring violations is  much higher than indicated by the SDWIS/Fed data, suggesting that the  total number of health-based violations is also larger than indicated.
section_title: Majority of Survey Respondents Support a Goal for the Quality of Data on Monitoring Violations
paragraphs: Regarding the low quality of SDWIS/Fed data on monitoring violations, a  majority of survey respondents who expressed an opinion (20 of 34)  indicated they thought EPA should—as it has for health-based violations— establish a numerical data quality goal for the percentage of monitoring  violations that are completely and accurately reported. These respondents  had a range of views on what a numerical goal should be, from a low of 41  percent to a high of 100 percent; the average was about 83 percent. When  asked to describe the actions they thought EPA and the states need to take  to achieve their preferred goal, the most common responses focused on  increasing management prioritization, training, and information system  technology. When respondents who indicated they thought EPA should  not establish a data quality goal were asked to explain their answers, the  most common response was that the quality of data on monitoring  violations was not a high priority. For example, one respondent said that  unless the states and EPA can address inadequate staffing, monitoring  violations will continue to be the lowest priority.
section_title: State Data on Enforcement Actions Were Incomplete, According to State and EPA Officials
paragraphs: EPA’s data verification audits in 2005 through 2009 did not include any  analysis of the accuracy of the data the states reported to SDWIS/Fed on  their enforcement actions. According to EPA’s audits for 2002 through  2004, the audited states did not accurately report to EPA 27 percent of the  enforcement actions they took against community water systems.  However, EPA arrived at this estimate by comparing data in SDWIS/Fed  with the data in SDWIS/State to determine whether they matched. EPA did  not examine original source documents that could have shown whether  the data in SDWIS/State accurately represented the status of the states’  enforcement actions. The approach EPA used in its audits to estimate the  accuracy of the reporting of enforcement actions by states differed from  the approach it took in its audits of violations data, in which EPA  examined the sampling data that community water systems provided to  the states. Consequently, EPA’s estimates of the completeness and  accuracy of enforcement data were less likely to be as reliable as its audits  of violations data.
EPA has not conducted recent audits of enforcement data, but officials we  spoke with from EPA’s drinking water and enforcement offices and three  regions—as well as survey respondents—stated that current SDWIS/Fed  data underreport the percentage of water systems where enforcement  actions have been taken. They also indicated that the SDWIS/Fed data do  not accurately report the percentage of water systems that have returned  to compliance. For example, state officials told us that when they have  quarterly discussions with the regions about the status of enforcement  actions as shown in SDWIS/Fed they discover that the database is not  accurate because the states have not consistently entered the data on  enforcement actions into SDWIS/State.
We examined violations that occurred from 2005 through 2009 to  determine what the states have reported to EPA. According to our analysis  of SDWIS/Fed data on enforcement, the states reported that less than half  of these health-based and monitoring violations were resolved as of March  31, 2010 (see fig. 4). Specifically, we found that about 59 percent of health- based violations and about 49 percent of monitoring violations committed  by community water systems had not been resolved. However, given that  the enforcement data have not been audited for several years, as well as  the concerns of officials we spoke with, we cannot be certain that the  results of our analysis accurately reflect the status of enforcement actions.
section_title: EPA and State Officials Indicated That Violation and Enforcement Data Are Unreliable for Several Reasons
paragraphs: EPA and state officials responding to our survey or in interviews cited  several factors as contributing to inaccuracies in SDWIS/Fed data on  health-based and monitoring violations and the status of enforcement. For  the violations data, some factors were cited as contributing to both  compliance determination and data flow errors—such as inadequate  training and guidance—but the importance of these factors varied by the  type of error. For enforcement data problems, other factors were often  cited, such as higher priorities, inadequate guidance, and information  system flaws.
subsections: section_title: Incorrect Compliance Determination Data and Data Flow Errors Occur for Several Reasons, According to EPA and State Officials
paragraphs: We asked survey respondents whether they thought any of five factors  (information system structure; training by state or federal agencies;  funding from state or federal agencies; state staffing levels; or guidance  from EPA) contributed to incorrect compliance determinations and to the  less common data flow errors; we also asked them to indicate if other  factors were important. As figure 5 shows, at least half of the 41  respondents identified each of the factors as contributing to compliance  determination errors, with training and staffing cited most often. And, as  the figure shows, more than two-thirds of the 41 respondents cited the  information system structure as contributing to data flow errors and more  than half of respondents cited state staffing and training as contributing  factors.
Respondents also provided more detailed information on the factors they  identified as contributing to incorrect compliance determinations and data  flow errors. For example, with regard to incorrect compliance  determinations, one respondent said that training for new drinking water  rules was limited and training for old drinking water rules was virtually  nonexistent. Another respondent said that staffing levels were at an all- time low while another said that states had always experienced a revolving  door for compliance staff. An EPA official responded that a state might not  issue a violation because of “sympathy” for a water system if the state  viewed the violation as not being a major health problem. With regard to  data flow errors, several respondents said that SDWIS does not have  adequate quality control features to clearly identify errors that might occur  when the states transfer violations data from SDWIS/State to SDWIS/Fed.
When we asked survey respondents to identify the most important steps  they believe that EPA could take to address compliance determination  errors, the most frequent suggestions—from 18 of the 41 respondents— concerned training and guidance. For example, 9 respondents said that  EPA needs to improve the timeliness of guidance on how to make  compliance determinations to ensure that the guidance does not come  after the date that a drinking water rule takes effect. With respect to  actions states should take, the most frequent comments related to  management and training, from 25 and 22 respondents, respectively. Many  of the comments regarding management called for states to conduct more  thorough oversight or to hold their staff accountable.
Survey respondents also identified the most important steps they believe  that EPA could take to address data flow errors. The most frequent  suggestions for lowering the error rate concerned information system  structures and management, from 16 and 15 respondents, respectively.  With respect to information systems, respondents said that EPA should  take action to address the quality, complexity, or ease of use of SDWIS.  Most who commented on management called for more oversight and  accountability.
section_title: Unreliable Data on Enforcement Actions Are Attributed to Higher Priorities, Inadequate Guidance, and Information System Flaws
paragraphs: According to EPA and state officials we interviewed, as well as survey  respondents, the factors that contributed to concerns about incomplete  data on enforcement actions and water systems’ return to compliance are  similar to those that contributed to unreliable data on violations. For  example, EPA and state officials told us that some state agencies have not  routinely and thoroughly entered data on enforcement actions or returns  to compliance into SDWIS/Fed because it is a low priority for their limited  staff. Officials from EPA regions said this is particularly the case for  monitoring violations that states may have considered less serious than  violations of health standards.
State and EPA officials also cited a lack of guidance from EPA on what  conditions must exist for a system with a violation to be recorded as  returned to compliance as having been a factor contributing to incomplete  data on enforcement. Recognizing the importance of these definitions,  EPA collaborated with states and the Association of State Drinking Water  Administrators on guidance it issued in 2010. EPA regional officials told us  the new definitions would likely lead to improvements in the states’  reporting on returns to compliance. In addition, officials we spoke with  stated that SDWIS/Fed used to have an automated function that would  categorize some common violations as returned to compliance if certain  subsequent conditions existed. However, that function is no longer  available, meaning that state officials need to enter the information  manually. In its comments on a draft of this report, EPA said that the  function was removed because it did not work correctly.
See appendix II for more details on the results of our survey.
section_title: Incomplete and Inaccurate SDWIS/Fed Data Hamper EPA’s Ability to Manage the PWSS Program and Communicate Progress toward Its Strategic Objective
paragraphs: Incomplete and inaccurate data on violations and enforcement actions  limit EPA’s ability to identify water systems with the most serious  compliance problems and ensure its enforcement goals are met.  Unreported violations and unreliable enforcement data also impede EPA’s  ability to monitor and fully communicate to Congress and the public the  agency’s progress toward its strategic objective of reducing the public’s  exposure to contaminants in drinking water.
subsections: section_title: Incomplete and Inaccurate Data on Violations and Enforcement Actions Reduce EPA’s Ability to Ensure Its Enforcement Goals Are Met
paragraphs: Incomplete and inaccurate data on violations and enforcement actions  reduce EPA’s ability to ensure that it is achieving its goal of targeting for  enforcement those systems with the most serious compliance problems.  Specifically, the lack of reliable data in SDWIS/Fed reduces the usefulness  of EPA’s Enforcement Targeting Tool for identifying water systems with  the most serious compliance problems. That is, water systems without a  complete violations record in SDWIS/Fed could receive a lower  enforcement targeting score indicating a higher level of compliance than  other systems whose violation record is complete. Conversely, systems  whose return to compliance has not been recorded in SDWIS/Fed could  receive a score that is higher, or worse, than warranted. According to  EPA’s current enforcement policy, water systems whose scores equal or  exceed 11 points are considered to have serious compliance problems and  are targeted for enforcement actions.
To demonstrate the effect that unreported health and monitoring  violations have on the implementation of the Enforcement Targeting Tool,  we calculated two scores for each community water system audited by  EPA in 2007, 2008, and 2009—a total of 1,225 systems over the period. One  score was based on more complete data incorporating the violations found  in the data verification audits, and the other score was based on violations  in SDWIS/Fed. Because the audited data are a more complete dataset, we  expected to see, and indeed found, differences between the two scores for  each system. For 16 percent of the systems, the point difference between  the two scores alone equaled or exceeded the 11-point threshold. Another  14 percent had scores that were 6 to 10 points higher, which would  increase the likelihood that these systems would have been prioritized for  enforcement under EPA’s targeting tool. The results of our analysis are  shown in figure 6.
Overall, according to our analysis, 73 percent of the water systems (or 892)  had a different score using the two sets of data. Twenty-seven percent  (333) of the water systems showed no difference between the scores  calculated using the two sets of data. We found that the majority of score  differences were the result of unreported monitoring violations. The  Enforcement Targeting Tool assigns a much lower weight to monitoring  violations than to health-based violations, but, as previously discussed, the  number of monitoring violations plays an important role in limiting EPA’s  ability to identify systems with serious compliance problems. While most  of the 1,225 water systems had a higher score with audited data than with  SDWIS/Fed data, 2 percent (21 systems) had lower scores because EPA  found in its audit that the violations had not occurred.
When the SDWIS/Fed data are incomplete, EPA’s ability to identify and set  priorities for enforcement in water systems is compromised. For example,  because the Enforcement Targeting Tool uses SDWIS/Fed data that may  be missing violations, some systems may not be assigned enough points to  exceed EPA’s threshold of 10 points for priority enforcement action. For  some of these systems, one or two additional points may be all that are  needed to exceed the threshold and in other cases, as described below, the  point difference for a particular system exceeded the threshold by an  extraordinary amount. For example, we calculated the following for three  systems:    A 170 point difference: The score we calculated was 3 points for one  water system in Vermont using SDWIS/Fed data, but 173 when we  accounted for unreported health and monitoring violations that EPA  found in its 2009 audit.
A 138 point difference: The score we calculated was 0 using SDWIS/Fed  data for a tribal system in New York that EPA’s Region 2 office  oversees as the primacy agency, but 138 when we accounted for  unreported health and monitoring violations found during EPA’s 2009  audit.
A 95 point difference: The score we calculated was 0 using SDWIS/Fed  data for a system in Utah, but 95 when we used data from EPA’s audit.  The difference was entirely attributable to unreported monitoring  violations.
Our analysis echoes concerns voiced by respondents to our survey; 22 of  41 respondents indicated that the usefulness of EPA’s Enforcement  Targeting Tool is affected by limitations in the SDWIS/Fed database. One  respondent said “missing data will significantly affect the usefulness of the  results.” Another respondent said the tool “is hinging on the information  recorded in SDWIS/Fed” and that “the tool is as good as the data  provided.”
Survey respondents and state and EPA officials also reported that  incomplete or inaccurate data on the resolution of violations could result  in a water system receiving a higher score for enforcement priority than it  merits. EPA and state officials told us that states do not always indicate in  SDWIS/Fed that a violation is resolved, perhaps causing the Enforcement  Targeting Tool to mistakenly place the system on the targeted  enforcement list. According to one survey respondent, this condition will  “confuse states and lead to continued poor quality data.” Another  respondent said that use of the Enforcement Targeting Tool “is a waste of  time” without steps taken to fix this issue. State officials told us that in  their regular review of the targeted list with EPA regional officials, they  can recognize when a system has been erroneously included on the  targeting list because resolved violations were not recorded and they can  correct the discrepancy. However, EPA officials have told us this data  correction process is a time-consuming one that places additional  demands on limited state and EPA enforcement staff.
Incomplete SDWIS/Fed data can also limit EPA’s ability to ensure that the  states meet the agency’s enforcement goal that targeted systems have  returned, or are returning, to compliance in a timely fashion. EPA’s  Enforcement Response Policy calls for states to work with systems to  resolve violations or put the system on a “path to compliance” within 6  months of when the system becomes a priority system on an Enforcement  Targeting Tool list. However, unreported data on enforcement actions can  hamper EPA’s ability to determine whether states have met that goal. For  instance, while states might take an enforcement response that leads, or  will lead, a water system to resolve the violation, states frequently do not  enter this information into the SDWIS/Fed database or enter the  information months or years later, according to EPA and state officials we  spoke with. Either situation hampers EPA’s ability to track the timeliness  of enforcement responses.
section_title: Unreported Violations and Enforcement Data Impede EPA’s Ability to Monitor and Report Progress Toward Its Strategic Objective of Reducing Exposure to Contaminants in Drinking Water
paragraphs: Unreported violations and enforcement data impede EPA’s ability to fully  measure and communicate its progress toward meeting the strategic  objective of reducing human exposure to contaminants in drinking water.  The agency has established a number of indicators and targets that it uses  to measure its progress toward meeting that objective. However, the  unreliable quality of the violations data and concerns about the accuracy  of enforcement data in SDWIS/Fed make it difficult for EPA to reliably  communicate the relative public health risk posed by community water  systems’ noncompliance with SDWA and the progress made in resolving  noncompliance in a timely manner. For example:    EPA’s 2011 national water program guidance contains an indicator for  the number and percentage of systems serving less than 10,000 people  with certain repeated health-based violations. EPA’s ability to set and  reliably use this type of indicator requires complete and accurate data  on violations, but as we have shown, the SDWIS/Fed data on violations  are not reliable.
EPA’s 2011 national water program guidance also contains an indicator  for the average time taken for systems serving less than 10,000 people  to return to compliance after committing certain health-based  violations. However, the ability to set and reliably use an indicator of  this type requires complete and accurate data on enforcement actions.  As we have previously indicated, EPA and state officials we  interviewed told us the enforcement data in SDWIS/Fed are not  reliable.
Unreliable data quality also limits EPA’s ability to introduce or modify  targets to manage its program and communicate progress in meeting the  program’s goals. Quality data are necessary to accurately measure  performance relative to strategic targets. Two key EPA strategic targets  associated with the agency’s strategic objective of reducing exposure to  contaminants in drinking water—the percentage of community water  systems that met all health-based standards and the percentage of the  population served by community water systems that received drinking  water that met all applicable health-based drinking water standards—are  broad measures of compliance. However, these measures do not provide  information on the relative severity of the violations or account for  systems that have multiple health-based violations, offering the public a  narrow view of the quality of the nation’s water systems and not clearly  communicating the public health risk posed by these systems’  noncompliance with SDWA. For example, a water system with multiple  health-based violations is effectively “counted” the same as a system with  one health-based violation. Thus, the relative health risk posed by  different systems’ noncompliance is not apparent. Without complete and  accurate SDWIS/Fed data it is difficult to develop a new measure or  modify these strategic targets. Similarly, without complete and accurate  data from the states, EPA will be unable to establish reliable measures or  targets regarding the rate of reduction in health-based violations or  compliance with monitoring requirements or further EPA’s core value of  transparency.
section_title: Actions EPA and States Are Taking to Improve the Quality of Data in SDWIS Have Not Been Fully Successful and More Actions Are Planned
paragraphs: EPA and the states have taken actions over many years to identify and  address the causes of incomplete and inaccurate violations data, but those  efforts have not been fully successful, according to those we surveyed.  EPA has conducted audits to assess the quality of state violation data in  SDWIS/Fed and developed recommendations for improving data quality.  Survey respondents generally reported that those audits have contributed  to improvement, but EPA has discontinued them. EPA and the states also  established work groups to address data management and quality. In  addition EPA has emphasized the importance of specific data quality  management tools, although it has not required states or regions to use  them. More recent EPA initiatives include a new strategy for data sharing,  plans to redesign SDWIS, and a new tool to help the states make and  report compliance determinations and enforcement actions.
subsections: section_title: According to Survey Respondents, EPA’s Audit Recommendations Contributed to Improving Data Quality, but the Agency Has Discontinued Them, at Least Temporarily
paragraphs: As described earlier, EPA used its data verification audits to assess the  quality of the violations data and, to a lesser extent, the enforcement data  the states have submitted to SDWIS/Fed. The agency also used the audits  it conducted from 1996 through 2004 to develop state-specific and national  recommendations for improving data quality. EPA and state officials we  surveyed had mixed, but generally favorable, views about the value of the  audits’ recommendations with regard to improving data quality. Eight  respondents said the recommendations were very effective in improving  data quality, while most respondents (26 of 39) said the recommendations  were only slightly or moderately effective.
According to respondents, the audits pointed out states’ inefficiencies and  poor practices. For example, one respondent said that states are able to  use the results as a guide to improve training for staff and improve data  quality. Despite the recommendations offered to help states, six  respondents indicated that the states or regions did not adequately change  their practices in response to the audit findings. For example, one EPA  headquarters manager commented that states may incorrectly interpret  systemic problems identified through the audit as isolated problems to be  corrected only at the water systems covered by the individual audits.  Nonetheless, seven respondents stated that the audits’ scope or  methodology was not adequate to determine data quality.
EPA discontinued the audits of violations data in 2010 due to funding  constraints. According to the Director of the Office of Ground Water and  Drinking Water, EPA may be able to resume the audits in 2011, but at a  much reduced number. EPA conducted an average of about 17 audits of  states, regions, and other primacy agencies in 2007 through 2009, but the  director told us in December 2010 the agency may be able to do 4 or 5 in  2011. EPA had not done any 2011 audits as of June 2011. In its comments  on a draft of this report, EPA said that the Office of Water will conduct six  to eight audits in 2011.
section_title: EPA and the States Have Established Joint Work Groups to Address Data Issues
paragraphs: In the 1990s, EPA and the states jointly established two work groups  charged with providing analysis and recommendations on various aspects  of data management and formed a third such group in 2010. According to  most of their members, the two older groups—the Data Management  Steering Committee and Data Technical Advisory Committee—were  effective at helping EPA and the states improve data quality. On the other  hand, the members of the newer Data Quality Work Group were almost  evenly divided on whether this new group has been effective or not  effective.
The Data Management Steering Committee is charged with supporting  EPA and the states in their cooperative efforts to enhance management of  drinking water data. In explaining their answers to a question about the  group’s effectiveness, one-third of the members said the committee had  helped EPA and the states understand the nature of the data quality  problem and about one-half said it provided direction. However, one-third  of the members commented on the lack of implementation of the  committee’s recommendations.
The Data Technical Advisory Committee is responsible for recommending  ways to obtain the data EPA needs to carry out its PWSS program  responsibilities. About three-fifths of the members commented that the  committee had helped EPA and the states understand data problems and  had provided direction. However, similar to the steering committee,  advisory committee members had concerns about EPA’s implementation  of recommendations, with close to half saying that the agency’s  implementation had been inadequate.
The Data Quality Work Group met several times in 2010 and outlined draft  recommendations for improving data quality, including additional training,  standard operating procedures for staff managing any new drinking water  rules, checklists of rule milestone dates for states, and quality  assurance/quality control checks for SDWIS data. However, the Office of  Ground Water and Drinking Water diverted the work group staff in 2010 to  focus their attention on implementing the Administrator’s Drinking Water  Strategy before the group could issue final recommendations, according to  senior office staff. Perhaps in light of that, among the most common  comments from members of the group was that the work group was too  new to evaluate or that its activity level had been inadequate. The Office of  Ground Water and Drinking Water staff noted they would consider the  work group’s draft recommendations in its redesign of SDWIS.
section_title: The Office of Ground Water and Drinking Water Has Emphasized the Importance of Data Quality but Not Required States or Regions to Take Actions
paragraphs: In March 2008, EPA reported the results of the audits it conducted of state  data in 2002 through 2004. As it had done in its prior reports on state  audits, EPA included recommendations for improving data quality in this  report; these recommendations took the form of an action plan in its 2008  report. The 2008 action plan was a joint effort of EPA and the Association  of State Drinking Water Administrators to provide recommendations for  achieving the goal set in 2006 of 90 percent complete and accurate data for  health-based violations, as well as improving the quality of monitoring  violations data. According to the action plan, the largest challenge was  ensuring that all data reflecting determinations of violations were entered  into SDWIS/Fed. The plan called for, among other things, the development  of new data management tools as well as the implementation of tools that  EPA has developed over the years to improve compliance determinations.  However, as discussed below, widespread implementation of those tools  has not yet occurred.
In April 2009, the Director of EPA’s Office of Ground Water and Drinking  Water issued a memorandum to EPA’s regional water management  directors calling attention to (1) the incomplete implementation of the  2008 action plan and (2) the importance of increasing oversight and  accountability of the states for the quality of drinking water data. Noting  that the quality of drinking water data in SDWIS/Fed was called into  question in the media in the late 1990s and was the subject of a 2004 report  by EPA’s Office of Inspector General, the director stated that the quality  of data continued to be too low. She also cited the 2006 agreement  between EPA and the Association of State Drinking Water Administrators  to set a goal that 90 percent of health-based drinking water violations be  completely and accurately reported to SDWIS/Fed and said that more than  10 states had met the goal, but the overall goal had not been met.
subsections: section_title: The Director of the Office of Ground Water and Drinking Water Requested Help in Implementing the 2008 Action Plan
paragraphs: In her memorandum, the director noted that one of the conditions of  primacy the states must meet is to report all violations. To improve data  quality, the director called upon the regions to increase their efforts to  implement the 2008 action plan. Her memorandum and the action plan call  for the states to increase their use of several data management tools that  EPA has developed over the years to improve accuracy, including  SDWIS/State, electronic data verification (eDV), and electronic reporting  from laboratories to states. Although EPA has developed these tools to  improve compliance determinations and data flow, the states are not  required to use them as a condition of primacy or their PWSS grant  agreements. We gathered information from EPA regarding the current  status of these tools and asked survey respondents to comment on the  factors that have limited the use of these tools and the steps they believed  should be taken to increase the tools’ use.
EPA reported to us in March 2011 that eight states, one tribe, and one  territory were not using SDWIS/State at all. Of the states that do use  SDWIS/State, some report using the database only for particular drinking  water rules. For example, an EPA survey of states in mid-2010 found that  20 of the 55 primacy agencies were using SDWIS/State to make  compliance determination decisions for the surface water treatment rule  and 35 were using it for the total coliform rule.
Due to limitations in the data available to us and inherent difficulties in  establishing a cause-and-effect relationship, we could not determine  whether a state’s use of SDWIS/State leads to more reliable data on  violations of particular SDWA rules. However, 26 of the 41 respondents  indicated that more widespread use of SDWIS/State would improve data  quality; 5 respondents indicated it would not; and 10 indicated they did not  know. Of the 26 respondents who provided detail on why they thought  more widespread use of SDWIS/State would improve data quality, 17  indicated it would promote more consistent and accurate compliance  determinations through automation. The most common theme among the  respondents’ suggestions for what EPA should do to address the factors  that have prevented full use of SDWIS related to the quality, complexity,  and ease of use of the system. For example, one respondent said that EPA  should be aware of the needs of drinking water managers, who are the  principal users of SDWIS/State, rather than database managers.
section_title: Electronic Data Verification
paragraphs: According to EPA officials, as of March 2011, only seven states had done  pilot tests of EPA’s electronic quality control tool for SDWIS/State, known  as electronic data verification, or eDV. EPA officials told us that the eDV  tool needs additional refinement to be fully compatible with SDWIS/State.  eDV could assist states in making compliance determinations according to  18 of the 24 survey respondents who were familiar with it. They said that  using the tool would improve data quality by improving states’ oversight or  auditing capability. However, EPA and state officials told us this tool can  be used only by states that use SDWIS/State to manage all of the drinking  water rules, and survey respondents noted that the need to fully use  SDWIS/State was a factor that prevented more states from using eDV. The  survey respondents’ most common recommendation for EPA was to  improve the quality and ease the use of the tool. For example, one  respondent said that EPA needs to update the tool to keep pace with  changes in regulations. At the same time, many respondents suggested that  the states need to make a greater commitment to using SDWIS/State and  eDV more.
section_title: Electronic Reporting from Laboratories to States
paragraphs: EPA has developed a tool that testing laboratories can use to  electronically transmit the results of community and other public water  systems’ monitoring directly to the state. However, according to EPA’s  mid-2010 survey, only 19 states were using this tool. Of the 40 survey  respondents who expressed an opinion, 39 believed that more widespread  use of this tool would improve data quality, and 25 of these respondents  stated the tool would reduce data entry errors and increase accuracy.  Another 11 respondents said the tool would increase the speed of data  exchange between the states and EPA, and 10 said it would free up state  resources that could be used to improve data quality in other ways. For  example, one respondent said that, in the long run, electronic reporting  should save state resources by reducing the need for data entry staff but  that in the short run, switching to electronic reporting requires technical  support for the laboratories and additional resources.
According to our survey respondents, the two leading barriers to having  more states require electronic reporting are (1) laboratories’ inadequate  capability to implement the reporting technology and (2) the states’ lack of  legal authority to make the tool’s use a requirement. Specifically, many  survey respondents said that laboratories, particularly small ones, are not  always adequately equipped or staffed to adopt electronic reporting.  Several survey respondents said that EPA needs to provide support to  laboratories to make it easier to adopt the tool. Some respondents also  said that as EPA’s current SDWA regulations do not require electronic  reporting and some state laws prohibit state agencies from including  requirements in their PWSS programs that are more stringent than what is  required by SDWA, the state agencies are unable to require electronic  reporting. Nine respondents said that EPA should require electronic  reporting.
Several respondents also identified EPA’s Cross-Media Electronic  Reporting Rule as a regulatory barrier to more widespread use of  electronic reporting. This rule provides the legal framework for  electronic reporting under all of the agency’s environmental regulations.
The rule requires states, tribes, and local governments that wish to use  electronic reporting for implementing authorized federal environmental  programs to obtain EPA approval, which may require modifications to  electronic reporting systems to meet EPA requirements. One state  respondent said that many states do not have the financial resources to  build a system to receive data electronically because of the constraints of  this rule, noting that the rule places very tight requirements on the security  required for receiving data electronically. Several respondents called on  EPA to review the need for the rule or abolish it.
section_title: The Director of the Office of Ground Water and Drinking Water also Called for Increased Regional Oversight and Accountability
paragraphs: Citing the need for strong regional oversight of the states’ PWSS programs,  the director also requested in her April 2009 memorandum that the regions  take specific actions over and above those identified in the 2008 action  plan. Specifically, she called for the regions to    provide documentation of good standard operating procedures and  lessons learned that may enable EPA to improve SDWIS/Fed data  quality, among other things;    discuss with states annually (or more frequently) the completeness and  accuracy of the violations data reported to SDWIS/Fed, including a  review of the state’s implementation of recommendations contained in  previous data verification reports; and  include language in future PWSS grant agreements indicating that the  state must make compliance determinations that are consistent with  applicable drinking water regulations, report all violations and  enforcement actions to SDWIS/Fed in a timely fashion, and otherwise  comply with 40 CFR §142.15. Also include any corrective action steps  identified by data verification audits or program reviews in the state’s  annual work plan.
Although the Director of the Office of Ground Water and Drinking Water  requested these actions, she told us her office does not have the authority  to require the regions—which report directly to the EPA Administrator— to do so. The director and other drinking water program managers we  interviewed told us the regions had responded to her request, but the  office had not assessed and could not document the extent to which the  regions had complied with the requests to discuss data quality with the  states or add relevant language to grant agreements.  In its comments on  a draft of this report, EPA stated that all of the regions have incorporated  data quality into their discussions with states and data quality has been  incorporated into grant agreements or state workplans. However, we were  unable to verify these statements; in response to our request during the  comment process, EPA said that documentation was not available.  According to EPA, it made the statements on the basis of e-mail  communications and discussions between the managers and staff in the  Office of Ground Water and Drinking Water and regional management and  staff.
The director also requested comments from the regions’ water  management directors on four proposed measures that would assist the  office in monitoring the regions’ oversight of the states’ performance,  including several directly related to the steps discussed above:  the percentage of states within a region with which the region has an  annual discussion regarding data quality;  the percentage of states that have an action plan to correct deficiencies  relating to drinking water compliance determinations or data reporting  that were noted in the most recent EPA data verification audit;  the percentage of a region’s annual PWSS grants that include grant  conditions requiring the states to make compliance determinations that  are consistent with drinking water regulations; and  the extent to which a region is achieving EPA’s goal that 90 percent of  health-based violations are completely and accurately reported to  SDWIS/Fed, when a region is acting as the primacy authority for a  particular rule in a state.
According to Office of Ground Water and Drinking Water officials, the  regions responded to the director’s request for comments but did not fully  support the proposed performance measures, and none of these have been  implemented as of March 2011.
section_title: EPA Administrator Issued New Strategy for Data Sharing and EPA Announced Plans to Redesign SDWIS
paragraphs: In March 2010, the Administrator of EPA issued a drinking water strategy  that called for, among other goals, the agency and the states to increase  data sharing on water systems. As part of this strategy, EPA announced it  will redesign SDWIS to help to meet the Administrator’s goals for data  sharing. According to the director of the Office of Ground Water and  Drinking Water, software for the next generation of SDWIS is only at the  beginning stages of development; she anticipates it will be ready by 2014,  depending upon the availability of funding. To help achieve the goals of  the strategy, EPA signed a memorandum of understanding in November  2010 with three associations that represent state agencies and officials.  EPA also formed an Implementation Work Group comprising agency and  state officials to further the data sharing goals spelled out in the  memorandum.
The November 2010 memorandum of understanding on data sharing— which is a voluntary agreement among the parties—outlines the vision,  goals, terms, and conditions under which drinking water monitoring data  are to be exchanged between the states and EPA. The anticipated benefits  of data sharing include allowing states to more readily compare their  water system monitoring results with EPA regulations and with the  SDWIS/Fed data before the states submit the data to EPA. EPA intends to  use the shared data for a variety of purposes, including calculating  national and state data completeness, accuracy, and timeliness; evaluating  differences in state interpretations of EPA regulations; and conducting  national program oversight. The mission of the Implementation Working  Group is to recommend ways for states to share appropriate compliance  monitoring data that eventually will be housed in the next generation of  SDWIS.
EPA officials said they expect this redesign of SDWIS—and accompanying  revisions to state data submission requirements—to expand the amount of  data that EPA receives electronically from the states. With SDWIS/Fed,  EPA generally only receives data from the states on inventories, violations,  and enforcement actions. According to the Director of EPA’s Office of  Ground Water and Drinking Water, the next generation of SDWIS would  give EPA access to the compliance monitoring and enforcement data now  collected by the states. She told us that having direct access to the states’  raw monitoring data would improve EPA’s ability to better understand  national patterns of compliance and to diagnose problems faced by states.  For instance, according to the director, the data could reveal that  particular rules are hampered by a misunderstanding of the requirements,  and EPA could use that information to write regulations that are easier to  understand and report. The director said that a redesigned SDWIS could  also reveal and address instances when a state has made a compliance  determination error. However, she also said that some compliance  determination errors can be addressed with a redesigned SDWIS, but  others might need to be addressed through better training or writing  regulations more clearly so that state staff understand what constitutes a  violation.
According to the director, under the agency’s current position, states will  continue to have the option to use the next generation of SDWIS. EPA will  continue to provide ways for states that do not use SDWIS to transfer their  data to the agency. However, EPA will expect those states, like those using  SDWIS, to share their compliance monitoring data with EPA.
section_title: EPA Is Developing a New Compliance Determination and Violations/Enforcement Reporting Tool
paragraphs: In November 2010, the Office of Ground Water and Drinking Water  unveiled an initial version of another tool for improving compliance  determinations and data quality—the Compliance Determination and  Violation/Enforcement Reporting Tool. According to the office, the tool  was designed by Region 5 staff to consolidate, update, and supplement  EPA guidance on SDWA violations of specific requirements in one  electronic document. The tool includes violation descriptions, compliance  determinations, violation reporting instructions, common discrepancies  from data verifications, related EPA memos, enforcement tracking  instructions, and return to compliance definitions. The target audience for  the tool is state and regional compliance, enforcement, and data staff. The  tool is being developed in modules for each National Primary Drinking  Water Regulation. The first module of the tool was for the lead and copper  rule. EPA officials said they anticipate all modules will be developed by  the end of fiscal year 2011 if funding is available.
section_title: Conclusions
paragraphs: EPA relies on the soundness of state-reported data to ensure that  community water systems are complying with SDWA and that the states  and regions are taking appropriate enforcement actions against  noncompliant water systems. As our analysis of EPA’s audit data for 2007  through 2009 shows, however, states continue to fall short in providing  accurate and complete data on health-based and monitoring violations.  EPA’s data verification audits after 2004 did not examine the quality of  data on enforcement actions and systems’ return to compliance, but EPA  officials and survey respondents told us that current state data on such  actions are also incomplete or inaccurate. In addition to discontinuing its  audits of enforcement data after 2004, EPA also discontinued its audits of  violation data after 2009 because of budget constraints. The agency hopes  to resume these audits of violation data in 2011, but the number of states  to be audited would be greatly reduced from an annual average of about 17  from 2007 through 2009 to 6 to 8. Conducting fewer audits of state- reported data—both violations and enforcement data—will hamper the  effectiveness of EPA’s oversight of the states and its ability to assess its  efforts to improve data quality.
EPA and the Association of State Drinking Water Administrators  established a goal of completely and accurately reporting 90 percent of  health-based violations, but EPA has not set a similar goal for monitoring  violations. Monitoring violations may reflect a wide range of  circumstances, such as instances in which monitoring was done but was  not reported to the state in a timely fashion or the potentially more serious  situation in which required monitoring was not done at all. We found,  however, that the number of monitoring violations was positively and  statistically significantly related to the rate of health-based violations.  Recognizing the importance of having complete and accurate data on  monitoring violations, a majority of those state and EPA officials we  surveyed who voiced an opinion supported the idea of having a goal for  the quality of data on monitoring violations.
As called for by GPRA, EPA has established several performance  measures with associated targets and indicators for community water  systems to assess progress toward the agency’s strategic objective of  reducing exposure to contaminants in drinking water. Each year, EPA  publicly reports systems’ performance levels relative to the targets using  data from SDWIS/Fed. To be useful and appropriate, these performance  measures should clearly reflect conditions that directly relate to human  exposure to contaminants. However, we found that some of the measures  that EPA relies upon to gauge national compliance levels measure how  many systems are out of compliance but not the extent to which they are  out of compliance, which does not clearly communicate the public health  risk posed by these systems’ noncompliance with SDWA. In addition, some  measures that EPA uses depend on data on violations and the status of  enforcement actions that are unreliable. We found that incomplete and  inaccurate data could impede EPA’s ability to monitor and report progress  toward its strategic targets for those measures, including having those  systems return to compliance in a timely manner.
Recognizing its long-standing problem of receiving incomplete and  inaccurate state data on violations, EPA has made efforts to improve the  quality of data reported to SDWIS/Fed. However, many of those efforts,  including those to implement EPA’s 2008 action plan, have not been fully  successful. In light of the need for more improvement, the Director of the  Office of Ground Water and Drinking Water requested in her 2009  memorandum that the regional water managers take numerous steps to  implement the action plan—including encouraging the states to increase  their use of SDWIS/State, electronic data verification, and electronic  reporting from laboratories to states. Both the action plan and additional  steps requested in the memorandum have the potential to address the  factors EPA and state officials identified as contributing to unreliable data  quality. According to the director, however, the states currently are not  required to use the data management tools that EPA has developed as a  condition of primacy or their PWSS grant agreements, and many have  chosen not to do so. In response to our survey, EPA and state drinking  water officials generally said these tools would help improve data quality  but noted barriers they believe have prevented more widespread use of  them. For example, the most common theme among the survey  respondents’ suggestions for increasing the use of the SDWIS/State and  electronic data verification tools was to address their quality, complexity,  and ease of use. The Director of the Office of Ground Water and Drinking  Water also asked the regional water management directors in her  memorandum to increase their oversight of state programs. For example,  she asked the regions to include language in future grant agreements  indicating that the state, among other things, must make compliance  determinations that are consistent with applicable drinking water  regulations. However, the director also indicated that her office does not  have the authority to require the regions to take the actions requested in  the memorandum, and could not document the extent to which the  regions had done so. EPA’s plan to develop a next generation of SDWIS  and to increase its access to state data might help the agency ensure that it  receives higher quality violations and enforcement data from the states.  However, it is uncertain if and when the new system or increased access  to data will be available. In the meantime, further efforts to overcome the  barriers to implementation of the 2008 action plan and the director’s 2009  memorandum are needed to improve state data.
section_title: Recommendations for Executive Action
paragraphs: To improve EPA’s ability to oversee the states’ implementation of the Safe  Drinking Water Act and provide Congress and the public with more  complete and accurate information on compliance, we recommend that  the Administrator of EPA take the following four actions:    Resume data verification audits to routinely evaluate the quality of  selected drinking water data on health-based and monitoring violations  that the states provide to EPA. These audits should also evaluate the  quality of data on the enforcement actions that states and other  primacy agencies have taken to correct violations.
Work with the states to establish a goal, or goals, for the completeness  and accuracy of data on monitoring violations. In setting these goals,  EPA may want to consider whether certain types of monitoring  violations merit specific targets. For example, the agency may decide  that a goal for the states to completely and accurately report when  required monitoring was not done should differ from a goal for  reporting when monitoring was done but not reported on time.
Consider whether EPA’s performance measures for community water  systems could be constructed to more clearly communicate the  aggregate public health risk posed by these systems’ noncompliance  with SDWA and progress in having those systems return to compliance  in a timely manner.
Work with the EPA regions and states to assess the progress made in  implementing the steps called for by the 2008 action plan and the  Director of the Office of Ground Water and Drinking Water’s 2009  memorandum; identify the barriers that have prevented more  widespread implementation of the action plan and memorandum; and  develop and publish a strategy for overcoming those barriers.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to EPA for review and comment. The  agency provided written comments, which are reproduced in appendix III.  EPA partially agreed with two of our recommendations, disagreed with  one, and neither agreed nor disagreed with another. Our responses to  EPA’s comments on our recommendations follow, and our responses to  EPA’s attachment of substantive comments are in appendix III. EPA also  provided technical comments that we have incorporated as appropriate.
In its overall comments, EPA said that it recognizes the importance and  value of high quality data to complement the activities that comprise its  oversight of primacy agencies. EPA also acknowledged that GAO found  data quality problems similar to those previously found by the agency  during data verification audits and that underreporting violations data and  enforcement actions may limit the public’s full knowledge of the status of  public water system compliance. EPA noted that it has implemented a  number of activities to improve data quality and its ability to oversee the  drinking water program. We agree that EPA has taken steps to improve  data quality and describe many of them in our report.
EPA also noted that complete and accurate data are important in order to  effectively target enforcement to those systems with the most serious  compliance problems. The agency added that its 2009 Enforcement  Targeting Tool provides an incentive to the states to keep their  enforcement data current to ensure that the tool yields accurate scores.  We agree that the tool underscores the importance to the states of keeping  enforcement data current. However, the scores generated by the tool will  also be incorrect if data on the existence of violations are incomplete or  inaccurate. We believe that whereas the use of the tool provides an  incentive to the states to improve the accuracy of their enforcement data,  it does not necessarily provide them an incentive to improve the accuracy  of their violations data.
EPA partially agreed with our first recommendation that it resume data  verification audits of violations and enforcement actions. The agency  stated that it has found that data verification audits provide valuable  information on data completeness and accuracy and that it plans to  conduct six to eight audits during calendar year 2011. However, EPA did  not commit to conducting data verification audits beyond 2011. Instead,  EPA said that until the next generation of SDWIS is deployed, thus  enabling the agency to view compliance monitoring data and compliance  determinations directly, it will consider using data verification audits to  evaluate data quality. EPA did not comment on how it would evaluate data  on enforcement actions taken to correct violations. We understand that  the next generation of SDWIS may enable EPA to more directly monitor  water systems data and oversee the states’ compliance determinations. We  note that the Director of the Office of Ground Water and Drinking Water  said that the new system would not be available until 2014, depending on  the availability of funding. We continue to believe that EPA should commit  to, not merely consider, conducting data verification audits until the new  system is available, and that those audits should also evaluate the  completeness and accuracy of enforcement data.
EPA did not clearly indicate its agreement or disagreement with our  second recommendation that it work with the states to establish a goal, or  goals, for the completeness and accuracy of data on monitoring violations.  The agency stated that it appreciates the need for improved data quality  for those types of violations. However, EPA neither indicated that it would  adopt a goal nor offered any reasons for why a goal—such as the one it  has for the quality of data on health-based violations—would be  inappropriate. Instead, the agency suggested that along with technology  enhancements as part of the next generation of SDWIS, (1) it will consider  changes to its approach to reporting violations data, and (2) will explore  the possibility of revising the Enforcement Targeting Tool, which could  improve its oversight capabilities. We are not able to evaluate these  changes given their speculative nature, and it is not clear how they might  be relevant to achieving a higher degree of data quality. EPA also stated  that the regions’ annual incorporation of data quality in state grants and  workplans will improve EPA’s oversight capabilities. We agree that  increased emphasis from the regions is necessary and could lead to  improved data quality. However, we continue to believe that setting a goal  for the quality of data on monitoring violations would emphasize its  importance and encourage the states to make and report correct  compliance determinations.
EPA disagreed with our third recommendation that it consider whether its  performance measures could be constructed to more clearly communicate  the aggregate public health risk posed by systems’ noncompliance with  SDWA. The agency noted that its program guidance currently includes a  measure that attempts to address the duration (in “person months”) of  time consumers may be exposed to health-based violations. We describe  this measure in the background section of our report. However, we believe  this measure has the same limitation as other EPA strategic targets, in that  it does not distinguish between water systems with multiple health-based  violations in a particular month and those systems with a single violation  in that month. EPA also stated that it uses a variety of tools that may  convey information on risks associated with noncompliance and show  progress toward returning systems to compliance better than a new  performance measure would. Among the tools EPA identified is a Web site  containing detailed information about the violations for individual water  systems. EPA also said that it recently posted drinking water data to its  Enforcement and Compliance History Online tool. Similarly, EPA said  water systems directly convey to their customers information on public  health risks associated with violations through Public Notifications and  Consumer Confidence Reports. We acknowledge these tools provide the  public with details on the violations that states and water systems have  reported for individual water systems. However, our recommendation  encourages EPA to consider changes to its performance measures to  provide Congress and the public a clearer understanding of  noncompliance with SDWA at a national level and not elicit more  information about the performance of individual water systems. We  continue to believe that it is important for EPA to develop a national  performance measure that helps gauge EPA’s overall management of the  drinking water program.
Regarding our fourth recommendation that EPA work with the regions  and states to assess progress in, and develop a strategy for overcoming  barriers to implementing the 2008 action plan and the Director’s 2009  memorandum, the agency expressed partial agreement by saying it will  continue to assess the progress of improving data quality. EPA also noted  that since these documents were issued, the office has worked with state  and regional staff to understand data quality challenges and opportunities  for improvement. Specifically, EPA commented that all of the regions have  incorporated data quality into their discussions with states and that data  quality has been incorporated into grant agreements or state workplans.  We agree that those actions would signal progress toward implementing  the Director’s memorandum. However, we were unable to verify these  statements because EPA told us that documentation was not available.
EPA told us it made the statements on the basis of e-mail communications  and discussions between the managers and staff in the Office of Ground  Water and Drinking Water and regional management and staff. EPA also  stated that the Data Quality Work Group formed by the Director developed  a list of recommendations to address underlying data quality problems and  that it will continue to evaluate those recommendations. As we note in the  report, the recommendations were in draft, and EPA’s comments did not  provide evidence that they have been adopted. EPA identified other future  actions that it believes will lead to improved data quality. For example,  EPA emphasized the effect that a next generation of SDWIS could have on  improving data quality. We do not disagree that these actions, if taken,  may contribute to improved data quality. However, we point out that EPA  has already developed tools that states could use to improve the quality of  their data on violations but that those tools have not been widely used.  There is no requirement that the states use the next generation of SDWIS,  if and when it is available. Furthermore, EPA did not directly address the  recommendation to identify the barriers that have prevented more  widespread implementation of the action plan and memorandum and  develop and publish a strategy for overcoming those barriers.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution for 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees, the Administrator of EPA, and other interested  parties. In addition, the report will be available at no charge on the GAO  Web site at http://www.gao.gov.
If you or your staff members have any questions about this report, please  contact me at (202) 512-3841 or trimbled@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. GAO staff who made major contributions to this  report are listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To examine the quality of the Safe Drinking Water Information  System/Federal (SDWIS/Fed) data that the Environmental Protection  Agency (EPA) uses to measure community water systems’ compliance  with the health-based and monitoring requirements in the Safe Drinking  Water Act (SDWA), we examined the results of audits EPA conducted  from 1996 through 2009 in which it assessed—for a sample of states—the  completeness and accuracy of violations data those states submitted to  SDWIS/Fed. We evaluated the methods that EPA used to conduct those  audits to test the methods’ validity and determined that, while limited,  these methods were sufficiently reliable for the purposes of our review.  We also conducted our own analysis of EPA’s audit data from 2007  through 2009 in order to arrive at estimates of the quality of the data that  states reported to SDWIS/Fed. We focused our analysis on community  water systems only.
The sample design for the EPA data verification audits consists of a  nonprobability sample of primacy agencies within a given year and a  probability sample of community water systems within each selected  primacy agency. Based on our review of the sample design, we determined  that it is not appropriate for our purposes to make quantitative statements  or inferences about the entire nation from the selected primacy agencies  or comparisons with sampled primacy agency data quality results from the  previous years. As such, we only generated estimates to the states audited  within a given year. Table 1 provides a description of the number of  primacy agencies that were included in the sample for each year.
subsections: section_title: Classification of Violation Types
paragraphs: We classified violations into one of two types: health-based violations and  monitoring violations. By definition, monitoring violations include “other”  violations such as Public Notification and Consumer Confidence Report  violations. We reviewed and decided to use definitions of violation types  provided by EPA to make these classifications. We included lead and  copper treatment technology violations as health-based violations in our  analysis.
section_title: Data Quality Estimates
paragraphs: We defined three separate measures of data quality: accuracy,  completeness, and overall quality (a combination of accuracy and  completeness). These measures are consistent with the measures used by  EPA in previous years. We reviewed and decided to use definitions  provided by EPA to calculate these measures.
To estimate the percentage of violations that were accurate and complete,  we first created a data set with one observation per violation, and then we  used a procedure in statistical software that appropriately accounts for the  stratified cluster sample design. We calculated point estimates and 95  percent confidence intervals. We did not report estimates that have  margins of error that exceed plus or minus 20 percentage points at the 95  percent confidence level.
To determine whether there was a relationship between monitoring  violations and health-based violations, we used audit data to estimate  regression models controlling for size, source and administrative control.  We tested various specifications of three types of statistical models to  ensure that the significance and magnitude of our estimates were  consistent across statistical models.
section_title: Estimates of Compliance Determination and Data Flow Errors
paragraphs: We conducted a subpopulation analysis to estimate the percentage of  incomplete (not reported) violations that were either compliance  determination or data flow reporting discrepancies. We calculated point  estimates and 95 percent confidence intervals. We did not report estimates  that have margins of error that exceed plus or minus 20 percentage points  at the 95 percent confidence level.
The second component of our first objective was to examine the quality of  SDWIS/Fed data on the status of the states’ and EPA regions’ enforcement  actions. Because EPA’s recent audits of state data did not assess the  completeness and accuracy of enforcement data in SDWIS/Fed, we  examined EPA’s national SDWIS/Fed data from 2005 through 2009 to  determine the percentage of violations the states have identified as either  resolved (known as returned to compliance), addressed through an  enforcement action but not yet resolved, or not addressed. We then  interviewed EPA officials to obtain their views on the completeness and  accuracy of those data, and also analyzed relevant comments from survey  respondents. The survey was of EPA and state drinking water officials to  obtain their views on a range of issues related to data quality. (See app. II  for more details on our survey methodology).
To identify factors that have affected data quality, we analyzed 41 survey  responses representing the views of all 44 members of three joint EPA- state work groups that were created to address various aspects of data  management.
To examine the ways in which SDWIS data quality could affect EPA’s  management of the Public Water System Supervision (PWSS) program, we  examined the importance of data quality for two aspects of EPA’s  management of the PWSS program. First, we examined the potential  impact data quality could have on EPA’s Enforcement Response Policy.  This policy uses a targeting tool that assigns scores to community water  systems that are a high priority for enforcement action because of  unresolved violations. To demonstrate the effect that underreported health  and monitoring violations can have on the Enforcement Targeting Tool,  we calculated two scores for each of the approximately 1,200 water  systems audited by EPA in 2007, 2008, and 2009. One score was based on  violations found in the data verification audits, and the second score was  based on violations found in SDWIS/Fed. We then subtracted the two  scores for each system to obtain a point difference, a result that we used  to illustrate the impact of incomplete data on the scoring process. We used  the same methodology EPA uses to create the enforcement score: We  assigned point values to unresolved violations (acute health violations are  worth 10 points, nonacute health violations and some monitoring  violations are worth 5 points, and 1 point for all other monitoring and  reporting violations) and then added these points together to produce an  overall score for each system. However, the enforcement scores we  calculated cannot be considered a water system’s actual score for three  reasons:  1.  EPA’s targeting tool scores 5 years of violation data, whereas the audits  that EPA conducted reviewed state files to identify violations that had  occurred in shorter periods of time. Those time periods typically ranged  from 1 to 3 years, depending on the drinking water regulation.  2.  Due to limited data in EPA’s audit database, our enforcement scores  only include underreported violation discrepancies and do not include  any discrepancies related to accuracy. However, underreported  violations accounted for nearly 97 percent of the discrepancies.  3.  EPA’s enforcement score includes an additional penalty that is tied to  the year of the oldest unaddressed violation. For instance, if a violation  is 5 years old, EPA adds an additional 5 points to the score. Because  our analysis considered violations from a shorter period of time, we  could not duplicate this additional penalty.
While these three limitations prevent us from duplicating EPA’s exact  targeting tool, our analysis presents a conservative estimate of the effect  that poor data quality has on the enforcement scoring process. It is likely  that additional years of underreported violations, plus other enforcement  penalties, would reveal further distortions of the scoring process.
In addition, we examined survey respondents’ views on the impact that  data quality may have on implementation of the Enforcement Response  Policy. We also interviewed EPA and state officials to obtain their views  on the matter.
Second, we examined the impact data quality could have on the agency’s  ability to inform the public and Congress about water systems’ compliance  with drinking water standards. In particular, we examined whether EPA’s  claims about community water systems’ performance relative to  Government Performance and Reporting Act (GPRA) goals were affected  by the use of incomplete and inaccurate SDWIS/Fed data. Two key GPRA  measures are the percentage of community water systems and the  percentage of population served by those systems that meet all health- based drinking water standards (i.e., had no health-based violations) in a  fiscal year. We used EPA’s data verification audit data to estimate the  percentage of community water systems that met all health-based drinking  water standards for selected states within each audit year. Based on our  review of EPA’s data verification audit sample design, we determined that  the sample was not designed to produce reliable estimates of the  percentage of the population served by systems with health-based  violations. Therefore, we focused our analysis on the percentage of  community water systems that met all health-based drinking water  standards.
To estimate the percentage of community water systems that met all  health-based drinking water standards from the data verification audit  data, we counted the number of health-based violations for each  community water system in the sample and calculated point estimates and  95 percent confidence intervals of the percentage of systems that met all  health-based drinking water standards after accounting for the results of  the data validation audits.
To examine the actions EPA and the states have been taking to improve  the quality of data in SDWIS/Fed, we interviewed EPA officials and  obtained and reviewed documentation on steps the agency has taken, or is  considering, to modernize SDWIS and improve data quality. We also  examined survey respondents’ views on steps that EPA and the states  could take to address data quality—including the adoption of particular  data management tools—and ways in which the three EPA-state work  groups could be more effective. For more information on our survey and  content analysis, see appendix II.
We conducted this performance audit from February 2010 through June  2011 in accordance with generally accepted auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient  and appropriate evidence to provide a reasonable basis for our findings  and conclusions based on our audit objectives. We believe that the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Survey Methodology and Analysis
subsections: section_title: Selection of Survey Respondents
paragraphs: To obtain the views of knowledgeable EPA and state drinking water  officials about SDWIS data management and data quality, we surveyed the  members of three EPA-state work groups that were formed to address  various aspects of drinking water data: the Data Management Steering  Committee, the Data Technical Advisory Committee, and the Data Quality  Work Group. EPA and the Association of State Drinking Water  Administrators provided us with the names and e-mail addresses of the 46  members of these groups. The work group members come from EPA  headquarters, EPA regions, states, and the Association of State Drinking  Water Administrators. The Data Management Steering Committee had 18  members, the Data Technical Advisory Committee had 15 members, and  the Data Quality Work Group had 25 members. Nine officials served on  more than one of the committees.
section_title: Survey Design and Pretesting
paragraphs: Our survey asked a range of questions related to the drinking water  violations data that states and other primacy agencies provide to EPA. We  asked the respondents to comment on the factors that have contributed to  data errors, the steps that should be taken to correct those errors, the  impact that data errors could have on EPA’s Enforcement Response  Policy, and various data management tools. We also asked the respondents  to evaluate the effectiveness of the work group or groups on which they  served.
The practical difficulties of conducting any survey may introduce errors,  commonly referred to as nonsampling errors. For example, respondents  may have difficulty in interpreting a particular question or may lack  information necessary to provide valid and reliable responses. In order to  minimize these errors, we conducted pretests of the draft survey with one  EPA headquarters official, one EPA regional official, and one state official  by telephone. The Chief of the Infrastructure Branch of EPA’s Office of  Ground Water and Drinking Water suggested that we conduct the pretest  with those individuals. During these pretests, we checked whether (1)  questions were clear and unambiguous, (2) terminology was used  correctly, (3) the questionnaire did not place undue burden on  respondents, (4) the information could feasibly be obtained, and (5) the  survey was comprehensive and unbiased. In addition, the survey was peer  reviewed by a GAO senior survey methodologist. We made changes to the  content and the format of the survey based on the feedback we received.
Survey Administration  We administered our survey in August and September of 2010. We first  phoned each work group member to alert them to our plan to send the  survey and to request their participation. Through those phone calls, we  learned that one of the work group members had retired and another had  transferred to a different position and was no longer a member of a work  group. As a result, our survey population decreased to 44.
Prior to fielding the survey, we sent an e-mail to each member of the work  groups to further explain its purpose. We notified work group members  electronically when the survey was available, and sent e-mail reminders  prior to our requested deadline of September 13, 2010. We also made  phone calls to several survey recipients during the extension period to  request their participation. In total we received 41 completed surveys.  However, the three work group members from the Association of State  Drinking Water Administrators collaborated to prepare one response and  two EPA regional officials collaborated to prepare one response.  Therefore, the 41 completed surveys represent the views of all 44 members  of the survey population.
section_title: Data Analysis
paragraphs: The survey contained closed-ended questions that asked respondents to  select from a finite number of options. For example, some questions asked  respondents to select “Yes,” “No,” or “Don’t Know.” Others asked  respondents to select from a list of factors that may have contributed to  drinking water violation data errors. Our analysis of the responses to these  questions simply involved counting the number of responses for each  option. In the report, there are instances in which we identify all of the  responses and other instances in which we identify the most common  response.
We also asked respondents to evaluate the effectiveness of certain data  management tools or the EPA-state work groups. For these questions, we  offered the respondents a four-point range of answers: very effective,  moderately effective, slightly effective, and not effective. Respondents  could also answer “Don’t Know” to these questions. In the report, we  sometimes identified the most common response while in other instances  we combined the number of “moderately effective” and “slightly effective”  responses because each was relatively common.
Several survey questions asked for opinions on the creation of a goal for  the percentage of monitoring violations that are completely and accurately  reported to SDWIS/Fed. We asked those respondents who supported the  idea of having such a goal to state what they think the percentage should  be. We summed those percentages and divided by the number of  respondents who answered that question to arrive at an average.
The responses to the closed-ended questions are provided in this  appendix.
section_title: Content Analysis
paragraphs: The survey also contained open-ended questions that asked respondents  to provide a narrative response. In order to succinctly summarize the  open-ended responses, we performed a content analysis in which we  grouped the responses into a coding structure that represented common  themes. We decided that the responses to each open-ended question  would have a coding structure with two dimensions. To explain this, it is  useful to discuss the link between closed-ended and open-ended  questions. For example, one closed-ended question asked the respondents  to select a factor—such as training by EPA or the states or guidance from  EPA—that they believe has contributed to compliance determination  errors. The subsequent open-ended question asked them to elaborate on  why they thought the factor or factors they selected have contributed to  compliance determination errors. Note that the respondents did not  necessarily elaborate on each of the factors they selected in the prior  question. As part of our content analysis, we sought to first identify the  first dimension code for the response. In this example, the first dimension  codes mirrored the factors (e.g., training or guidance) that the respondent  selected to write about. A second dimension code provided a more  detailed description of what the respondent said about the first order  code. For example, second dimension codes for that question included  amount or quality, timing, and targeting.
A team of three GAO analysts jointly reviewed several completed surveys  to develop an initial draft of a structure for coding the open-ended  responses. To further identify meaningful first and second dimension  codes for the coding structure, the three GAO analysts independently  reviewed the open-ended responses for four completed surveys. Each  analyst made a judgment about appropriate codes that described the  themes in the open-ended responses. The analysts compared their  decisions and reconciled any disagreements regarding appropriate codes  by refining the criteria used to categorize the responses.
After the team agreed upon the coding structure, it continued its analysis  of the responses to the closed-ended questions. The three GAO analysts  were each assigned to independently review the responses to specific sets  of questions. For example, Analyst A and Analyst B independently  reviewed and coded the open-ended answers to questions 1 through 4.  Analysts A and B then compared their coding decisions and reconciled any  disagreements. If they could not reconcile a disagreement, Analyst C was  consulted to achieve agreement. The three analysts rotated assignments so  that each performed the role of “tiebreaker” when the other two could not  agree on a coding decision.
subsections: section_title: Respondents’ Answers to Closed-Ended Survey Questions
paragraphs: Our survey of EPA and state drinking water officials contained numerous  closed-ended questions about various data management issues. Table 2  presents those questions and the respondents’ answers.
section_title: Coding Decisions for Responses to Open-Ended Survey Questions
paragraphs: Many of the questions in our survey of EPA and state drinking water  officials asked for open-ended responses. We developed several “coding  schemes” to categorize the responses to those questions. Our schemes  generally had first and second dimension codes. For example, question 2  asked the respondents to explain how factors they selected contributed to  compliance determination errors. The first dimension codes for that  question included information system structure, training, funding, staffing,  and guidance, among others. The second dimension codes included  amount; targeting; timeliness; quality, complexity, or ease of use;  automation; and others. A response to question 2 might have included a  comment that related to information system structures and, more  specifically, a comment about the quality, complexity, or ease of use of  information systems. In that situation, we would have coded the response  as falling into those first and second dimension codes. Because some sets  of questions generated answers that could be similarly coded, we used  some coding schemes for multiple questions. On the other hand, we used  some schemes for only one question.
section_title: Appendix III: Comments from the Environmental Protection Agency
paragraphs: The following are GAO’s comments responding to the comments in  Appendix A of the Environmental Protection Agency’s letter dated   June 8, 2011.
subsections: section_title: GAO Comments
paragraphs: 1.  EPA commented that there are over 152,000 public water systems in  the United States that are also subject to the requirements of the  National Primary Drinking Water Regulations. We do not disagree, but  did not modify the report in response to EPA’s comment. Footnote 11  of the report describes the universe of public water systems.  2.  EPA said that violations that are reported as “other” should not be  included with monitoring and reporting violations and that it would be  beneficial to explain how reporting violations differ from monitoring  violations. Our analysis is consistent with EPA’s 2008 analysis of data  quality which also combined “other” violations, such as violations of  consumer confidence reporting and public notification requirements— with monitoring and reporting violations. Therefore, we did not modify  our analysis or the report in response to EPA’s comment. We also did  not modify the background section to further explain the difference  between monitoring and reporting violations because we provide  examples of different violations in a subsequent section of the report.  3.  EPA requested that we provide a footnote in the report describing new  performance indicators for small water systems. Specifically, EPA  requested that we explain the agency’s intent behind the indicators and  its plan to evaluate their utility. We did not modify the report in  response to this comment since we describe two of these indicators in  the section of the report that addresses EPA’s ability to monitor and  report progress toward its strategic objective of reducing exposure to  contaminants in drinking water.  4.  EPA said it was unsure of the basis for our statement that monitoring  violations are predictors of health-based violations. EPA also noted  some of the variations between monitoring and reporting violations  and asked for more details regarding our analysis. Our statement was  based on aggregate regression analyses (negative binomial and zero- inflated Poisson models) with limited controls. It does not take into  account which type of monitoring and reporting violation occurred,  and cannot differentiate between lack of monitoring and monitoring  that was not reported or was delivered late. The regression was  intended to illustrate the link between overall counts of monitoring  and reporting violations and counts of health-based violations, and  does not provide insight into the nature of the link or the reasons that  monitoring and reporting violations might condition the number of  health-based violations. We realize the implications of our statement  are limited, but believe the correlation between overall counts of  monitoring violations and health-based violations offers useful insight.  5.  EPA requested that we present an analysis of the quality of violations  data for the Lead and Copper Rule separately from other National  Primary Drinking Water Regulations, as it has done. We acknowledge  that there may be value in conducting data quality analyses for specific  drinking water regulations, as EPA did, for example, in its 2008 report  on data quality. That report showed that the data quality for the Lead  and Copper Rule was lower than for other types of drinking water  regulation. However, because the data verification audit data we  analyzed was from a sample of community water systems from a  sample of states, our results included margins of error. Analyzing data  quality for particular drinking water regulations results in larger  margins of error than analyzing date quality for all health-based  violations. In light of that circumstance, we decided to conduct our  analysis of all health-based violations. We did not modify our analysis  or report in response to this comment.  6.  EPA commented that it intends to conduct six to eight data verification  audits in calendar year 2011. We have modified the report to reflect  that comment, but note that EPA’s statement concerns audits it has yet  to conduct. EPA also said that it would appreciate our including any  specific suggestions made by survey respondents on how the data  verification audits can be improved. We have added a footnote with  examples of comments from survey respondents.  7.  EPA requested that we update the report to reflect the status of work  done by the Office of Ground Water and Drinking Water to assess  regional responses and document the extent to which the regions had  complied with requests to discuss data quality with states. EPA went  on to say it understands that all the regions include data quality as an  issue for discussions with their states and as part of their grant  agreements or state work plans. We have modified the report to  include EPA’s statements. However, we were not able to verify the  accuracy of those statements because EPA did not have supporting  documentation.  8.  EPA said that it agrees that the regions did not fully support new  measures for tracking regional oversight suggested by headquarters  but that the lack of those measures has not prevented the regions or  headquarters from continuing to work with the states to address data  quality challenges. We note that the proposed performance measures  would assist the Office of Ground Water and Drinking Water to  monitor the regions’ oversight of the states, not the states’  performance. We assume that EPA headquarters proposed these  performance measures because it thought they would help encourage  the regions to increase their oversight. Without them, EPA  headquarters may find it more difficult to oversee the regions. We did  not modify the report in response to this comment.
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Diane B. Raynes, Assistant  Director; James Ashley; Elizabeth Beardsley; Mark Braza; Ross Campbell;  Anna Maria Ortiz; Carla D. Rojas Paz; Kelly Rubin; Jerome Sandau; Jeffrey  Sanders; Carol Herrnstadt Shulman; and Vasiliki Theodoropoulos made  significant contributions to this report. Robert Alarapon, Mick Ray, and  Lisa Vojta also made important contributions to this report.