Hazardous Waste: Benefits of EPA's Information System Are Limited
Released: 1995-09-28T00:00:00

Letter

Background

Congress enacted RCRA to protect human health and the environment and to conserve energy and material resources. In implementing RCRA, EPA is responsible for developing federal hazardous waste management and disposal requirements and for ensuring compliance with these requirements. Two offices within EPA share responsibility for RCRA—the Office of Solid Waste (OSW) and the Office of Enforcement and Compliance Assurance.

EPA may authorize states to implement their own RCRA programs, or portions thereof, provided that their programs meet federal requirements. Currently, 46 states, the District of Columbia, and Guam, have been authorized to implement at least some portion of RCRA. States that are authorized receive annual grants from EPA and are subject to EPA oversight. In fiscal year 1994, EPA gave approximately $92 million in RCRA grant money to these states and territories.

To help manage RCRA, EPA and authorized states collect and manage data on more than 250,000 hazardous waste handlers, including 210,900 small quantity generators; 15,300 large quantity generators; 19,700 transporters; and 4,300 treatment, storage, and disposal facilities. Day-to-day RCRA activities include assigning hazardous waste handler identification numbers; issuing permits; tracking inspections, violations, and enforcement actions; and maintaining other related data to assess the extent to which facilities are complying with RCRA regulations.

In 1985, to overcome problems with the information system it was using, known as the Hazardous Waste Data Management System, EPA began developing RCRIS. Both the states and EPA had reported that the Hazardous Waste Data Management System was not adequately supporting the program in many areas, including permitting, compliance monitoring, and enforcement. In addition, the system was criticized as cumbersome, difficult to use, and confusing, and the database was criticized as excessively complex.

The overall goal of RCRIS was to help both the federal government and the states better manage the hazardous waste program. The National Governors’ Association assisted EPA in identifying state needs for RCRIS. EPA determined, based on information collected by the Association, that for RCRIS to be successful, the system had to meet the following four key objectives: (1) provide individual databases for each state, (2) be user friendly, (3) be flexible enough to meet federal requirements, as well as the specific needs of individual states, and (4) provide a mechanism for maintaining highly reliable data.

RCRIS became operational nationwide in 1991. The system comprises four key modules that mirror the major RCRA program activities. Table 1 provides the name and purpose of these modules.

Although OSW did not maintain information about RCRIS operational costs, a recent report by the EPA Inspector General stated that fiscal year 1993 operational costs were $7.5 million. This figure, however, excludes EPA’s and states’ personnel costs for time spent (1) entering, retrieving, and manipulating data in RCRIS and (2) developing and operating other systems and work-arounds to compensate for RCRIS inadequacies. Officials in OSW said they did not know these costs.

Scope and Methodology

Our work was performed at several offices at EPA headquarters, including the Office of Solid Waste, the Office of Enforcement and Compliance Assurance, and Office of Inspector General. These offices were located in Washington, D.C., and Arlington, Virginia. We also worked at EPA regional offices in Atlanta, Georgia; Dallas, Texas; and San Francisco, California; and at state offices responsible for RCRA in Tallahassee, Florida; Jackson, Mississippi; Little Rock, Arkansas; Austin, Texas; Carson City, Nevada; and Sacramento, California.

We conducted our review from August 1994 through May 1995, in accordance with generally accepted government auditing standards. We requested comments on a draft of this report from the Administrator of EPA. The Director for the Office of Solid Waste provided us with comments on July 18, 1995. We have incorporated these comments where appropriate.

RCRIS Is Not Meeting User Needs

RCRIS falls short of meeting its overall goal of helping EPA and the states manage the hazardous waste program because it has not met three of its primary objectives. While RCRIS provides individual databases for states to use to maintain their data, the system is difficult to use, difficult to tailor to meet individual state and EPA regional needs, and does not provide a mechanism for maintaining highly reliable data.

States Can Maintain Their Own RCRIS Databases

RCRIS provides two databases for each state—one maintained by the state and one maintained by the EPA regional office. These databases are intended to support day-to-day RCRA activities carried out by the states and EPA regional offices. Extracts from these databases are merged to construct a database of required information for regional oversight. A national oversight database for EPA headquarters and national reporting is then formed by merging extracts from the regional merged databases.

RCRIS Is Difficult to Use

RCRIS is not easy to use, particularly with regard to data entry and retrieval. Data entry is cumbersome largely because of the design of the four key RCRIS modules. For example, each module uses numerous data entry screens, many of which allow the user to modify data in only a few data fields at one time. The RCRIS user manual shows that a user could possibly interact with up to 150 data entry screens to add, modify, or delete data.

Because of this design, users are often required to switch between modules, screens, and files—even when making simple changes to the data. For example, to modify a hazardous waste handler’s information and add data for an inspection of a specific facility, a user must access the handler identification module, enter the handler’s identification number, update the handler information, exit the handler module, enter the compliance monitoring and enforcement module, reenter the handler’s identification number, and then enter the inspection information.

Data entry is also difficult because codes are used extensively to represent much of the information that is recorded. These codes are used to describe various aspects of a hazardous waste handler, such as the process code that specifies the current waste treatment, storage, or disposal process being used. RCRIS’ four key modules include about 875 codes. The RCRIS user manual notes that understanding the majority of the codes is essential for successful data entry, maintenance, processing, and reporting.

Users have also found data retrieval difficult. While users can obtain several standard reports from the system, these reports are voluminous. For example, up to four cardboard boxes are needed to hold one standard report on facilities. Consequently, users need to query the system to obtain specific information, but creating customized reports is very difficult. For example, to obtain information on a specific facility’s history or status directly from the system, a user must first know which files contain data on the facility. The user must then query each file to obtain the desired data. Finally, the user must know how to merge the extracted data to produce a single report. Users in the six states and the three EPA regions we visited said that creating these customized reports is very difficult.

RCRIS Does Not Satisfy Individual State and EPA Regional Needs

Another primary objective for RCRIS was that it be flexible enough to allow states and EPA regions to tailor the system to meet their individual program needs. However, users in five of the six state offices and all three of the regional offices we visited stated that RCRIS has not met their individual needs. Many of these users said they have no desire to tailor RCRIS to meet their needs because the system is just too difficult to use.

We identified users in two EPA regions who said they had been able to better meet some individual needs using RCRIS. For example, because RCRIS’ standard reports do not include the precise information needed, these users developed customized query programs and reports, and have made them available to other users across the country. These users said they were able to do this because they invested a significant amount of time in learning the intricacies of the system and had become expert in using it.

In addition, because almost all the states were using or developing their own systems, EPA wanted them to be able to extract the required data from their systems and submit it to RCRIS electronically. Although states can do this, state users told us that creating conversion software to reformat their data into RCRIS’ data structure and then modifying the conversion software each time EPA changes RCRIS’ data structure can be extremely difficult. Consequently, only one state transmits any data to RCRIS electronically.

RCRIS Data Are Not Reliable

RCRIS does not provide a mechanism for maintaining highly reliable data. While EPA does not require data quality reviews of the data in RCRIS, the few ad hoc reviews that have been conducted have uncovered significant problems. For example, a Region VI (Dallas) review of facility demographic data on about 45,000 hazardous waste handlers identified over 15,000 errors on the location of the handlers. Similarly, Region IV (Atlanta) found anecdotal evidence of discrepancies between the data in RCRIS and the information in hard copy files. According to EPA officials, states have corrected most of these discrepancies. However, users in all regional and state offices we visited consistently expressed concerns about the quality of RCRIS data.

In addition, as part of a review of whether land disposal facilities were complying with groundwater monitoring requirements, we found RCRIS data to be unreliable. We selected a random sample of 30 facilities out of RCRIS’ universe of 1,427 classified by EPA as land disposal facilities. For each sampled facility, we reviewed data in 14 data fields that were most relevant to groundwater monitoring requirements. We estimated minimum error rates ranging from 8 to 37 percent for eight of the data fields. These data provided information on the types of violations detected and enforcement actions taken.

These data reliability problems can be attributed, in part, to a lack of controls over the quality of data when EPA converted from the predecessor system to RCRIS. Although EPA recognized that the conversion would result in inaccurate data, it did not require verification and correction of the data—a generally accepted practice. Instead, EPA loaded the inaccurate data and then left it up to each state and region to perform a post-conversion assessment of data quality. Further, although EPA (1) provided software to help the states and regions detect data quality errors after the conversion, (2) set aside 5 percent of state grant funding to support data cleanups, and (3) set a 90-percent accuracy goal for the converted data, the agency allowed users to choose which data they would check and did not determine whether states and regions had met the accuracy goal. Thus, officials in all three regions we visited had concerns about lingering data quality problems because of unverified data from the predecessor system.

A second factor contributing to data quality problems is EPA headquarters’ failure to provide precise definitions for certain data elements in the system. Officials in all regions and five of the six states we visited said that RCRA data definitions are interpreted differently and have led to differences among states in how data in the same fields are recorded in RCRIS. For example, while EPA requires users to identify facilities that burn hazardous waste, the agency does not specify whether only a facility that burns hazardous waste as a regular part of its operations should be considered a burn facility, or if all facilities that have ever burned waste, regardless of frequency, should be included. Headquarters officials agreed that this is a critical problem and acknowledged that no one has taken control of data standardization issues.

Another factor contributing to unreliable data is that RCRIS’ front-end edit checks do not effectively help ensure that data are accurate when they are entered. They do not check whether data that are entered are consistent with other related data fields. For example, while violations typically result from an inspection, violation data can be entered into the RCRIS violation file even if the inspection data were never entered because users are not prompted to enter the related inspection. Therefore, incomplete information may remain in the system.

Finally, EPA does not have a data quality assurance program that would include establishing reliability standards and require periodic reviews of RCRIS data to determine if they meet these standards. EPA headquarters officials said they do not require regular data quality reviews because they believe that it is the regions’ and states’ responsibility to ensure data in the system are reliable. EPA officials in the three regions we visited told us that they use a RCRIS software feature to verify certain data, but that overall data quality reviews have only been done on an ad hoc basis, and that these reviews had focused on limited data elements.

RCRIS Shortfalls Have Not Significantly Affected the RCRA Program

Although RCRIS has not met its original objectives, system problems have not significantly affected the RCRA program because users do not rely on it as a primary management tool. However, RCRIS’ problems have placed additional burdens on EPA and the states to develop work arounds and additional systems. All of the regions and states we visited were using at least one PC-based system to supplement RCRIS. In some cases, these systems did not address unique needs, but rather duplicated the same functions as RCRIS in a format that was easier to use.

EPA Officials Do Not Rely on RCRIS Data

While EPA officials stated that RCRIS has improved some aspects of systems support, officials in all three regions we visited and headquarters, including the Director of the Office of Solid Waste, told us that the system is not reliable enough to strategically manage the RCRA program. Some data in RCRIS are used for program planning, state grant workload models, and regulatory development. However, rather than relying on information directly out of the system, EPA headquarters officials said they routinely send national-level RCRIS reports back to regional offices and the states for subsequent verification and correction before they accept and use the data. EPA program officials and staff also told us that RCRIS does not provide all the information they need. For example, these officials pointed out that universe information—the total number of hazardous waste handlers in particular classes—cannot be obtained from RCRIS. As a result of these problems, the Director stated that he believes it is necessary to rethink RCRA’s information needs, particularly with regard to data needed for environmental and management indicators.

States Use Their Individual Systems

All six of the state locations we visited use their own systems to help them meet their information needs. State officials said they enter data into RCRIS primarily to satisfy their RCRA grant commitments, which require them to devote specific resources to supporting RCRIS and entering required data into the system. The officials also said they maintain their own systems because these systems are easier to use and they meet the state’s individual needs much better than RCRIS. For example, California maintains separate databases for managing its regulated universe of hazardous waste handlers because its data needs are more detailed and broader in scope than EPA’s.

EPA Efforts to Address Shortfalls

During our review, EPA acknowledged RCRIS’ many problems. Further, OSW admits that it does not know who the system’s users are or how frequently the system is used. In addition, as previously noted, EPA does not know how much money is being spent on RCRIS field support or on developing and maintaining supplemental systems.

Recognizing RCRIS’ shortcomings, OSW has begun a new strategic effort—the Waste Information Needs (WIN) initiative—to determine the information and technology needs of EPA headquarters, EPA regions, and states. EPA intends to produce a plan addressing these needs by September 1995. In the interim, EPA plans to continue operating RCRIS even though (1) most states we visited do not use it to meet their needs and (2) EPA acknowledged that RCRIS data are unreliable.

Conclusions

RCRIS is providing few of the benefits that were originally intended. The system is difficult to use and its data are unreliable. Consequently, many of the users RCRIS was created to serve do not rely on the system or the information in it. Instead, states operate their own systems, and EPA uses other means to get reliable data. Until EPA identifies what information it needs to manage RCRA, EPA will continue to operate a national system with unknown costs, few benefits, and documented burdens.

Recommendations

We recommend, as a part of the recently initiated reassessment of RCRA information needs, that the Administrator of the Environmental Protection Agency require the Assistant Administrator of the Office of Solid Waste and Emergency Response to determine what information EPA needs to oversee states’ implementation develop clear data definitions for all EPA required data, develop a data quality assurance program that establishes data reliability standards and methods to ensure data reliability, and develop and implement a cost-effective solution for meeting identified needs.

In the interim, given the burden RCRIS imposes on the states and the lack of EPA’s reliance on the system, we recommend that the Administrator of the Environmental Protection Agency direct the Assistant Administrator of the Office of Solid Waste and Emergency Response to assess RCRIS data reporting requirements and eliminate those that are identified as nonessential.

Agency Comments and Our Evaluation

In commenting on a draft of this report, the Director for the Office of Solid Waste stated that EPA generally agreed with our analyses, findings, and recommendations. However, the Director requested clarification on two of our recommendations.

First, our draft report proposed, as part of the WIN initiative, that EPA define the information needs of the states that require system support from EPA to manage their programs. The Director stated that EPA did not know if we were recommending that the agency only define information needs or if we were recommending that it also supply system support (development and/or operational) to those states. The Director said EPA realizes that it cannot build one automated system to support all users and that EPA wants to get out of the business of supplying system support to the states.

We made this proposal because of our concern that in conducting the WIN initiative, EPA would again gather information requirements from state and EPA users, and then develop a system that it hoped would satisfy all of these requirements. Our intent was for EPA to explicitly identify which states require systems support in order to narrow the requirements for the new system to only those users who will rely on the system. We have deleted our recommendation based on the Director’s recognition that EPA cannot build one system to meet federal and state needs; however, we caution EPA as it proceeds with the WIN initiative to determine and make clear to the states specifically what level of system support will be provided.

Secondly, we proposed that EPA consider eliminating, while it completed the WIN initiative, the requirement for states to supply data to RCRIS. The Director stated that EPA realizes that the burdens associated with maintaining RCRIS need to be reduced, but noted that EPA relies on such data for program implementation and management. EPA did agree to consider eliminating, or significantly reducing, the requirement for states to supply data to RCRIS.

We did not intend for EPA to take an all-or-nothing approach to RCRIS. In EPA’s consideration of the option, we expected that certain data elements would be identified as essential while others could be eliminated. We have modified our recommendation to reflect this.

As arranged with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the date of this letter. At that time, we will send copies to the Administrator of the Environmental Protection Agency, Director of the Office of Management and Budget, and interested congressional committees. Copies will also be made available to others upon request.

Please call me at (202) 512-6253 if you or your staff have any questions concerning this report. Other major contributors are listed in appendix III.

Scope and Methodology

To address our objectives, we used a structured interview document to discuss use of the system and its information with regional and state users. We interviewed officials and staff at EPA regional offices in Dallas, Texas; Atlanta, Georgia; and San Francisco, California. We chose these locations because they are three of the largest RCRA regions responsible for 17 states and 3 territories, and because headquarters officials believed they would adequately represent the diversity in use of RCRIS.

We also interviewed representatives of six state environmental protection offices—the largest in each of the regions—Texas, Florida, and California—and the smallest—Arkansas, Mississippi, and Nevada. At each location, we met with the RCRIS database administrators and other system staff, as well as officials who were responsible for managing and implementing the RCRA program. We also reviewed numerous documents related to RCRIS use, operation, and output.

We reviewed detailed information on the RCRA program, EPA’s previous attempts to collect hazardous waste data through automation, and the development and operation history of RCRIS; and examined prior GAO and EPA Inspector General reports that discussed weaknesses in EPA’s information systems environment. We interviewed EPA headquarters officials from the Office of Solid Waste and the Office of Enforcement and Compliance Assurance. These officials included the Director of the Office of Solid Waste, the RCRIS system manager, and staff involved in RCRA and RCRIS activities. We also interviewed representatives from the National Governor’s Association and reviewed documents pertaining to its involvement in assisting EPA to identify the objectives and requirements in RCRIS.

In addition, we used relevant segments of the information systems audit methodology published by the EDP Auditors Foundation to evaluate the extent to which RCRIS’ development and operation met generally accepted systems practices. We reviewed and analyzed documents on EPA’s actions, including documentation on user requirements, system feasibility, functional requirements, change control procedures, and data management. We also reviewed EPA planning documents including information on an Office of Solid Waste strategic initiative.

Our work was performed at several offices at EPA headquarters including the Office of Solid Waste, the Office of Enforcement and Compliance Assurance, and Office of Inspector General. These offices were located in Washington, D.C., and Arlington, Virginia. We also worked at EPA regional offices in Atlanta, Georgia; Dallas, Texas; and San Francisco, California; and at state offices responsible for RCRA in Tallahassee, Florida; Jackson, Mississippi; Little Rock, Arkansas; Austin, Texas; Carson City, Nevada; and Sacramento, California.

We conducted our review from August 1994 through May 1995, in accordance with generally accepted government auditing standards. We requested comments on a draft of this report from the Administrator of EPA. The Director for the Office of Solid Waste provided us with comments on July 18, 1995. We have incorporated these comments where appropriate.

Error Rates for 14 Selected Resource Conservation and Recovery Information System Data Elements

Major Contributors to This Report

Accounting and Information Management Division, Washington, D.C.

Dallas Regional Office, Dallas, Texas

William H. Thompson, Evaluator-in-Charge Luis Escalante, Jr., Senior Evaluator Shannon Q. Cross, Senior Computer Specialist The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (301) 258-4097 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

Address Correction Requested