Agency for Toxic Substances and Disease Registry: Policies and Procedures for Public Health Product Preparation Should Be Strengthened
Released: 2010-05-20T00:00:00

Letter

Background

ATSDR investigates community exposures related to chemical sites and releases; works with federal, tribal, state, and local agencies to identify potential exposures; assesses associated health effects; and recommends actions to stop, prevent, or minimize these harmful effects, among other things.

ATSDR History

ATSDR was established by the Comprehensive Environmental Response, Compensation, and Liability Act of 1980 (CERCLA), which created what is known as the Superfund program to clean up the nation’s most dangerous hazardous waste sites. CERCLA established ATSDR to carry out Superfund’s health-related activities, including the establishment of a national registry of and the provision of medical care and testing to persons exposed to toxic substances and the provision of survey and screening programs to determine the relationship between such exposure and illness. In 1985, ATSDR was formally organized to begin to carry out its responsibilities under the law. The Superfund Amendments and Reauthorization Act of 1986 (SARA) broadened ATSDR’s responsibilities to include, among other things, public health assessments, establishment and maintenance of toxicologic databases, information dissemination, and health education. SARA required that ATSDR conduct a public health assessment at each site proposed for or on the National Priorities List (NPL), and authorized ATSDR to perform public health assessments upon petition by an individual or physician and to conduct additional follow-up health studies if needed.

Sources of ATSDR Work and Types of Public Health Products

ATSDR may initiate work to prepare public health products for several reasons. Work can be necessitated pursuant to SARA by a site’s proposal to or listing on the NPL, requested by an ATSDR partner such as EPA, negotiated as part of a work plan for federal facilities, petitioned by individuals or physicians, or generated internally by ATSDR officials. Once work is initiated, ATSDR may prepare any of several different types of products, including the following: Public health assessments evaluate data and information on the release of hazardous substances into the environment in order to assess any past, current, or future impact on public health, develop health advisories or other recommendations, and identify studies or actions needed to evaluate and mitigate or prevent human health effects.

Health consultations review available information or collect new data to respond to a specific health question or request for information about a potential environmental hazard. Health consultations are focused on a specific exposure issue and provide guidance on the specific health- related question.

Health studies are epidemiological research conducted to investigate and characterize the association between exposure to chemicals in the environment and health problems of people who have been exposed to chemicals.

Exposure investigations collect and analyze site-specific environmental or biological samples to determine whether individuals have been exposed to hazardous substances. Exposure investigations are often designed to examine individuals most likely to be exposed to hazardous substances, rather than a sample of individuals from the exposed community that would provide information about the community as a whole.

The time required to complete ATSDR public health products varies, and may depend on the nature and complexity of the work site. For example, some public health assessments and most health studies take one or more years to complete, whereas some health consultations are completed within weeks. ATSDR also prepares emergency response products—most of which are completed within hours or days—which are intended to help interpret the implications of exposure data. These “real-time” investigations of health exposures include technical assistance and health consultations, and are often initiated in response to requests from agencies such as EPA or state health or environmental departments.

ATSDR Organizational Structure

Although ATSDR was established within the Department of Health and Human Service’s (HHS) Public Health Service, the Director of CDC serves as the Administrator of ATSDR, and CDC performs many administrative functions for ATSDR, such as human capital and financial management services. ATSDR is located within CDC’s Office of Noncommunicable Diseases, Injury and Environmental Health. In 2003, ATSDR’s administrative functions were combined with those of CDC’s National Center for Environmental Health (NCEH). ATSDR and NCEH share an Office of the Director (OD), which is led by a director and deputy director. ATSDR has four divisions, each of which is divided into either branches, programs, or regional offices (see fig. 1). Each division is led by a director and deputy director, and DHAC, DHS, and DTEM each also have an associate director for science. Each branch or program within a division is led by a chief.

OD—As shown in figure 1, the OD has eight functional areas: communication science; community environmental health; financial and administrative services; information systems; policy, planning and evaluation; program development; science; and terrorism preparedness and emergency response. These functional areas are responsible for providing scientific and programmatic support for agency staff and conducting review and clearance for public health products produced by ATSDR divisions. Specifically, the Office of Science is responsible for the clearance, cross-clearance, and external peer review of ATSDR public health products. The Office of Science also coordinates the NCEH/ATSDR Board of Scientific Counselors, which provides advice and guidance to ATSDR’s director on external peer review of ATSDR programs and issues including program goals, objectives, strategies, and priorities. The board’s advice and guidance are intended to assist ATSDR in ensuring scientific quality, timeliness, utility, and dissemination of scientific results.

DHAC—DHAC produces a number of products, including public health assessments, health consultations, and exposure investigations. DHAC’s Cooperative Agreement and Program Evaluation Branch is charged with supporting and overseeing the work produced by ATSDR’s cooperative agreement partners, which currently include 29 state agencies and 1 tribal government (see fig. 2). In order to become a cooperative agreement partner, state and tribal governments must respond to a request for applications that ATSDR posts, and have their application reviewed, scored, and funded by ATSDR. Currently, ATSDR funds cooperative agreement partners for a 5-year funding cycle. Through these partnerships, ATSDR provides funding and technical support for state and tribal government employees to assess environmental health concerns at sites within their jurisdictions and to conduct needed public health interventions. Cooperative agreement partners prepare public health products that are monitored, reviewed, and cleared by ATSDR. DHAC staff work with staff in regional offices to provide technical assistance to cooperative agreement partners and the public, and to sponsor activities in communities that have been exposed to hazardous chemicals.

DHS—DHS is responsible for conducting epidemiologic health studies, designing and conducting surveillance programs, and establishing and maintaining registries. The division collects information to determine whether a chemical exposure is making people sick, and collects data on persons identified as having been exposed to a specific contaminant or event. DHS is also involved with the ongoing collection, analysis, and interpretation of health data. These products are released as ATSDR reports and are often published in the scientific literature.

DRO—DRO staffs an ATSDR regional office within each of the 10 EPA regions, in EPA Headquarters in Washington, D.C., and in two satellite offices in Helena, Montana, and Anchorage, Alaska (see fig. 2). These offices are responsible for acting as regional liaisons to establish working relationships with EPA, other federal and state agencies, individual citizens, and community groups to maintain current and historic knowledge of issues related to hazardous chemical sites in their regions. Working in collaboration with DHAC, DRO staff also prepare a number of time-sensitive products, such as health consultations.

DTEM—DTEM assists in the production of health consultations and provides technical assistance in response to chemical spills and acute events. DTEM staff also work closely with DRO staff to provide real-time public health advice in case of a chemical release. DTEM is responsible serving as a resource for information and assistance on toxic substances in the environment, and prepares toxicological profiles for hazardous substances.

Internal Control

The Standards for Internal Control in the Federal Government provides the overall framework for establishing guidelines for internal control, and helps government managers achieve desired objectives. As applied to ATSDR, this could include the preparation of quality public health products. Internal control, which is synonymous with management control, comprises the plans, methods, and procedures used to meet missions, goals, and objectives. The related Internal Control Management and Evaluation Tool assists agencies in maintaining or implementing effective control. Internal control is not one event, but a series of actions and activities that occur throughout an entity’s operations and on an ongoing basis. The responsibility of good internal control rests with all managers; they set the objectives, put the control mechanisms and activities in place, and monitor and evaluate these mechanisms and activities. However, all employees in the organization play important roles in this process. Internal control includes activities such as establishing policies and procedures, assessing risks associated with achieving agen objectives, ensuring effective information sharing throughout the organization, conducting ongoing monitoring of agency activities, and establishing key areas of authority and responsibility for agency management and staff.

The Standards note, however, that while internal control helps government managers achieve desired objectives, it cannot provide absolute assurance that all agency objectives will be met. There are many factors outside the control and influence of management that can affect agency’s ability to achieve its objectives. For example, human mistakes, judgment errors, and acts of collusion to circumvent control can affect meeting agency objectives. Therefore, once in place, internal control provides reasonable assurance, not absolute assurance, that an agency’s objectives are being achieved.

ATSDR’s Policies and Procedures for P Health Product Preparation Lack Some Critical Controls to P Reasonable Assurance of Quality

The policies and procedures that ATSDR has established for public health product preparation lack some of the critical controls to provide ot been reasonable assurance of product quality. The controls that have n incorporated involve information flow, risk assessment, and management roles, responsibilities, and monitoring. Although the agency has established some policies and procedures to govern initiation of w prepare public health products, ATSDR lacks policies and procedu (1) establish how information about newly initiated work should effectively flow between all levels of management and staff, and (2) describe how to comprehensively assess and categorize the risk of work being initiated at the agency. Additionally, while some policies and procedures state the roles of staff in product development, many do identify the roles and responsibilities of management. Moreover, although policies and procedures include some routine oversight of product development, they do not consistently require that management monitor the development of key components of these products. Finally, while ATSDR has implemented policies and procedures governing review and clearance, some sections of the policies and proced current practices, and the policies and procedures do not ensure that all p roducts consistently receive appropriate review.

ATSDR’s Policies and Procedures for Work Initiation Do Not Establish and Describe Information Flow or Adequate Assessment of Risk

Although ATSDR has established some policies and procedures to govern the initiation of work to prepare public health products, its policies and procedures do not establish and describe how information about newly initiated work should flow between all levels of management and staff. Consequently, the agency cannot be certain that all management and staff have the information they need to do their jobs effectively. The Standards for Internal Control in the Federal Government states that for an entity to run and control its operations, it must have relevant, reliable, and timely communications relating to internal as well as external events. Information is needed throughout the agency to achieve all of its objectives, and effective communications should occur in a broad sense with information flowing down, across, and up the organization.

Since ATSDR has not established policies or procedures on how information about newly initiated work should flow between all levels of management and staff, it generally relies on various meetings held at different levels throughout the agency to inform management and staff about newly initiated work. ATSDR officials stated that when site activities are controversial or of special interest, management is informed through weekly Issues Management meetings and Senior Staff meetings between ATSDR’s OD and division directors. The former ATSDR director said that product initiation had an important role in the Issues Management meetings. According to ATSDR officials, information from these meetings is shared with mid- and lower-level management through notes and face-to-face meetings. Officials stated that newly initiated work may also be discussed during other regular meetings within the divisions and branches.

In addition to a lack of policies or procedures on information flow, for several years ATSDR has operated with fragmented databases in which information about newly initiated work is entered and tracked, none of which are accessible to, or ensure information flows to, people at all levels of the agency. ATSDR previously used a tracking system called HazDat, which was taken off line in 2007. Since that time, ATSDR management and staff have been without an agencywide system that is capable of providing information about newly initiated work to people at all levels of the agency. There are several other agency databases that contain information about newly initiated work, but none of these systems are accessible to people at all levels of the agency. Examples of these databases include a DHAC Tracking and Triage Database and a Petition Database. However, ATSDR officials told us that access to the DHAC Tracking and Triage Database was limited to management and staff within DHAC, and that the Petition Database was accessible by only three agency employees. Additionally, ATSDR officials told us that no regularly scheduled reports were generated from these databases, although division management was provided with a weekly update on petitions under review. ATSDR’s OD also created a database about 2 years ago to track issues discussed during the weekly Issues Management meetings. However, while the Issues Management database is used as a tool during the meetings, it is not accessible to division directors or other ATSDR management and staff outside of the meeting.

Although ATSDR has not established policies or procedures that establish and describe information flow within the agency, it is implementing a new agencywide system called Sequoia, which may improve the flow of information about newly initiated work between management and staff. While data entry into Sequoia began in 2007, the former director of ATSDR told us that resource limitations slowed Sequoia’s development and that the use of fragmented databases was a temporary measure until Sequoia was completed. However, officials told us that while they expected that Sequoia would replace other existing databases, further evaluation is needed to determine if Sequoia could do everything required by management or if some information will still have to be captured in separate systems. ATSDR officials told us that Sequoia was designed to track requests, cost recovery reimbursement, exposure data, work flow for site-specific products, and information pertaining to other products or services done on particular sites. Sequoia includes some major features that were not available in previous ATSDR systems, such as providing a centralized database that is available to all ATSDR staff for tracking incoming work requests, and providing a system for reporting and retrieving information on the public health impact and outcome of public health activities. According to agency officials, in January 2010 ATSDR employees began using Sequoia for planning site and project activities, recording the results of their investigative and community outreach efforts, and reporting the public health accomplishments of their activities.

ATSDR also lacks comprehensive policies and procedures for assessing and categorizing the risk of work being initiated at the agency. The Standards for Internal Control in the Federal Government states that effective internal control should provide for an assessment of the risks the agency faces from both external and internal sources and that management needs to comprehensively identify risks and consider all significant interactions between the agency and other parties. Risk identification methods may include qualitative and quantitative ranking activities, management conferences, forecasting and strategic planning, and consideration of findings from audits and other assessments. Risk assessment also includes deciding how to manage the risk and what actions should be taken, and the Internal Control Management and Evaluation Tool notes that management should formulate an approach for risk management and decide on the internal control activities required to mitigate those risks.

ATSDR previously incorporated some of the principles of risk assessment when the agency officially classified sites as “high priority” or “focus sites.” ATSDR officials told us that these sites were typically identified by senior management and staff as those sites where chemical exposures may be of significant concern, which may require extensive agency resources, or may involve other site complexities. If a site was classified as a focus site, which typically occurred as work was being initiated, any public health products resulting from that site were required to undergo a higher level of review during review and clearance. However, ATSDR officials told us that they stopped using these classifications several years ago. Instead, agency officials and employees now use terms such as “high profile concern” or “sites of interest” to refer to those sites that might require additional review and clearance because they have high interest from the media or the Congress, or involve issues of difficult or emerging science. Officials stated that these sites were now managed through meetings such as the Issues Management meeting. They stated that they believed that the Issues Management process incorporated many of the principles of risk assessment by enabling senior agency management to identify and discuss important sites each week. Nevertheless, terms such as high profile are not official agency designations and do not trigger any additional required management monitoring during product development or required higher levels of review and clearance. Additionally, while certain high profile sites may be identified as they are initiated and discussed during Issues Management meetings, not all new sites are being reviewed by OD and division management to assess and categorize the risk to the agency of the public health products resulting from the sites.

One ATSDR division, however, uses a process with elements similar to risk assessment in the way that it prioritizes work requests. DHAC generally uses a triage process for all ATSDR work requests requiring DHAC staff assistance. This process categorizes work requests as high, medium, or low priority. A request’s priority level is based on three criteria, which in order of importance are extent of exposure, public health impact, and community and political interest, according to an ATSDR document explaining the DHAC triage process. A triage decision team, consisting of management-level staff from DHAC and DRO, decides on the priority level for the work request, and that information, along with other information about the request, is tracked in the DHAC Tracking and Triage Database. However, this information is used only to prioritize DHAC work requests and assign staff accordingly. This process is not used by other ATSDR divisions, and is not an official agency designation that triggers any additional requirements for that site or related public health products, such as additional management monitoring during product development or required higher levels of review and clearance. Because ATSDR does not currently have policies and procedures that describe how the agency is to comprehensively assess and categorize the risk of work it initiates to prepare public health products, management cannot ensure that it has consistently managed the risk related to all new work, or established product preparation procedures commensurate with the risk.

ATSDR’s Policies and Procedures for Product Development Do Not Provide for Clear Management Roles and Responsibilities or Consistent Monitoring of Product Development

While some of ATSDR’s policies and procedures state the roles of staff in developing public health products, many do not identify the roles and responsibilities of management for ensuring that staff follow those policies and procedures. The Standards for Internal Control in the Federal Government states that management is responsible for developing the detailed policies, procedures, and practices to fit their agency’s operations. The Standards states that the agency’s organizational structure should clearly define key areas of authority and responsibility and establish appropriate lines of reporting. Internal control activities include approvals and the maintenance of related records to help ensure that management’s directives are carried out. The Internal Control Management and Evaluation Tool also states that managers and supervisors need to know their responsibilities for internal control and need to make control and control monitoring part of their regular operating processes.

The ATSDR documents that provide guidance on developing products do not clearly delineate management roles and responsibilities. The Public Health Assessment Guidance Manual (PHAGM) is the document that officials and employees of DHAC, DRO, and cooperative agreement partners identified as the primary document that guides their work. The PHAGM describes how to analyze site-specific data, make recommendations, and develop conclusion categories. This document is used by DHAC, DRO, and cooperative agreement staff to develop public health assessments, health consultations, and exposure investigations. The PHAGM guides staff in developing these products, but it does not establish lines of reporting or detail the responsibilities of management for monitoring product development. Additionally, although the PHAGM states that ATSDR promotes a team approach in conducting the public health assessment process, it does not describe how ATSDR management fits into this team approach. And while ATSDR officials stated that the PHAGM was not developed as a management guide, ATSDR does not have any other documents that provide guidance to management on their responsibilities for monitoring the development of public health assessments and health consultations. In addition to the PHAGM, there are a number of chemical- and exposure-specific and technical guidance documents that are used as supplements, as well as guidance specific to site work. These documents give additional information to staff on specific chemicals, how and when to use certain scientific methods, and site team procedures. However, like the PHAGM, these documents neither establish lines of reporting nor detail the responsibilities of management for monitoring product development. Furthermore, while the NCEH/ATSDR Policy: Clearance of Information Products, which guides ATSDR’s review and clearance process, states that before a product is submitted for clearance immediate supervisors should ensure that the product is based on sound, ethical science and ensure the quality of the product, the policy provides no further guidance to immediate supervisors on carrying out these responsibilities. Because there is an absence of clearly defined lines of reporting and roles and responsibilities of management in these documents, management at various levels may not understand their specific responsibilities for overseeing product development.

Although ATSDR’s policies and procedures include some routine monitoring of the development of products produced by both ATSDR staff and staff of cooperative agreement partners, they do not consistently require that agency management monitor the development of key components of these products. The Standards states that internal control should be designed to ensure that ongoing monitoring occurs in the course of normal operations, and this includes management reviews of actual agency performance. ATSDR’s policies and procedures require monitoring of key components of health studies, which use a detailed study protocol to guide a health study’s development. ATSDR’s Guidance for ATSDR Health Studies, which provides ATSDR staff with instructions on how to conduct a health study, states that a study protocol helps to ensure the quality of a health study and includes components such as a study’s objectives, methodology, and timeline for completing key activities and milestones of a health study. At a minimum, if the study being conducted is deemed research, a study protocol is reviewed and approved within the appropriate division and may be sent out for scientific peer review before the health study begins. In addition, any health study involving human subjects must also be submitted to and approved by an established institutional review board. The guidance explains that ongoing health study reviews are conducted to ensure that the study protocol is being followed, appropriate changes are made, the project remains on its established timeline, and enhancements to study quality are made when appropriate. Exposure investigations also use protocols that must be approved before the project is funded. These protocols include components such as a statement of the project’s objectives as well as data analysis methods that will be used in completing the project.

In contrast, ATSDR’s policies and procedures for the development of public health assessments and health consultations, which are among the agency’s core products, do not require management monitoring of key components of these products. For example, the PHAGM identifies an exposure assessment and health effects evaluation as the two primary technical components of the public health assessment process. Other components of the public health assessment process include data collection, community involvement, and development of conclusions and recommendations. However, there is no requirement that staff’s work in any of these areas be reviewed and approved by management during the development of a product to ensure its accuracy and appropriateness. Furthermore, DHAC and DRO officials told us that there was no formal requirement for management to monitor or approve key components of public health products produced by their divisions, such as the product’s methodology. When asked about monitoring requirements, a DRO official said that identifying the expertise needed for work at a site during the DHAC triage process helped to ensure that staff assigned to prepare a public health product had the skill sets required to make knowledgeable decisions on key components of a public health product. However, while identifying staff with the needed expertise to develop a public health product at initiation is beneficial, it is not a substitute for ongoing monitoring, which allows problems to be identified and addressed if they occur during a product’s development. Further, during our small group interviews one DHAC employee expressed concern that because there were cases where only one person was developing a product, there would be no one to monitor that work until the product was submitted for review and clearance.

Although ATSDR’s policies and procedures for the development of public health assessments and health consultations do not require management monitoring of key components of these products, ATSDR officials said they held routine meetings during which issues specific to product development could be discussed. According to ATSDR officials, division directors schedule routine meetings with branch chiefs, and other team and site meetings are held. For example, according to ATSDR officials, DHAC and DHS officials meet with their respective division branch chiefs at least once each week to discuss projects and collaborate on site activities. However, none of the routine meetings described have established requirements for monitoring the development of key components of public health products. Additionally, weekly Issues Management and Senior Staff meetings, which are attended by senior division management, are used to discuss the work conducted at sites. ATSDR officials said that product development may be monitored during these meetings. However, these meetings rely on division management to bring problems or concerns regarding product development to the attention of the OD, and according to ATSDR officials, Issues Management meetings focus only on “sites of interest.” Thus, while products related to “sites of interest” may be discussed at these meetings, current ATSDR procedures do not ensure the discussion of key components of products for ATSDR sites not identified as “sites of interest.” Additionally, items on the agenda of the Issues Management meeting are not prioritized to ensure that the most significant problems associated with the development of a public health assessment or health consultation are promptly addressed.

Because ATSDR’s policies and procedures do not describe management’s role for ensuring consistent monitoring of key product components, problems occurring during the development of ATSDR public health products may not be identified or addressed by management until the review and clearance phase, if at all. For example, in December 2001, the International Joint Commission requested ATSDR’s assistance in evaluating the public health implications of the presence of hazardous materials in the Great Lakes region. According to ATSDR and Institute of Medicine reports, problems with ATSDR’s Great Lakes report were not identified by management until the first draft of the document was completed in April 2004. Due to scientific concerns identified in the document once review and clearance began, the document underwent several years of reviews and revisions, and a final report was not issued until December 2008.

ATSDR’s Review and Clearance Policies and Procedures Do Not Always Reflect Current Practices and Do Not Establish a Process for Ensuring Consistent Review of All Products

While ATSDR has implemented policies and procedures governing the review and clearance of its public health products, some sections of ATSDR’s review and clearance policies and procedures do not reflect current practices. The Standards for Internal Control in the Federal Government states that management is responsible for developing detailed policies, procedures, and practices to fit their agency’s operations and ensuring that they are built into and become an integral part of operations. Additionally, the Internal Control Management and Evaluation Tool calls for policies and procedures to be regularly evaluated to ensure that they are still appropriate and working as intended. ATSDR uses the NCEH/ATSDR Policy: Clearance of Information Products to guide the review and clearance process. The clearance policy includes the NCEH/ATSDR Clearance Quick-Reference Guide, which outlines the required levels of review and clearance for each type of public health product. The clearance policy states that public health products may undergo required or discretionary review. The policy requires that all public health products be cleared through the initiating division, and many public health products require additional review, such as review by the Office of Science. Some public health products may also undergo additional discretionary review when the originating division believes that a division outside of the required review process should be consulted. However, some sections of the NCEH/ATSDR Policy: Clearance of Information Products do not reflect current ATSDR practices. For example, the policy “highly recommends” that all public health products be reviewed and cleared by at least four individuals: the immediate supervisor, the branch chief, the associate director for science, and the division director. In addition, the policy’s NCEH/ATSDR Clearance Quick-Reference Guide indicates that all public health assessments, health consultations, and exposure investigations must be reviewed and cleared by the division director or the division associate director for science. Yet according to DHAC management and staff, the review and clearance of DHAC products usually stops after review by branch chiefs within the division. Additionally, because the NCEH/ATSDR Clearance Quick- Reference Guide is several years old, it does not describe the review and clearance requirements for new types of agency products such as “letter health consultations,” which agency officials described as an expedited version of a health consultation. Also, the NCEH/ATSDR Clearance Quick-Reference Guide indicates that public health assessments, health consultations, and exposure investigations at “high priority” sites or “focus sites” must receive additional levels of review, but, as noted above, the agency no longer uses these designations.

As of February 2010, the NCEH/ATSDR Policy: Clearance of Information Products also did not reflect current practices because it did not direct staff to use a CDC-required electronic clearance system called Documentum. The current clearance policy was effective in March 2006, prior to implementation of Documentum. Documentum is an electronic tool used by ATSDR to route public health products to the appropriate staff for review and clearance and to track the progress of each product during the process. In November 2009, officials told us that the agency planned to issue a revised clearance policy by the end of 2009.

Additionally, although ATSDR officials said that staff should be having their products electronically reviewed and cleared through Documentum, management and staff told us during interviews that not all documents were being cleared using this system. Instead, documents that were not being entered into Documentum were being reviewed and cleared using a manual version of the review and clearance process. In November 2009, one ATSDR official estimated that only about 20 percent of DHAC documents, which include public health assessments, health consultations, and exposure investigations, were cleared using Documentum. However, per CDC policy, as of January 2010 all CDC centers, including ATSDR, were required to use Documentum to review and clear all agency products. In February 2010, ATSDR officials stated that all ATSDR divisions were currently using Documentum for the review and clearance of all documents but that the agency was still working to revise and update the clearance policy. Because some sections of ATSDR’s review and clearance policies and procedures do not reflect current agency practices, staff cannot rely on them to accurately and consistently determine what review and clearance procedures to follow. Additionally, because there has not been uniform compliance with using Documentum, agency officials have been limited in their ability to track the review and clearance history for all of the agency’s products and to ensure that the appropriate level of review was being conducted.

In addition to not reflecting current practices, ATSDR’s policies and procedures governing product review and clearance do not establish a process for ensuring that all products consistently receive appropriate review. The agency’s clearance policy and procedures generally direct management and staff to use discretion to identify products that warrant a higher level of review, rather than determining review and clearance levels through a risk assessment process. As stated above, the Standards for Internal Control in the Federal Government states that effective internal control should provide for an assessment of the risks the agency faces, and that management needs to comprehensively identify risks and consider all significant interactions between the agency and other parties. However, the level of review and clearance that ATSDR products undergo varies by product type, rather than being determined by a comprehensive risk assessment of that particular product or site. For example, health study reports prepared by ATSDR staff are required to be reviewed and cleared by the originating division and ATSDR’s Office of Science, and some must also undergo external peer review. In contrast, most public health assessments, health consultations, and exposure investigations are not required to be reviewed and cleared by ATSDR management any higher than DHAC branch chiefs. Based on the discretion of management and staff, some public health assessments, health consultations, and exposure investigations may also be submitted for additional review if they meet certain criteria. According to the ATSDR clearance policy, discretionary review is warranted when management or staff determines that a document (1) contains new or revised ATSDR policy (2) could have a high degree of visibility or (3) contains highly sensitive information. In addition, the DHAC Director has issued his own informal criteria to indicate which public health assessments and health consultations should undergo additional review beyond the branch chief level. However, even though ATSDR and DHAC have established criteria, there is no required point during a product’s preparation where management and staff collectively determine whether a product meets the criteria, and if additional review is warranted. Because ATSDR does not conduct a comprehensive risk assessment of its products or sites, and its policies and procedures instead rely on management and staff discretion to make these determinations, the agency cannot ensure its products consistently receive the appropriate level of review and clearance.

Management and staff discretion is also required in determining whether a public health assessment or health consultation should be submitted for external peer review. The ATSDR Peer Review Policy describes which public health products require external peer review, and states that all studies, results, or research that ATSDR carries out or funds in whole or in part must be peer reviewed. However, the policy specifically identifies public health assessments as one of the products that ATSDR does not consider “studies, results, or research.” Because public health assessments are not required to undergo external peer review, ATSDR officials told us that management or staff could use their discretion to determine that a public health assessment or health consultation should be submitted for external peer review. According to ATSDR data, only 2 of the 282 public health assessments and health consultations that were published in fiscal year 2008 underwent external peer review.

During the March 2009 hearing before the House Committee on Science and Technology’s Subcommittee on Investigations and Oversight, two participants suggested that ATSDR’s public health assessments and health consultations should be required to undergo external peer review as a way to help ensure their quality. A 2000 National Research Council report about peer review practices at EPA noted that peer review could promote efficiency if conducted in the early stages of a product’s development, as well as assess and potentially improve the end products of scientific work. However, the report also noted that peer review had limitations, in that peer review could not substitute for technically competent work in the development of a product and could not ensure that regulatory policies and actions would be based on good science.

Regional employees, ATSDR team leads, and nonmanagement employees in ATSDR Headquarters expressed mixed opinions to us about the use of external peer review for ATSDR public health products. In responses to a short questionnaire we administered during interviews with team leads and nonmanagement employees, 80 percent (24 of 30) said that external peer review would be either beneficial or sometimes beneficial in ensuring the quality of ATSDR public health products. Some of these employees reported that using external peer review may increase perceptions of the objectivity, credibility, and strength of their public health products. With regard to limitations, some employees reported that external peer review could cause further delays in the review and clearance process. Similarly, others suggested that external peer review should be conducted only for very complicated public health products or products with high levels of community concern or congressional interest.

Conclusions

While administrative and management controls cannot guarantee product quality, they can help ensure the development of timely and credible public health products at ATSDR. And although ATSDR has established some policies and procedures to govern the preparation of its public health products, it lacks some critical controls to provide reasonable assurance of product quality, particularly for public health assessments, health consultations, and exposure investigations. The controls that have not been incorporated involve information flow, risk assessment, management roles and responsibilities, and monitoring.

The lack of an agencywide product tracking system at ATSDR has hindered the effective flow of information about public health products between all levels of staff and management. It has also limited management’s ability to monitor agency work and ensure that resources are being allocated appropriately, placing the OD in a reactive rather than leadership position with respect to the divisions and the public health work it manages. Once the Sequoia system becomes fully operational, management and staff should have a greater ability to obtain and share information about the agency’s site-specific work, but it is too soon to determine whether they will take full advantage of Sequoia’s capabilities. Furthermore, once implemented, those capabilities require that staff and cooperative agreement partners input data into the system as was intended.

Additionally, without conducting risk assessments for the work being undertaken by the agency and using those risk assessments to guide agency processes for public health product preparation, ATSDR cannot provide reasonable assurance that its products have undergone the appropriate level of monitoring and review. If established, a risk assessment process could be used to determine the proper level of scrutiny throughout the initiation, development, and review and clearance phases, including whether or not a product should undergo external peer review, thereby ensuring that this determination is made consistently across the agency. Basing this process on a set of criteria, and documenting and tracking risk assessment decisions in agency systems, should help ensure an effective process. ATSDR has already incorporated some of the elements of risk assessment in the existing DHAC triage process for categorizing the priority of work requests.

Finally, because the agency’s policies lack guidance for management about their role in monitoring product development, ATSDR cannot be sure that management has a clear understanding of the role they are supposed to play in supervising a product’s preparation. Additionally, ATSDR’s policies and procedures for the development of public health assessments and health consultations do not require management’s monitoring and approval of key components of a product during its development. Without adequate monitoring by management during a product’s development, product errors may not be caught or significant publication delays may occur during the review and clearance phase, potentially undermining public confidence in the agency’s products.

Policies and procedures alone, however, cannot ensure the quality of ATSDR’s public health products and, as noted above, internal controls provide only reasonable assurance, not absolute assurance, that an agency’s objectives are being achieved. Issues outside of the influence of management, such as human mistakes, judgment errors, or acts by employees to circumvent management control, could also affect ATSDR’s product quality. Nonetheless, improving ATSDR’s policies and procedures regarding public health product preparation would help the agency provide greater assurance to those inside and outside the agency of the quality of these products.

Recommendations for Executive Action

To strengthen ATSDR’s policies and procedures, and ensure that they provide reasonable assurance of public health product quality, we recommend that the director of ATSDR take the following two actions: Develop policies and procedures to ensure that a risk assessment is conducted at the time site-specific work is initiated, and that any assigned risk level be reevaluated throughout product preparation to ensure that it remains appropriate.

Revise existing policies and procedures, or develop new guidance, to provide documented direction for various levels of management on their roles and responsibilities in the monitoring of all products prior to review and clearance, such as requirements for management monitoring and approval of key components of these products.

Agency Comments and Our Evaluation

ATSDR reviewed a draft of this report and provided written comments, which appear in appendix I. While ATSDR neither agreed nor disagreed with our recommendations and did not address them directly, in its comments ATSDR stated that the agency has begun to incorporate our recommendations.

Although ATSDR did not comment directly on our recommendation that the agency conduct a risk assessment at the time site-specific work is initiated and reevaluate the assessment throughout product preparation, in its comments ATSDR stated that senior management was looking into formalizing and unifying coordination, triage, and prioritization of all incoming requests across the agency. ATSDR also acknowledged a need to make its prioritization process more explicit throughout the agency. It is imperative that ASTDR formalize its processes agencywide and ensure that its processes include a risk assessment to determine the proper level of scrutiny a product should receive throughout its preparation, including whether or not it should undergo external peer review.

Related to our recommendation that ATSDR revise or develop policies and procedures to include direction for management in monitoring products prior to review and clearance, ATSDR noted that its process to formalize and unify coordination, triage, and prioritization of all incoming requests was expected to include the specification of management and staff roles and responsibilities from initiation through publication. It is important that ATSDR take this step in order to help ensure that management has a clear understanding of their responsibilities in supervising a product’s preparation.

In its comments, ATSDR noted that multiple guidelines are used to conduct its work and it uses an issues management process for agency risk management. Our findings document these guidelines and the issues management process, and describe their limitations in establishing effective information flow among all levels of management and staff, in providing a comprehensive assessment and categorization of the risk of work being initiated at the agency, and in identifying the roles and responsibilities of management.

ATSDR also acknowledged that it would benefit from formalizing additional internal controls, and stated that as part of its review of the agency clearance policy it was incorporating a way to sample documents that were previously cleared to ensure that scientific principles are being applied across all divisions. ATSDR also stated that it expected Sequoia, its agencywide electronic project tracking system, to be fully implemented by the end of the year.

Finally, ATSDR commented that we did not assess public comment as a part of our report, which it indicated was a critical component of the agency’s quality assurance process. While we agree that public comment provides valuable input on those products which are subject to a public comment period, it augments but does not substitute for thorough internal review of a product or formal, external peer review of a product by carefully selected experts.

ATSDR also provided technical comments, which we incorporated as appropriate.

As we agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services and other interested parties. The report also will be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-7114 or bascettac@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix II.

Appendix I: Comments from the Agency for Toxic Substances and Disease Registry

Appendix II: GAO Contact and Staff Acknowledgments

Contact

Acknowledgments

In addition to the contact named above, key contributors to this report were Karen Doran, Assistant Director; George Bogart; Amy C. Leone; Roseanne Price; Mario D. Ramsey; Christina Ritchie; and Carla Willis.