Agency for Toxic Substances and Disease Registry: Policies and Procedures for Public Health Product Preparation Should Be Strengthened
Published: 2010-04-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Agency for Toxic Substances and Disease Registry (ATSDR) has faced concerns related to the quality of some of the public health products it publishes. ATSDR investigates community exposures related to certain hazardous chemical sites and releases; assesses associated health effects; and recommends actions to stop, prevent, or minimize harmful effects. ATSDR publishes many types of products, including public health assessments, health consultations, exposure investigations, and health study reports. GAO was asked to examine the extent to which ATSDR's policies and procedures for product preparation, including work initiation, product development, and review and clearance, provide reasonable assurance of product quality. GAO reviewed ATSDR policies and procedures and interviewed agency officials and employees.
section_title: What GAO Found
paragraphs: The policies and procedures that ATSDR has established for public health product preparation lack some of the critical controls to provide reasonable assurance of product quality. To provide reasonable assurance that agency objectives are being met, federal internal control standards call for agencies to establish policies and procedures, assess risks associated with achieving agency objectives, ensure effective information sharing throughout the organization, monitor agency activities, and establish key areas of authority and responsibility for management and staff. While ATSDR has established some policies and procedures to guide the preparation of its public health products, the policies and procedures do not establish how information is to flow between management and staff during initiation. Absent such policies and procedures, ATSDR generally relies on various meetings to inform management and staff about new work. The agency is also implementing a new database, which may improve information flow. Furthermore, ATSDR does not comprehensively evaluate and categorize the risk of work being initiated. While the agency used to officially classify some hazardous chemical sites as "high-priority" or "focus sites," and require any products resulting from those sites to undergo a higher level of review and clearance, it no longer does so. Because ATSDR does not comprehensively assess and categorize the risk of work being initiated at the agency, management cannot ensure that they have consistently managed the risk related to new work. Additionally, many of ATSDR's policies and procedures that guide product development do not clearly define management roles and responsibilities and do not consistently require that management monitor the development of key components of these products. These deficiencies may lead management to be unclear about their responsibilities, and may result in problems that occur during product development not being identified or addressed until review and clearance, if at all. For example, ATSDR and Institute of Medicine reports show that because scientific concerns were not identified during development of an ATSDR report regarding chemical releases in the Great Lakes region, the document underwent several years of review, and a final report was not issued until more than 4 years after the first draft was written. Moreover, because some review and clearance policies do not reflect current practices, ATSDR staff cannot rely on these policies to accurately or consistently determine review and clearance procedures. Furthermore, review and clearance policies and procedures direct management and staff to use discretion to identify products that require higher levels of review, rather than making this determination through a comprehensive risk assessment process. While ATSDR policy sets out criteria for when additional review may occur, such as when a document could have a high degree of visibility, there is no required point during a product's preparation when management and staff collectively determine whether a product meets the criteria, and whether additional review is warranted. Thus, the agency cannot ensure that all products consistently receive the appropriate level of review.

section_title: Letter
paragraphs: The Agency for Toxic Substances and Disease Registry (ATSDR) has  faced concerns related to the quality of some of the public health products  it publishes. ATSDR is responsible for investigating community exposures  related to certain hazardous chemical sites and releases; assessing  associated health effects; recommending actions to stop, prevent, or  minimize harmful effects; and conducting health studies. In conducting  these activities, the agency publishes many types of public health  products, including public health assessments, health consultations, health  study reports, and exposure investigations. In 1991, an expert panel we  convened conducted a detailed evaluation of the quality of a sample of 15  ATSDR public health assessments and found that there were deficiencies  with these products. More recently, reports by the Institute of Medicine  and ATSDR’s Board of Scientific Counselors have identified various  concerns such as the appropriateness and quality of the data used in  ATSDR’s products, the methodology and design of the studies, clearance  policies, and the use of external peer review and response to review  comments.
Some members of the Congress have also expressed concern about the  quality of ATSDR’s public health products. On April 1, 2008, the U.S. House  of Representatives’ Subcommittee on Investigations and Oversight,  Committee on Science and Technology, held a hearing to examine  ATSDR’s handling of the preparation of a health consultation on the  formaldehyde levels in trailers that the Federal Emergency Management  Agency (FEMA) provided to victims of Hurricanes Katrina and Rita. The  health consultation, which was published in February 2007, raised  congressional concerns about the quality of ATSDR’s products and the  involvement of agency leadership in the issuance of a flawed product. In  response to the hearing and further examination of ATSDR’s role in the  FEMA trailers health consultation, the subcommittee issued a report in  September 2008 to express its heightened concern that ATSDR was issuing  public health products of poor quality. On March 12, 2009, the  subcommittee held a second hearing—The Agency for Toxic Substances  and Disease Registry (ATSDR): Problems in the Past, Potential for the  Future?—which focused on its continued concern about the quality of  ATSDR’s products.
ATSDR has responded to these concerns, noting that multiple factors have  posed challenges for the agency, including limitations in the ability of  available science to answer community questions about the effect of  chemical exposures, limitations in ATSDR’s ability to collect data related  to exposures, and reductions since 2004 in the number of ATSDR staff and  resources available to conduct the agency’s mission. In his testimony for  the March 2009 congressional hearing, ATSDR’s director at that time  noted that the agency intended to reexamine its approach to carrying out  its mission in light of these challenges, and had convened a National  Conversation on Public Health and Chemical Exposures which includes  government, community groups, and industry to create an agenda for  revitalizing the public health approach to chemical exposures, which  would include future direction for ATSDR.
You have expressed interest in ensuring the quality of ATSDR’s public  health products. In this report we examine the extent to which ATSDR’s  policies and procedures for product initiation, development, and review  and clearance provide reasonable assurance of public health product  quality.
To do our work, we reviewed ATSDR’s policies and procedures and  interviewed officials to identify guidance related to the preparation of  public health products. Preparation of public health products  encompasses (1) initiation, which includes a decision by the agency to  begin work on a public health product and the assignment of staff to  prepare the product, (2) development, which includes management  approval to proceed with the development of a product and the actual  drafting of the public health product, and (3) review and clearance, which  is the process by which a product is internally or externally reviewed and  disseminated as a final public health product. We focused our review of  ATSDR’s policies and procedures on those related to public health  assessments, health consultations, exposure investigations, and health  study reports because these products are considered core public health  products by ATSDR and concerns have been raised about the quality of  products such as these, in which ATSDR identifies potential exposures to  hazardous chemicals and assesses associated health effects. Throughout  this report, we use the phrase “public health products” to refer solely to  those products on which we focused our review: public health  assessments, health consultations, exposure investigations, and health  studies. We compared the policies and procedures ATSDR uses to guide  the preparation of its public health products to the standards described in  the Standards for Internal Control in the Federal Government and the  related Internal Control Management and Evaluation Tool. We did not  evaluate ATSDR’s policies and procedures on human capital, financial  management, or scientific and technical risk assessment. Additionally, we  did not review ATSDR products to assess their quality. Accordingly, we do  not express any view about their accuracy, completeness, or scientific  credibility.
We excluded employees with purely administrative responsibilities from the population of  nonmanagement employees in each division. team leads and nonmanagement employees. To encourage the candor o the individuals who participated in the small group interviews, we did n record their names in our notes from those interviews and agreed not to  share our notes with ATSDR management. Additionally, at the conclusion  of each of the six small group interviews we administered a short  questionnaire to the participants to collect additional information about  their perspectives on the policies and procedures that guide their work,  and on improving public health product quality at the agency. Of the 33  questionnaires we distributed, we received 30 completed questionnaires.  The views expressed by these employees cannot be generalized to all  employees working within these divisions or in these roles.
We also conducted six on-site small group interviews. We interviewed  employees in 3 of ATSDR’s 10 Division of Regional Operations (DRO)  offices, and employees in 3 of 30 cooperative agreement partner offices.  We chose the three regions that issued the greatest number of public  health assessments and health consultations in fiscal year 2008. Those  regions were Region 5 (Illinois, Indiana, Michigan, Minnesota, Ohio, and  Wisconsin), Region 10 (Alaska, Idaho, Oregon, and Washington), and  Region 4 (Alabama, Florida, Georgia, Kentucky, Mississippi, North  Carolina, South Carolina, and Tennessee). We interviewed the three  cooperative agreement partners that were located in the same states as  our selected regional offices—the Washington State Department of Health,  the Illinois Department of Public Health, and the Georgia Division of  Public Health.
We also conducted interviews with officials, experts, and researchers  outside ATSDR to gain an understanding of ATSDR’s relationship with  other agencies, to get their perspectives on ATSDR’s work, and to learn  about the policies and procedures used by other prominent scientific  research organizations. We conducted interviews with federal officials  from the Centers for Disease Control and Prevention (CDC); the  Environmental Protection Agency (EPA), which ATSDR advises about the  health aspects of hazardous waste sites or spills; and the National  Institutes of Health’s (NIH) National Institute of Environmental Health  Sciences (NIEHS), with which ATSDR collaborates on various matters  related to environmental health science. We also conducted interviews  with officials from two national scientific research organizations, the  National Academies and the National Science Foundation. We interviewed  two experts in environmental health science who had experience working  with ATSDR, and we spoke with two advocacy organizations that work  with communities that have been affected by environmental health  problems.
We conducted this performance audit from April 2009 to April 2010, in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
subsections: section_title: Background
paragraphs: ATSDR investigates community exposures related to chemical sites and  releases; works with federal, tribal, state, and local agencies to identify  potential exposures; assesses associated health effects; and recommends  actions to stop, prevent, or minimize these harmful effects, among other  things.
subsections: section_title: ATSDR History
paragraphs: ATSDR was established by the Comprehensive Environmental Response,  Compensation, and Liability Act of 1980 (CERCLA), which created what is  known as the Superfund program to clean up the nation’s most dangerous  hazardous waste sites. CERCLA established ATSDR to carry out  Superfund’s health-related activities, including the establishment of a  national registry of and the provision of medical care and testing to  persons exposed to toxic substances and the provision of survey and  screening programs to determine the relationship between such exposure  and illness. In 1985, ATSDR was formally organized to begin to carry out  its responsibilities under the law. The Superfund Amendments and  Reauthorization Act of 1986 (SARA) broadened ATSDR’s responsibilities  to include, among other things, public health assessments, establishment  and maintenance of toxicologic databases, information dissemination, and  health education. SARA required that ATSDR conduct a public health  assessment at each site proposed for or on the National Priorities List  (NPL), and authorized ATSDR to perform public health assessments  upon petition by an individual or physician and to conduct additional  follow-up health studies if needed.
section_title: Sources of ATSDR Work and Types of Public Health Products
paragraphs: ATSDR may initiate work to prepare public health products for several  reasons. Work can be necessitated pursuant to SARA by a site’s proposal  to or listing on the NPL, requested by an ATSDR partner such as EPA,  negotiated as part of a work plan for federal facilities, petitioned by  individuals or physicians, or generated internally by ATSDR officials. Once  work is initiated, ATSDR may prepare any of several different types of  products, including the following:    Public health assessments evaluate data and information on the release  of hazardous substances into the environment in order to assess any past,  current, or future impact on public health, develop health advisories or  other recommendations, and identify studies or actions needed to evaluate  and mitigate or prevent human health effects.
Health consultations review available information or collect new data to  respond to a specific health question or request for information about a  potential environmental hazard. Health consultations are focused on a  specific exposure issue and provide guidance on the specific health- related question.
Health studies are epidemiological research conducted to investigate and  characterize the association between exposure to chemicals in the  environment and health problems of people who have been exposed to  chemicals.
Exposure investigations collect and analyze site-specific environmental  or biological samples to determine whether individuals have been exposed  to hazardous substances. Exposure investigations are often designed to  examine individuals most likely to be exposed to hazardous substances,  rather than a sample of individuals from the exposed community that  would provide information about the community as a whole.
The time required to complete ATSDR public health products varies, and  may depend on the nature and complexity of the work site. For example,  some public health assessments and most health studies take one or more  years to complete, whereas some health consultations are completed  within weeks. ATSDR also prepares emergency response products—most  of which are completed within hours or days—which are intended to help  interpret the implications of exposure data. These “real-time”  investigations of health exposures include technical assistance and health  consultations, and are often initiated in response to requests from  agencies such as EPA or state health or environmental departments.
section_title: ATSDR Organizational Structure
paragraphs: Although ATSDR was established within the Department of Health and  Human Service’s (HHS) Public Health Service, the Director of CDC serves  as the Administrator of ATSDR, and CDC performs many administrative  functions for ATSDR, such as human capital and financial management  services. ATSDR is located within CDC’s Office of Noncommunicable  Diseases, Injury and Environmental Health. In 2003, ATSDR’s  administrative functions were combined with those of CDC’s National  Center for Environmental Health (NCEH). ATSDR and NCEH share an  Office of the Director (OD), which is led by a director and deputy  director. ATSDR has four divisions, each of which is divided into either  branches, programs, or regional offices (see fig. 1). Each division is led by  a director and deputy director, and DHAC, DHS, and DTEM each also have  an associate director for science. Each branch or program within a  division is led by a chief.
OD—As shown in figure 1, the OD has eight functional areas:  communication science; community environmental health; financial and  administrative services; information systems; policy, planning and  evaluation; program development; science; and terrorism preparedness  and emergency response. These functional areas are responsible for  providing scientific and programmatic support for agency staff and  conducting review and clearance for public health products produced by  ATSDR divisions. Specifically, the Office of Science is responsible for the  clearance, cross-clearance, and external peer review of ATSDR public  health products. The Office of Science also coordinates the NCEH/ATSDR  Board of Scientific Counselors, which provides advice and guidance to  ATSDR’s director on external peer review of ATSDR programs and issues  including program goals, objectives, strategies, and priorities. The  board’s advice and guidance are intended to assist ATSDR in ensuring  scientific quality, timeliness, utility, and dissemination of scientific results.
DHAC—DHAC produces a number of products, including public health  assessments, health consultations, and exposure investigations. DHAC’s  Cooperative Agreement and Program Evaluation Branch is charged with  supporting and overseeing the work produced by ATSDR’s cooperative  agreement partners, which currently include 29 state agencies and 1 tribal  government (see fig. 2). In order to become a cooperative agreement  partner, state and tribal governments must respond to a request for  applications that ATSDR posts, and have their application reviewed,  scored, and funded by ATSDR. Currently, ATSDR funds cooperative  agreement partners for a 5-year funding cycle. Through these  partnerships, ATSDR provides funding and technical support for state and  tribal government employees to assess environmental health concerns at  sites within their jurisdictions and to conduct needed public health  interventions. Cooperative agreement partners prepare public health  products that are monitored, reviewed, and cleared by ATSDR. DHAC staff  work with staff in regional offices to provide technical assistance to  cooperative agreement partners and the public, and to sponsor activities in  communities that have been exposed to hazardous chemicals.
DHS—DHS is responsible for conducting epidemiologic health studies,  designing and conducting surveillance programs, and establishing and  maintaining registries. The division collects information to determine  whether a chemical exposure is making people sick, and collects data on  persons identified as having been exposed to a specific contaminant or  event. DHS is also involved with the ongoing collection, analysis, and  interpretation of health data. These products are released as ATSDR  reports and are often published in the scientific literature.
DRO—DRO staffs an ATSDR regional office within each of the 10 EPA  regions, in EPA Headquarters in Washington, D.C., and in two satellite  offices in Helena, Montana, and Anchorage, Alaska (see fig. 2). These  offices are responsible for acting as regional liaisons to establish working  relationships with EPA, other federal and state agencies, individual  citizens, and community groups to maintain current and historic  knowledge of issues related to hazardous chemical sites in their regions.  Working in collaboration with DHAC, DRO staff also prepare a number of  time-sensitive products, such as health consultations.
DTEM—DTEM assists in the production of health consultations and  provides technical assistance in response to chemical spills and acute  events. DTEM staff also work closely with DRO staff to provide real-time  public health advice in case of a chemical release. DTEM is responsible  serving as a resource for information and assistance on toxic substances  in the environment, and prepares toxicological profiles for hazardous  substances.
section_title: Internal Control
paragraphs: The Standards for Internal Control in the Federal Government provides  the overall framework for establishing guidelines for internal control, and  helps government managers achieve desired objectives. As applied to  ATSDR, this could include the preparation of quality public health  products. Internal control, which is synonymous with management  control, comprises the plans, methods, and procedures used to meet  missions, goals, and objectives. The related Internal Control Management  and Evaluation Tool assists agencies in maintaining or implementing  effective control. Internal control is not one event, but a series of actions  and activities that occur throughout an entity’s operations and on an  ongoing basis. The responsibility of good internal control rests with all  managers; they set the objectives, put the control mechanisms and  activities in place, and monitor and evaluate these mechanisms and  activities. However, all employees in the organization play important roles  in this process. Internal control includes activities such as establishing  policies and procedures, assessing risks associated with achieving agen objectives, ensuring effective information sharing throughout the  organization, conducting ongoing monitoring of agency activities, and  establishing key areas of authority and responsibility for agency  management and staff.
The Standards note, however, that while internal control helps  government managers achieve desired objectives, it cannot provide  absolute assurance that all agency objectives will be met. There are many factors outside the control and influence of management that can affect agency’s ability to achieve its objectives. For example, human mistakes,  judgment errors, and acts of collusion to circumvent control can affect  meeting agency objectives. Therefore, once in place, internal control  provides reasonable assurance, not absolute assurance, that an agency’s  objectives are being achieved.
section_title: ATSDR’s Policies and Procedures for P Health Product Preparation Lack Some Critical Controls to P Reasonable Assurance of Quality
paragraphs: The policies and procedures that ATSDR has established for public health product preparation lack some of the critical controls to provide  ot been  reasonable assurance of product quality. The controls that have n incorporated involve information flow, risk assessment, and management roles, responsibilities, and monitoring. Although the agency has  established some policies and procedures to govern initiation of w prepare public health products, ATSDR lacks policies and procedu (1) establish how information about newly initiated work should  effectively flow between all levels of management and staff, and   (2) describe how to comprehensively assess and categorize the risk of  work being initiated at the agency. Additionally, while some policies and  procedures state the roles of staff in product development, many do identify the roles and responsibilities of management. Moreover, although policies and procedures include some routine oversight of product  development, they do not consistently require that management monitor  the development of key components of these products. Finally, while  ATSDR has implemented policies and procedures governing review and  clearance, some sections of the policies and proced current practices, and the policies and procedures do not ensure that all  p roducts consistently receive appropriate review.
subsections: section_title: ATSDR’s Policies and Procedures for Work Initiation Do Not Establish and Describe Information Flow or Adequate Assessment of Risk
paragraphs: Although ATSDR has established some policies and procedures to govern  the initiation of work to prepare public health products, its policies and  procedures do not establish and describe how information about newly  initiated work should flow between all levels of management and staff.  Consequently, the agency cannot be certain that all management and staff  have the information they need to do their jobs effectively. The Standards  for Internal Control in the Federal Government states that for an entity to  run and control its operations, it must have relevant, reliable, and timely  communications relating to internal as well as external events.  Information is needed throughout the agency to achieve all of its  objectives, and effective communications should occur in a broad sense  with information flowing down, across, and up the organization.
Since ATSDR has not established policies or procedures on how  information about newly initiated work should flow between all levels of  management and staff, it generally relies on various meetings held at  different levels throughout the agency to inform management and staff  about newly initiated work. ATSDR officials stated that when site  activities are controversial or of special interest, management is informed  through weekly Issues Management meetings and Senior Staff meetings  between ATSDR’s OD and division directors. The former ATSDR director  said that product initiation had an important role in the Issues  Management meetings. According to ATSDR officials, information from  these meetings is shared with mid- and lower-level management through  notes and face-to-face meetings. Officials stated that newly initiated work  may also be discussed during other regular meetings within the divisions  and branches.
In addition to a lack of policies or procedures on information flow, for  several years ATSDR has operated with fragmented databases in which  information about newly initiated work is entered and tracked, none of  which are accessible to, or ensure information flows to, people at all levels  of the agency. ATSDR previously used a tracking system called HazDat,  which was taken off line in 2007. Since that time, ATSDR management  and staff have been without an agencywide system that is capable of  providing information about newly initiated work to people at all levels of  the agency. There are several other agency databases that contain  information about newly initiated work, but none of these systems are  accessible to people at all levels of the agency. Examples of these  databases include a DHAC Tracking and Triage Database and a Petition  Database. However, ATSDR officials told us that access to the DHAC  Tracking and Triage Database was limited to management and staff within  DHAC, and that the Petition Database was accessible by only three agency  employees. Additionally, ATSDR officials told us that no regularly  scheduled reports were generated from these databases, although division  management was provided with a weekly update on petitions under  review. ATSDR’s OD also created a database about 2 years ago to track  issues discussed during the weekly Issues Management meetings.  However, while the Issues Management database is used as a tool during  the meetings, it is not accessible to division directors or other ATSDR  management and staff outside of the meeting.
Although ATSDR has not established policies or procedures that establish  and describe information flow within the agency, it is implementing a new  agencywide system called Sequoia, which may improve the flow of  information about newly initiated work between management and staff.  While data entry into Sequoia began in 2007, the former director of ATSDR  told us that resource limitations slowed Sequoia’s development and that  the use of fragmented databases was a temporary measure until Sequoia  was completed. However, officials told us that while they expected that  Sequoia would replace other existing databases, further evaluation is  needed to determine if Sequoia could do everything required by  management or if some information will still have to be captured in  separate systems. ATSDR officials told us that Sequoia was designed to  track requests, cost recovery reimbursement, exposure data, work flow for  site-specific products, and information pertaining to other products or  services done on particular sites. Sequoia includes some major features  that were not available in previous ATSDR systems, such as providing a  centralized database that is available to all ATSDR staff for tracking  incoming work requests, and providing a system for reporting and  retrieving information on the public health impact and outcome of public  health activities. According to agency officials, in January 2010 ATSDR  employees began using Sequoia for planning site and project activities,  recording the results of their investigative and community outreach  efforts, and reporting the public health accomplishments of their activities.
ATSDR also lacks comprehensive policies and procedures for assessing  and categorizing the risk of work being initiated at the agency. The  Standards for Internal Control in the Federal Government states that  effective internal control should provide for an assessment of the risks the  agency faces from both external and internal sources and that  management needs to comprehensively identify risks and consider all  significant interactions between the agency and other parties. Risk  identification methods may include qualitative and quantitative ranking  activities, management conferences, forecasting and strategic planning,  and consideration of findings from audits and other assessments. Risk  assessment also includes deciding how to manage the risk and what  actions should be taken, and the Internal Control Management and  Evaluation Tool notes that management should formulate an approach for  risk management and decide on the internal control activities required to  mitigate those risks.
ATSDR previously incorporated some of the principles of risk assessment  when the agency officially classified sites as “high priority” or “focus  sites.” ATSDR officials told us that these sites were typically identified by  senior management and staff as those sites where chemical exposures may  be of significant concern, which may require extensive agency resources,  or may involve other site complexities. If a site was classified as a focus  site, which typically occurred as work was being initiated, any public  health products resulting from that site were required to undergo a higher  level of review during review and clearance. However, ATSDR officials  told us that they stopped using these classifications several years ago.  Instead, agency officials and employees now use terms such as “high  profile concern” or “sites of interest” to refer to those sites that might  require additional review and clearance because they have high interest  from the media or the Congress, or involve issues of difficult or emerging  science. Officials stated that these sites were now managed through  meetings such as the Issues Management meeting. They stated that they  believed that the Issues Management process incorporated many of the  principles of risk assessment by enabling senior agency management to  identify and discuss important sites each week. Nevertheless, terms such  as high profile are not official agency designations and do not trigger any  additional required management monitoring during product development  or required higher levels of review and clearance. Additionally, while  certain high profile sites may be identified as they are initiated and  discussed during Issues Management meetings, not all new sites are being  reviewed by OD and division management to assess and categorize the  risk to the agency of the public health products resulting from the sites.
One ATSDR division, however, uses a process with elements similar to  risk assessment in the way that it prioritizes work requests. DHAC  generally uses a triage process for all ATSDR work requests requiring  DHAC staff assistance. This process categorizes work requests as high,  medium, or low priority. A request’s priority level is based on three  criteria, which in order of importance are extent of exposure, public  health impact, and community and political interest, according to an  ATSDR document explaining the DHAC triage process. A triage decision  team, consisting of management-level staff from DHAC and DRO, decides  on the priority level for the work request, and that information, along with  other information about the request, is tracked in the DHAC Tracking and  Triage Database. However, this information is used only to prioritize  DHAC work requests and assign staff accordingly. This process is not used  by other ATSDR divisions, and is not an official agency designation that  triggers any additional requirements for that site or related public health  products, such as additional management monitoring during product  development or required higher levels of review and clearance. Because  ATSDR does not currently have policies and procedures that describe how  the agency is to comprehensively assess and categorize the risk of work it  initiates to prepare public health products, management cannot ensure  that it has consistently managed the risk related to all new work, or  established product preparation procedures commensurate with the risk.
section_title: ATSDR’s Policies and Procedures for Product Development Do Not Provide for Clear Management Roles and Responsibilities or Consistent Monitoring of Product Development
paragraphs: While some of ATSDR’s policies and procedures state the roles of staff in  developing public health products, many do not identify the roles and  responsibilities of management for ensuring that staff follow those policies  and procedures. The Standards for Internal Control in the Federal  Government states that management is responsible for developing the  detailed policies, procedures, and practices to fit their agency’s  operations. The Standards states that the agency’s organizational  structure should clearly define key areas of authority and responsibility  and establish appropriate lines of reporting. Internal control activities  include approvals and the maintenance of related records to help ensure  that management’s directives are carried out. The Internal Control  Management and Evaluation Tool also states that managers and  supervisors need to know their responsibilities for internal control and  need to make control and control monitoring part of their regular  operating processes.
The ATSDR documents that provide guidance on developing products do  not clearly delineate management roles and responsibilities. The Public  Health Assessment Guidance Manual (PHAGM) is the document that  officials and employees of DHAC, DRO, and cooperative agreement  partners identified as the primary document that guides their work. The  PHAGM describes how to analyze site-specific data, make  recommendations, and develop conclusion categories. This document is  used by DHAC, DRO, and cooperative agreement staff to develop public  health assessments, health consultations, and exposure investigations. The  PHAGM guides staff in developing these products, but it does not establish  lines of reporting or detail the responsibilities of management for  monitoring product development. Additionally, although the PHAGM  states that ATSDR promotes a team approach in conducting the public  health assessment process, it does not describe how ATSDR management  fits into this team approach. And while ATSDR officials stated that the  PHAGM was not developed as a management guide, ATSDR does not have  any other documents that provide guidance to management on their  responsibilities for monitoring the development of public health  assessments and health consultations. In addition to the PHAGM, there are  a number of chemical- and exposure-specific and technical guidance  documents that are used as supplements, as well as guidance specific to  site work. These documents give additional information to staff on specific  chemicals, how and when to use certain scientific methods, and site team  procedures. However, like the PHAGM, these documents neither establish  lines of reporting nor detail the responsibilities of management for  monitoring product development. Furthermore, while the NCEH/ATSDR  Policy: Clearance of Information Products, which guides ATSDR’s review  and clearance process, states that before a product is submitted for  clearance immediate supervisors should ensure that the product is based  on sound, ethical science and ensure the quality of the product, the policy  provides no further guidance to immediate supervisors on carrying out  these responsibilities. Because there is an absence of clearly defined lines  of reporting and roles and responsibilities of management in these  documents, management at various levels may not understand their  specific responsibilities for overseeing product development.
Although ATSDR’s policies and procedures include some routine  monitoring of the development of products produced by both ATSDR staff  and staff of cooperative agreement partners, they do not consistently  require that agency management monitor the development of key  components of these products. The Standards states that internal control  should be designed to ensure that ongoing monitoring occurs in the course  of normal operations, and this includes management reviews of actual  agency performance. ATSDR’s policies and procedures require monitoring  of key components of health studies, which use a detailed study protocol  to guide a health study’s development. ATSDR’s Guidance for ATSDR  Health Studies, which provides ATSDR staff with instructions on how to  conduct a health study, states that a study protocol helps to ensure the  quality of a health study and includes components such as a study’s  objectives, methodology, and timeline for completing key activities and  milestones of a health study. At a minimum, if the study being conducted  is deemed research, a study protocol is reviewed and approved within the  appropriate division and may be sent out for scientific peer review before  the health study begins. In addition, any health study involving human  subjects must also be submitted to and approved by an established  institutional review board. The guidance explains that ongoing health  study reviews are conducted to ensure that the study protocol is being  followed, appropriate changes are made, the project remains on its  established timeline, and enhancements to study quality are made when  appropriate. Exposure investigations also use protocols that must be  approved before the project is funded. These protocols include  components such as a statement of the project’s objectives as well as data  analysis methods that will be used in completing the project.
In contrast, ATSDR’s policies and procedures for the development of  public health assessments and health consultations, which are among the  agency’s core products, do not require management monitoring of key  components of these products. For example, the PHAGM identifies an  exposure assessment and health effects evaluation as the two primary  technical components of the public health assessment process. Other  components of the public health assessment process include data  collection, community involvement, and development of conclusions and  recommendations. However, there is no requirement that staff’s work in  any of these areas be reviewed and approved by management during the  development of a product to ensure its accuracy and appropriateness.  Furthermore, DHAC and DRO officials told us that there was no formal  requirement for management to monitor or approve key components of  public health products produced by their divisions, such as the product’s  methodology. When asked about monitoring requirements, a DRO official  said that identifying the expertise needed for work at a site during the  DHAC triage process helped to ensure that staff assigned to prepare a  public health product had the skill sets required to make knowledgeable  decisions on key components of a public health product. However, while  identifying staff with the needed expertise to develop a public health  product at initiation is beneficial, it is not a substitute for ongoing  monitoring, which allows problems to be identified and addressed if they  occur during a product’s development. Further, during our small group  interviews one DHAC employee expressed concern that because there  were cases where only one person was developing a product, there would  be no one to monitor that work until the product was submitted for review  and clearance.
Although ATSDR’s policies and procedures for the development of public  health assessments and health consultations do not require management  monitoring of key components of these products, ATSDR officials said  they held routine meetings during which issues specific to product  development could be discussed. According to ATSDR officials, division  directors schedule routine meetings with branch chiefs, and other team  and site meetings are held. For example, according to ATSDR officials,  DHAC and DHS officials meet with their respective division branch chiefs  at least once each week to discuss projects and collaborate on site  activities. However, none of the routine meetings described have  established requirements for monitoring the development of key  components of public health products. Additionally, weekly Issues  Management and Senior Staff meetings, which are attended by senior  division management, are used to discuss the work conducted at sites.  ATSDR officials said that product development may be monitored during  these meetings. However, these meetings rely on division management to  bring problems or concerns regarding product development to the  attention of the OD, and according to ATSDR officials, Issues Management  meetings focus only on “sites of interest.” Thus, while products related to  “sites of interest” may be discussed at these meetings, current ATSDR  procedures do not ensure the discussion of key components of products  for ATSDR sites not identified as “sites of interest.” Additionally, items on  the agenda of the Issues Management meeting are not prioritized to ensure  that the most significant problems associated with the development of a  public health assessment or health consultation are promptly addressed.
Because ATSDR’s policies and procedures do not describe management’s  role for ensuring consistent monitoring of key product components,  problems occurring during the development of ATSDR public health  products may not be identified or addressed by management until the  review and clearance phase, if at all. For example, in December 2001, the  International Joint Commission requested ATSDR’s assistance in  evaluating the public health implications of the presence of hazardous  materials in the Great Lakes region. According to ATSDR and Institute of  Medicine reports, problems with ATSDR’s Great Lakes report were not  identified by management until the first draft of the document was  completed in April 2004. Due to scientific concerns identified in the  document once review and clearance began, the document underwent  several years of reviews and revisions, and a final report was not issued  until December 2008.
section_title: ATSDR’s Review and Clearance Policies and Procedures Do Not Always Reflect Current Practices and Do Not Establish a Process for Ensuring Consistent Review of All Products
paragraphs: While ATSDR has implemented policies and procedures governing the  review and clearance of its public health products, some sections of  ATSDR’s review and clearance policies and procedures do not reflect  current practices. The Standards for Internal Control in the Federal  Government states that management is responsible for developing detailed  policies, procedures, and practices to fit their agency’s operations and  ensuring that they are built into and become an integral part of operations.  Additionally, the Internal Control Management and Evaluation Tool calls  for policies and procedures to be regularly evaluated to ensure that they  are still appropriate and working as intended. ATSDR uses the  NCEH/ATSDR Policy: Clearance of Information Products to guide the  review and clearance process. The clearance policy includes the  NCEH/ATSDR Clearance Quick-Reference Guide, which outlines the  required levels of review and clearance for each type of public health  product. The clearance policy states that public health products may  undergo required or discretionary review. The policy requires that all  public health products be cleared through the initiating division, and  many public health products require additional review, such as review by  the Office of Science. Some public health products may also undergo  additional discretionary review when the originating division believes that  a division outside of the required review process should be consulted.  However, some sections of the NCEH/ATSDR Policy: Clearance of  Information Products do not reflect current ATSDR practices. For  example, the policy “highly recommends” that all public health products  be reviewed and cleared by at least four individuals: the immediate  supervisor, the branch chief, the associate director for science, and the  division director. In addition, the policy’s NCEH/ATSDR Clearance  Quick-Reference Guide indicates that all public health assessments, health  consultations, and exposure investigations must be reviewed and cleared  by the division director or the division associate director for science. Yet  according to DHAC management and staff, the review and clearance of  DHAC products usually stops after review by branch chiefs within the  division. Additionally, because the NCEH/ATSDR Clearance Quick- Reference Guide is several years old, it does not describe the review and  clearance requirements for new types of agency products such as “letter  health consultations,” which agency officials described as an expedited  version of a health consultation. Also, the NCEH/ATSDR Clearance  Quick-Reference Guide indicates that public health assessments, health  consultations, and exposure investigations at “high priority” sites or “focus  sites” must receive additional levels of review, but, as noted above, the  agency no longer uses these designations.
As of February 2010, the NCEH/ATSDR Policy: Clearance of Information  Products also did not reflect current practices because it did not direct  staff to use a CDC-required electronic clearance system called  Documentum. The current clearance policy was effective in March 2006,  prior to implementation of Documentum. Documentum is an electronic  tool used by ATSDR to route public health products to the appropriate  staff for review and clearance and to track the progress of each product  during the process. In November 2009, officials told us that the agency  planned to issue a revised clearance policy by the end of 2009.
Additionally, although ATSDR officials said that staff should be having  their products electronically reviewed and cleared through Documentum,  management and staff told us during interviews that not all documents  were being cleared using this system. Instead, documents that were not  being entered into Documentum were being reviewed and cleared using a  manual version of the review and clearance process. In November 2009,  one ATSDR official estimated that only about 20 percent of DHAC  documents, which include public health assessments, health  consultations, and exposure investigations, were cleared using  Documentum. However, per CDC policy, as of January 2010 all CDC  centers, including ATSDR, were required to use Documentum to review  and clear all agency products. In February 2010, ATSDR officials stated  that all ATSDR divisions were currently using Documentum for the review  and clearance of all documents but that the agency was still working to  revise and update the clearance policy. Because some sections of ATSDR’s  review and clearance policies and procedures do not reflect current  agency practices, staff cannot rely on them to accurately and consistently  determine what review and clearance procedures to follow. Additionally,  because there has not been uniform compliance with using Documentum,  agency officials have been limited in their ability to track the review and  clearance history for all of the agency’s products and to ensure that the  appropriate level of review was being conducted.
In addition to not reflecting current practices, ATSDR’s policies and  procedures governing product review and clearance do not establish a  process for ensuring that all products consistently receive appropriate  review. The agency’s clearance policy and procedures generally direct  management and staff to use discretion to identify products that warrant a  higher level of review, rather than determining review and clearance levels  through a risk assessment process. As stated above, the Standards for  Internal Control in the Federal Government states that effective internal  control should provide for an assessment of the risks the agency faces,  and that management needs to comprehensively identify risks and  consider all significant interactions between the agency and other parties.  However, the level of review and clearance that ATSDR products undergo  varies by product type, rather than being determined by a comprehensive  risk assessment of that particular product or site. For example, health  study reports prepared by ATSDR staff are required to be reviewed and  cleared by the originating division and ATSDR’s Office of Science, and  some must also undergo external peer review. In contrast, most public  health assessments, health consultations, and exposure investigations are  not required to be reviewed and cleared by ATSDR management any  higher than DHAC branch chiefs. Based on the discretion of management  and staff, some public health assessments, health consultations, and  exposure investigations may also be submitted for additional review if  they meet certain criteria. According to the ATSDR clearance policy,  discretionary review is warranted when management or staff determines  that a document (1) contains new or revised ATSDR policy (2) could have  a high degree of visibility or (3) contains highly sensitive information. In  addition, the DHAC Director has issued his own informal criteria to  indicate which public health assessments and health consultations should  undergo additional review beyond the branch chief level. However, even  though ATSDR and DHAC have established criteria, there is no required  point during a product’s preparation where management and staff  collectively determine whether a product meets the criteria, and if  additional review is warranted. Because ATSDR does not conduct a  comprehensive risk assessment of its products or sites, and its policies  and procedures instead rely on management and staff discretion to make  these determinations, the agency cannot ensure its products consistently  receive the appropriate level of review and clearance.
Management and staff discretion is also required in determining whether a  public health assessment or health consultation should be submitted for  external peer review. The ATSDR Peer Review Policy describes which  public health products require external peer review, and states that all  studies, results, or research that ATSDR carries out or funds in whole or in  part must be peer reviewed. However, the policy specifically identifies  public health assessments as one of the products that ATSDR does not  consider “studies, results, or research.” Because public health  assessments are not required to undergo external peer review, ATSDR  officials told us that management or staff could use their discretion to  determine that a public health assessment or health consultation should  be submitted for external peer review. According to ATSDR data, only 2  of the 282 public health assessments and health consultations that were  published in fiscal year 2008 underwent external peer review.
During the March 2009 hearing before the House Committee on Science  and Technology’s Subcommittee on Investigations and Oversight, two  participants suggested that ATSDR’s public health assessments and health  consultations should be required to undergo external peer review as a way  to help ensure their quality. A 2000 National Research Council report  about peer review practices at EPA noted that peer review could promote  efficiency if conducted in the early stages of a product’s development, as  well as assess and potentially improve the end products of scientific  work. However, the report also noted that peer review had limitations, in  that peer review could not substitute for technically competent work in  the development of a product and could not ensure that regulatory policies  and actions would be based on good science.
Regional employees, ATSDR team leads, and nonmanagement employees  in ATSDR Headquarters expressed mixed opinions to us about the use of  external peer review for ATSDR public health products. In responses to a  short questionnaire we administered during interviews with team leads  and nonmanagement employees, 80 percent (24 of 30) said that external  peer review would be either beneficial or sometimes beneficial in ensuring  the quality of ATSDR public health products. Some of these employees  reported that using external peer review may increase perceptions of the  objectivity, credibility, and strength of their public health products. With  regard to limitations, some employees reported that external peer review  could cause further delays in the review and clearance process. Similarly,  others suggested that external peer review should be conducted only for  very complicated public health products or products with high levels of  community concern or congressional interest.
section_title: Conclusions
paragraphs: While administrative and management controls cannot guarantee product  quality, they can help ensure the development of timely and credible  public health products at ATSDR. And although ATSDR has established  some policies and procedures to govern the preparation of its public  health products, it lacks some critical controls to provide reasonable  assurance of product quality, particularly for public health assessments,  health consultations, and exposure investigations. The controls that have  not been incorporated involve information flow, risk assessment,  management roles and responsibilities, and monitoring.
The lack of an agencywide product tracking system at ATSDR has  hindered the effective flow of information about public health products  between all levels of staff and management. It has also limited  management’s ability to monitor agency work and ensure that resources  are being allocated appropriately, placing the OD in a reactive rather than  leadership position with respect to the divisions and the public health  work it manages. Once the Sequoia system becomes fully operational,  management and staff should have a greater ability to obtain and share  information about the agency’s site-specific work, but it is too soon to  determine whether they will take full advantage of Sequoia’s capabilities.  Furthermore, once implemented, those capabilities require that staff and  cooperative agreement partners input data into the system as was  intended.
Additionally, without conducting risk assessments for the work being  undertaken by the agency and using those risk assessments to guide  agency processes for public health product preparation, ATSDR cannot  provide reasonable assurance that its products have undergone the  appropriate level of monitoring and review. If established, a risk  assessment process could be used to determine the proper level of  scrutiny throughout the initiation, development, and review and clearance  phases, including whether or not a product should undergo external peer  review, thereby ensuring that this determination is made consistently  across the agency. Basing this process on a set of criteria, and  documenting and tracking risk assessment decisions in agency systems,  should help ensure an effective process. ATSDR has already incorporated  some of the elements of risk assessment in the existing DHAC triage  process for categorizing the priority of work requests.
Finally, because the agency’s policies lack guidance for management  about their role in monitoring product development, ATSDR cannot be  sure that management has a clear understanding of the role they are  supposed to play in supervising a product’s preparation. Additionally,  ATSDR’s policies and procedures for the development of public health  assessments and health consultations do not require management’s  monitoring and approval of key components of a product during its  development. Without adequate monitoring by management during a  product’s development, product errors may not be caught or significant  publication delays may occur during the review and clearance phase,  potentially undermining public confidence in the agency’s products.
Policies and procedures alone, however, cannot ensure the quality of  ATSDR’s public health products and, as noted above, internal controls  provide only reasonable assurance, not absolute assurance, that an  agency’s objectives are being achieved. Issues outside of the influence of  management, such as human mistakes, judgment errors, or acts by  employees to circumvent management control, could also affect ATSDR’s  product quality. Nonetheless, improving ATSDR’s policies and procedures  regarding public health product preparation would help the agency  provide greater assurance to those inside and outside the agency of the  quality of these products.
section_title: Recommendations for Executive Action
paragraphs: To strengthen ATSDR’s policies and procedures, and ensure that they  provide reasonable assurance of public health product quality, we  recommend that the director of ATSDR take the following two actions:    Develop policies and procedures to ensure that a risk assessment is  conducted at the time site-specific work is initiated, and that any assigned  risk level be reevaluated throughout product preparation to ensure that it  remains appropriate.
Revise existing policies and procedures, or develop new guidance, to  provide documented direction for various levels of management on their  roles and responsibilities in the monitoring of all products prior to review  and clearance, such as requirements for management monitoring and  approval of key components of these products.
section_title: Agency Comments and Our Evaluation
paragraphs: ATSDR reviewed a draft of this report and provided written comments,  which appear in appendix I. While ATSDR neither agreed nor disagreed  with our recommendations and did not address them directly, in its  comments ATSDR stated that the agency has begun to incorporate our  recommendations.
Although ATSDR did not comment directly on our recommendation that  the agency conduct a risk assessment at the time site-specific work is  initiated and reevaluate the assessment throughout product preparation, in  its comments ATSDR stated that senior management was looking into  formalizing and unifying coordination, triage, and prioritization of all  incoming requests across the agency. ATSDR also acknowledged a need to  make its prioritization process more explicit throughout the agency. It is  imperative that ASTDR formalize its processes agencywide and ensure  that its processes include a risk assessment to determine the proper level  of scrutiny a product should receive throughout its preparation, including  whether or not it should undergo external peer review.
Related to our recommendation that ATSDR revise or develop policies and  procedures to include direction for management in monitoring products  prior to review and clearance, ATSDR noted that its process to formalize  and unify coordination, triage, and prioritization of all incoming requests  was expected to include the specification of management and staff roles  and responsibilities from initiation through publication. It is important that  ATSDR take this step in order to help ensure that management has a clear  understanding of their responsibilities in supervising a product’s  preparation.
In its comments, ATSDR noted that multiple guidelines are used to  conduct its work and it uses an issues management process for agency  risk management. Our findings document these guidelines and the issues  management process, and describe their limitations in establishing  effective information flow among all levels of management and staff, in  providing a comprehensive assessment and categorization of the risk of  work being initiated at the agency, and in identifying the roles and  responsibilities of management.
ATSDR also acknowledged that it would benefit from formalizing  additional internal controls, and stated that as part of its review of the  agency clearance policy it was incorporating a way to sample documents  that were previously cleared to ensure that scientific principles are being  applied across all divisions. ATSDR also stated that it expected Sequoia,  its agencywide electronic project tracking system, to be fully implemented  by the end of the year.
Finally, ATSDR commented that we did not assess public comment as a  part of our report, which it indicated was a critical component of the  agency’s quality assurance process. While we agree that public comment  provides valuable input on those products which are subject to a public  comment period, it augments but does not substitute for thorough internal  review of a product or formal, external peer review of a product by  carefully selected experts.
ATSDR also provided technical comments, which we incorporated as  appropriate.
As we agreed with your offices, unless you publicly announce the contents  of this report earlier, we plan no further distribution of it until 30 days  from the report date. At that time, we will send copies to the Secretary of  Health and Human Services and other interested parties. The report also  will be available at no charge on GAO’s Web site at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-7114 or bascettac@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made major contributions to this report  are listed in appendix II.
section_title: Appendix I: Comments from the Agency for Toxic Substances and Disease Registry
section_title: Appendix II: GAO Contact and Staff Acknowledgments
subsections: section_title: Contact
section_title: Acknowledgments
paragraphs: In addition to the contact named above, key contributors to this report  were Karen Doran, Assistant Director; George Bogart; Amy C. Leone;  Roseanne Price; Mario D. Ramsey; Christina Ritchie; and Carla Willis.