Overlap and Duplication: Federal Inspections of Entities Registered with the Select Agent Program
Published: 2013-01-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Between 2009 and 2011, there were roughly 374 entities across the United States conducting research with select agents such as anthrax, which have the potential to threaten health and safety. Inspections are one means of ensuring safety and compliance with regulations. However, several federal agencies--CDC, APHIS, DOT, DHS, and DOD--conduct such inspections, creating significant potential for overlap and duplication of effort. In this context, GAO was asked to assess (1) the extent of overlap and potential duplication in federal inspections of select agent entities, (2) the costs of such overlap and effects on laboratory operations, and (3) actions to reduce the costs and negative effects of any overlap. To answer these objectives, GAO analyzed agency data, surveyed entities, held focus groups with lab staff, and interviewed agency officials.
section_title: What GAO Found
paragraphs: About 15 percent of entities registered to work with select agents were subject to inspection overlap (multiple federal agencies inspecting within a 2-year period). Entities experiencing overlap tended to be larger ones, with more laboratories, principal investigators, and staff. Although there was overlap between Department of Transportation (DOT) inspections and those of the Centers for Disease Control and Prevention (CDC) and the Animal and Plant Health Inspection Service (APHIS), they were generally not duplicative because specific inspection activities tended to differ, according to GAO's survey of entities experiencing overlap. For example, DOT inspections tended to focus on transportation issues, such as checking hazardous materials and transportation security plans, rather than general biosafety issues. The Department of Homeland Security (DHS) and Department of Defense (DOD) inspections, however, tended to be more duplicative with those of CDC and APHIS. For example, both review the same documents, require safety and security demonstrations, conduct inventory inspections and personnel interviews, and provide corrective action plans. While inspections are important for safety and compliance, there is no value added when federal agencies are expending resources to conduct the same work and, in some cases, reinspecting before entities have had time to respond to findings from a previous inspection.
The costs of overlapping federal inspections and effects on lab operations are difficult to quantify because agencies and entities generally do not track them and some costs are not quantifiable. Although GAO could not quantify the portion of federal and entity costs directly attributable to overlap, it could quantify the costs of inspections in general. According to agency data, the approximate overall federal cost for fiscal years 2010 and 2011 inspections was $2.1 million dollars. On average, the entity costs per inspection were nearly $15,000, and staff time per inspection was 380 hours, according to the GAO survey. While surveyed entities reported that inspections can help correct deficiencies and improve accountability, most reported moderate to significant nonquantifiable costs of inspections due to loss of productivity and delays in research. In addition, according to surveyed entities, overlapping inspections negatively affected lab productivity, staff morale, available time to complete research, and the research schedule. Because many of these entities are federal laboratories or are funded through federal grants, these costs are passed on to the federal government.
Actions to reduce the costs of overlapping and duplicative inspections include better coordination among federal agencies and greater consistency in the application of standards, according to various experts and surveyed entities. CDC has taken actions to better coordinate inspections with other agencies, for example, by increasing the use of joint inspections. But such actions, including joint inspections, do not fully address the negative effects of multiple inspections if agencies apply inconsistent standards and develop separate reports of findings. Well-trained inspectors, who apply consistent standards, are also needed. Collectively, these actions would reduce the negative effects of overlap and duplication and could increase agencies' acceptance of each other's inspection results.
section_title: What GAO Recommends
paragraphs: GAO recommends that CDC and APHIS work with DHS and DOD to coordinate inspections and ensure consistent application of inspection standards.
HHS, USDA, DHS, and DOD generally agreed with our recommendations and noted various actions they have already taken, or plan to take, to coordinate inspection efforts.

section_title: Letter
paragraphs: Between fiscal years 2009 and 2011, there were 374 entities registered  with the federal Select Agent Program (SAP) that were home to over  1,900 biosafety laboratories in the federal, academic, and private sectors  across the United States. These laboratories conduct research with  biological agents or toxins, known as “select agents,” such as anthrax,  which have the potential to severely threaten public health and safety,  animal and plant health, and animal and plant products. Consequently,  these issues are a key concern of entities and federal agencies  conducting, funding, or overseeing select agent work. Inspections are one  mechanism for ensuring that entities registered to work with select agents  comply with Select Agent Regulations (SAR) for biosafety, biosecurity,  and biocontainment. However, there are multiple federal agencies, as  well as state, local, and professional organizations, that currently inspect  these entities, creating the potential for overlap and duplication of effort.
Various federal agencies have the authority to inspect entities that  possess, use, or transfer select agents. The Department of Agriculture’s  (USDA) Animal and Plant Health Inspection Service (APHIS) and the  Department of Health and Human Services’ (HHS) Centers for Disease  Control and Prevention (CDC) inspect entities under the SAR to ensure  compliance with the SAP.  In addition, the Department of  Transportation’s (DOT) Pipeline and Hazardous Materials Safety  Administration’s (PHMSA), under DOT’s Hazardous Materials  Regulations, has the authority to regulate the safe and secure  transportation of all hazardous materials. Under the SAR, any entity that  uses, possesses, or transfers select agents must (1) register with CDC or  APHIS and (2) establish and implement plans to ensure the safety and  security of its activities. Prior to issuing a certificate of registration, CDC  and APHIS may inspect and evaluate the premises and records to ensure  compliance with the SAR. In addition, CDC and APHIS may inspect any  site, for compliance and other purposes, where activities regulated by the  SAR are conducted. DOT’s PHMSA also has the authority to perform  inspections pertaining to the transport of biological agents. Other federal  agencies also inspect laboratories that work with select agents, including  the Department of Defense (DOD) and its Army, Navy, and Air Force  Inspector General (IG) offices and the Department of Homeland Security  (DHS). DOD inspects to ensure compliance with federal, DOD, and  military service component requirements as well as contract  requirements. DHS conducts periodic inspections to ensure compliance  with funding requirements of entities for which they have funded research.  These inspections, however, also generally cover compliance with the  SAR. While DOD and DHS inspections assess compliance with the SAR,  these departments do not have authority to regulate all entities  possessing, using, or transferring select agents. Rather, these  departments issue their own agency regulations and directives that allow  them to inspect entities they own or fund (see table 1 for major areas  covered in agencies’ inspections). In addition to these federal agencies,  several others, such as the Occupational Safety and Health  Administration (OSHA), the National Institutes of Health (NIH), and the  Food and Drug Administration (FDA), may also inspect registered entities  for a variety of reasons, such as ensuring employee safety and health or  compliance with standards for recombinant DNA research. We did not  include them in our review because these inspections generally do not  focus on biosafety and biosecurity issues.
In March 2011, we issued our first annual report to Congress on federal  programs, agencies, offices, and initiatives, either within departments or  government-wide, that have duplicative goals or activities. The report  highlighted instances of (1) overlap—two or more agencies or programs  engaged in similar activities or providing similar services to similar  beneficiaries; (2) duplication—two or more agencies or programs  engaged in the same activities or providing the same services to the  same beneficiaries; and (3) fragmentation—more than one federal  agency engaged in similar activities, but not coordinating efforts. The  number of federal agencies involved in inspections of select agent entities  heightens the potential for overlap and duplication. Moreover, the  increasing number of inspections further increases the potential for  overlap. From fiscal year 2009 through 2011, CDC, APHIS, DOD, DHS,  and DOT collectively conducted over 800 inspections of select agent  entities.
In this context, you asked us to assess (1) the extent of overlap and  potential duplication in federal agencies’ inspections of entities that work  with select agents, (2) the costs of overlapping federal inspections and  effects on laboratory operations, and (3) actions to reduce the costs and  negative effects of overlapping inspections.
To answer these objectives, we interviewed agency officials and reviewed  pertinent legislation, regulations, and agency documents. We also  analyzed agency information on inspectors’ total compensation (salary  and benefits), hours spent on inspection activities, the associated travel  costs for inspections for fiscal years 2010 and 2011, and inspection data  to identify entities that had been inspected by more than one agency in a  2-year period (at any point between fiscal years 2009 and 2011). Using  the definition of overlap from our 2011 report, in the context of SAR  inspection requirements, we defined overlap as two or more federal  agencies inspecting the same entity on separate occasions within a 2- year period. Because CDC and APHIS manage the SAP jointly and  conduct joint inspections, where applicable, we collapse these agencies  in reporting on overlap to show where DOT, DHS, or DOD inspections  overlapped with either a CDC or APHIS inspection. We developed a  detailed list of inspection activities and identified duplication where the  two overlapping inspections covered the same activities. From the 374  entities registered to work with select agents between fiscal years 2009  and 2011, we identified entities that had been inspected on separate  occasions by more than one agency in a 2-year period. We then  conducted a web-based survey of all such entities to collect information  on (1) the extent to which there was duplication in specific inspection  activities related to the preparation, execution, and closeout phases of the  two inspections undergone, (2) the costs and operational effects of  inspections, (3) the solutions for mitigating the negative effects of multiple  inspections, and (4) the positive and negative aspects of joint inspections.  Our survey was sent to the responsible official (RO) of each entity that  was subject to overlapping inspections, and we received an 85 percent  response rate. To gather preliminary information and develop our survey,  we conducted focus groups with about 50 laboratory workers from entities  that have experienced federal inspections, including ROs, biosafety  officers, principal investigators, and technical staff from federal,  academic, and private entities that work with select agents. We  interviewed officials with CDC, APHIS, DHS, DOD, and DOT and  examined inspection files to understand general and agency-specific  inspection activities. We also spoke with professional organizations, such  as the American Biological Safety Association (ABSA), the American  Society for Microbiology, and experts in the safety of select agent  laboratories. We determined that the data we used were sufficiently  reliable for our purposes. (See app. I for a more-detailed discussion of our  scope and methodology.)
We conducted our work from March 2011 through January 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: In 2007, GAO reported on issues associated with the proliferation of  biosafety laboratories in the United States. In 2009, we noted that while  proliferation of these laboratories was taking place in the federal,  academic, and private sectors across the United States, the federal  oversight of these laboratories was fragmented—there was not a single  federal agency to provide oversight. As a result, numerous federal  agencies could be involved in separate and independent inspections of  these entities and their associated laboratories. The various agencies that  have a role in the oversight of select agent entities and can conduct  inspections are shown in figure 1.
CDC and APHIS have regulatory authority to assess compliance with  biosafety, biosecurity, and biocontainment requirements. Under the  current SAR, entity registration must be renewed every 3 years, and CDC  or APHIS may conduct an on-site inspection before the award of a new  certificate of registration or the renewal of an existing registration. These  inspections generally cover all aspects of the SAR. As a matter of policy,  to ensure that the entity is compliant with the SAR, CDC or APHIS  inspects the premises and records of applicants, including a review of all  required plans, before issuing the initial certificate of registration. In  addition, CDC or APHIS may conduct inspections to (1) respond to  concerns about an entity’s compliance, (2) verify corrections of  deficiencies identified through inspections or accomplishment of  Performance Improvement Plan goals; or when (3) modifications are  made to the entity’s registration, (4) a new building or laboratory is added,  (5) a higher-risk agent or toxin is added, (6) a change is made in security  infrastructure or policy and procedures, (7) a theft, loss, or release  incident occurs, or (8) a violation is reported. Any entity where select  agents are possessed, used, or transferred must allow CDC and APHIS  to inspect, with or without prior notification. In addition to CDC or APHIS  inspections, the SAR requires each registered entity to conduct annual  self-inspections under the direction of the entity’s RO.
subsections: section_title: DOT Inspections
paragraphs: DOT has the primary authority to regulate the safe and secure transport  of all hazardous materials shipped intrastate, interstate, and in foreign  commerce. Infectious substances, which include select agents, are  regulated as hazardous materials by DOT. DOT regulates select agents  in commercial transportation to, from, and within the United States, and  its oversight extends to all parts of the hazardous materials transportation  system, including classification of materials, packaging, handling, moving,  loading, and unloading of hazardous materials shipments in commerce.  PHMSA is the component of DOT responsible for this oversight. As its  authority is limited to transportation, its focus is on the shipping and  packaging aspects of biosafety and biosecurity requirements.  Consequently, entities that are active in the transfer of select agents may  have their shipping and handling facilities inspected by PHMSA.
section_title: DHS and DOD Inspections
paragraphs: DHS and DOD own or fund research at select agent entities, and these  entities may undergo additional inspections from these agencies as part  of the conditions of funding or based on the safety and security policies of  the parent agency. For example, entities that receive funds from DHS to  conduct laboratory work involving select agents are subject to on-site  compliance reviews and inspections by the DHS Regulatory Compliance  Office. In addition, select agent laboratories operated by DOD are subject  to inspections by the service IGs and other commands.
DHS established a regulatory compliance program to facilitate  department-wide implementation of and compliance with DHS policies for  biosafety, select agent security, and the care and use of animals in  research. DHS’s select agent research is subject to regulatory oversight  by CDC and APHIS. In addition, entities that receive funding from DHS to  conduct laboratory work involving select agents are subject to on-site  compliance reviews and inspections based on DHS Management  Directives. According to DHS, it “conducts significant additional  oversight because of unique sensitivities related to biodefense research,  as distinct from conventional public health research, and a desire to  ensure complete transparency for senior management of the department  about all ongoing biodefense efforts.” DHS also has responsibility for  ensuring biosecurity compliance of DHS funded research under  biodefense weapon treaties.
Through its long-standing Chemical and Biological Defense Program, the  DOD supports research on detection, identification, and characterization  of biological threats and the development of countermeasures against  those threats. DOD research activities take place at numerous facilities,  including military-owned entities as well as entities in academia and  private industry supported by contracts. DOD entities that are registered  with CDC or APHIS are required to follow SAR requirements as well as  service-specific requirements derived from DOD requirements. DOD- related select agent entities can therefore be subject to inspections from  CDC, APHIS, the service IGs, and other commands. Specifically, the  Department of the Army Inspector General (DAIG) conducts inspections  of five Army and contractor-owned entities, the Navy Medical IG (MEDIG)  conducts inspections of two Navy laboratories located in the United  States and three overseas, and the Air Force Materiel Command IG  conducts inspections of one Air Force select agent facility. In addition to  DAIG inspections, Army facilities are also subject to command/program  office (PEO) reviews by (1) Army Material Command Surety Management  Review Team, (2) Army Test and Evaluation Command (ATEC) Surety  Management Review Team, (3) Army Medical Command (MEDCOM)  Surety Management Review Team, and (4) Joint Program Office for  Chemical and Biological Defense (JPEO-CBD) Surety Management  Review Team. The DAIG and command/PEO teams each have a 2-year  inspection cycle and stagger the inspections so that each entity is  inspected once per year.
section_title: Other Federal Agencies Involved in Inspections
paragraphs: Other federal agencies, such as the NIH and USDA’s Agriculture  Research Service, have their own internal offices that may perform  inspections in addition to those performed by CDC or APHIS as part of  the SAP. In many cases, these agencies have internal regulations or  policies that are more prescriptive than the CDC or APHIS regulations,  according to the Trans-Federal Task Force. In addition, as the agency  responsible for the general oversight of workplace safety in the United  States, OSHA has oversight authority for the safety and health of workers  in all workplaces that fall under its jurisdiction, including individuals who  work with hazardous biological agents or toxins in high- and maximum-  containment research facilities. This includes jurisdiction over the safety  and health of workers employed by privately owned entities and, under  certain circumstances, federal high-containment facilities.
section_title: Inspection Activities
paragraphs: Inspection activities are generally conducted in three phases: (1)  preparation, (2) execution (the actual inspection), and (3) closeout  (postinspection activities). In order to prepare for an inspection, entities  conduct a variety of activities, such as responding to requests for  documents related to the SAR; reviewing and updating guidance, records,  and plans; and checking security and medical certifications of inspectors.  In the execution phase, inspection activities include, for example, tours of  the facility, document reviews, inventory audits, interviews of laboratory  staff, equipment tests, and examinations of physical security and shipping  and receiving of select agents. The closeout phase includes activities  such as discussing the inspection findings and report, developing  corrective action plans, and providing verification of corrective actions.  (For a detailed list of inspection activities included in the survey, see   app. II.)
Inspections consist of an extensive review of laboratory safety and  security. CDC and APHIS use specific checklists, which they developed  from the SAR, OSHA regulations, NIH guidelines for recombinant DNA  research, and the Biosafety in Microbiological and Biomedical  Laboratories (BMBL) manual, to guide their inspections. The BMBL  provides guidance for standards of practice for laboratory principles,  practices, and procedures. Other inspecting agencies also use the same  or similar checklists for their inspections. DOT inspections, for example,  are derived from PHMSA’s regulations for hazardous materials shipping,  packaging, testing, certification, safety and security, and record keeping.  The scope of a DOD inspection covers a wide range of functional areas,  such as security, safety and occupational health, surety management,  emergency response, occupational medical requirements, and  transportation.
section_title: Overlap and Duplication
paragraphs: The number of federal agencies involved in inspections of select agent  entities increases the potential for overlap and duplication. Further adding  to the potential for overlap is the increase in the number of inspections.  These inspections have increased for a variety of reasons, such as  heightened security concern in response to the events of September  2001, as well as an increase in select agent research and the number of  agencies funding the research. For example, in addition to announced  inspections that generally take place with registration certification and  renewal, the number of unannounced CDC inspections substantially  increased from fewer than 5 in 2006–07 to nearly 80 in 2012, according to  estimates from CDC. And while some federal agencies have increased  the use of joint inspections, in which two agencies are on-site for an  inspection at the same time, it is unclear whether this will address all of  the negative aspects of duplication if compliance is still assessed  separately.
section_title: Some Overlap and Duplication Exist in Inspections of Select Agent Laboratories
paragraphs: About 15 percent of the 374 entities that were registered to work with  select agents between fiscal years 2009 and 2011 were subject to  inspection overlap. This means that in a 2-year period, these entities were  inspected by more than one federal agency for biosafety, biosecurity, and  biocontainment compliance. In addition, specific inspection activities  were often duplicative, according to our survey of the 55 entities that were  subject to overlapping inspections. While the percentage of entities  affected by inspection overlap was relatively small, the entities affected  tended to be larger ones that work with a greater variety of agents and  with more laboratories, principal investigators, and laboratory staff. For  example, entities with five or more laboratories are more likely to be  subject to overlap than entities with fewer than five laboratories,  controlling for biosafety level (BSL) and the number of agents. Similarly,  entities working with five or more select agents are more likely to be  subject to overlap than entities working with fewer than four select agents,  controlling for the number of laboratories and BSL. As a result, the  overlap affected roughly a third of all laboratories, principal investigators,  and lab staff. Moreover, DOD-owned select agent entities are also subject  to inspections from internal inspecting entities. These inspections can  overlap with inspections from CDC, APHIS, and the service IGs.
Concerning duplication, the entities often prepare the same documents  for inspectors to review; conduct the same facility tours, inventory  inspections, and personnel interviews; and go through the same exit  conference and corrective action plan processes, according to survey  results. Inspections conducted by DOT, however, were generally not  duplicative of CDC and APHIS inspections because specific inspection  activities tended to differ. According to surveyed entities, DOT inspections  tended to focus narrowly on transportation issues—such as (1) checking  hazardous material and transportation security plans and (2) verifying the  labeling, testing, and assembly of United Nations (UN) certified  packagings—rather than general biosafety and biosecurity compliance.  DHS and DOD’s DAIG inspections tended to be more duplicative with  those of CDC and APHIS, in that numerous activities were the same  across these inspections, according to survey results.
subsections: section_title: Larger Entities Are More Likely to Experience Overlap in Inspections
paragraphs: Fifty-five, or 15 percent, of entities registered between fiscal years 2009  and 2011 were subject to overlapping inspections. Although the overlap  appears to affect only a small portion of registered entities, these entities  are home to 645 laboratories—roughly a third of all laboratories involved  in select agent work (see fig. 2 for details). On the basis of logistic  regression analysis, we found that the likelihood that an entity was  subject to overlap in federal inspections depends on the number of  laboratories, the highest BSL of its laboratories, and the number of select  agents. Specifically, entities with five or more laboratories are more likely  to be subject to overlap than entities with fewer than five, controlling for  BSL and the number of agents.
In addition, entities with at least one BSL-4 lab are more likely to be  subject to overlap than entities without a BSL-4 lab, controlling for the  number of laboratories and the number of select agents. In addition,  entities working with five or more select agents are more likely to be  subject to overlap than entities working with fewer than four select agents,  controlling for the number of laboratories and BSL. Finally, entities subject  to overlap have, on average, more staff and more principal investigators  (see fig. 3).
Overlap in inspections occurred most frequently between CDC or APHIS  and (1) DOT (34 overlapping inspections), (2) DAIG (13 overlapping  inspections), and (3) DHS (8 overlapping inspections). Considering only  CDC inspections, overlap occurred with 29 DOT inspections, 13 DAIG  inspections, and 3 DHS inspections. However, overlapping inspections do  not necessarily result in duplication.
section_title: DOT Inspections Are Less Duplicative Than DHS and DOD Inspections
paragraphs: While the overlap in inspections occurs most frequently between CDC or  APHIS and DOT, these inspections are not necessarily duplicative  because the specific inspection activities tend to differ. The activities are  likely to differ because the purpose of DOT inspections is narrowly  focused on issues and areas related to the transport of select agents,  whereas CDC and APHIS inspections are more-broadly focused. While  there is less-frequent overlap between CDC or APHIS inspections and  those of DHS or DOD, these inspections tend to be more duplicative, in  that the inspection activities are similar, according to survey results.
According to DOT inspection data, DOT’s PHSMA conducted 83  inspections of select agent entities between fiscal years 2009 and 2011.  DOT coordinates its inspections with CDC and APHIS under a working  Memorandum of Understanding (MOU), which spells out the framework  and responsibilities for the exchange and protection of information on  transfer of select agents. Under the MOU, CDC and APHIS annually  provide DOT with a list of registered entities that have transferred select  agents during the 12 preceding months. DOT, in turn, notifies CDC and  APHIS of any anticipated inspection of an entity 30 days before the  inspection and also provides them with the inspection results. According  to DOT officials, they try to coordinate joint inspections whenever  possible. After CDC provides a list of scheduled inspections, DOT informs  CDC about which inspections it will join. According to DOT officials,  where possible, they coordinate joint inspections with CDC because (1)  they learn things from each other, (2) with both agencies there at the  same time it is more encompassing view of the process, and (3) it brings  greater sophistication to inspections. Such coordination appears effective,  given survey results of entities that experienced DOT inspections that  overlapped with CDC or APHIS inspections. These entities reported little  duplication in specific activities in the preparation, execution, and closeout  phases of inspections (see table 2). Among the activities that tended to  be duplicative were verifying the medical, security, or other credentials of  inspectors in the preparation phase; participating in interviews with  inspectors in the execution phase; and holding exit conferences with the  inspecting agency in the closeout phase. Among the activities that tended  to be DOT-specific were checking hazardous materials and transportation  security plans and verifying the labeling, testing, and assembly of UN- certified packagings.
DHS identified 42 government, university, private, and not-for-profit  entities that were receiving DHS funding for work involving select agents.  DHS’s Regulatory Compliance Office conducted or participated in 19 on- site inspections of 13 of these entities between fiscal years 2009 and  2011. Of those 19 inspections, 8 were joint inspections, including: 5 joint  inspections with CDC; 1 joint inspection with APHIS; and 2 joint  inspections with both CDC and APHIS. According to DHS officials, there  are numerous reasons for DHS to visit a select agent entity that has also  been inspected by CDC or APHIS. These reasons include, for example,  CDC or APHIS placing the entity on a Performance Improvement Plan or  making substantial recommendations to correct regulatory  noncompliance. If it appears that the compliance issues identified by CDC  or APHIS could affect the DHS program, DHS may also make a site visit  to understand the effect on DHS research and to assist the entity in  mitigating the effect through appropriate corrective action.
According to DHS officials, their inspections are fundamentally different  from CDC and APHIS inspections. DHS describes its compliance  inspections as broader in some ways than those of CDC or APHIS,  because they are designed to ensure that DHS-sponsored research  activities not only comply with select agent requirements, but with other  relevant regulations and guidelines as well. According to DHS officials,  CDC and APHIS, as regulatory agencies, conduct more comprehensive  inspections at the institutional level. DHS compliance inspections,  however, go beyond compliance with select agent regulations to include  general biosafety, animal care and use, research protocols and  procedures, institutional review and oversight (for example, Institutional  Biosafety Committees), and adherence to best practices. Nevertheless,  survey responses from entities that experienced DHS inspections that  overlapped with CDC or APHIS inspections reported some duplication in  specific activities in the preparation, execution, and closeout phases of  inspections (see table 3). Among the activities that tended to be  duplicative were verifying medical, security, or other credentials of  inspectors and arranging staff availability in the preparation phase;  participating in interviews with inspectors in the execution phase; and  holding exit conferences and developing and implementing corrective  action plans in the closeout phase.
According to DOD inspection data, the DAIG conducted 16 biosurety  inspections of its Army and contractor-owned entities between fiscal  years 2009 and 2011, 5 of which were conducted jointly with CDC. Of the  16 DAIG inspections, 13 overlapped with an inspection from another  federal agency. The Navy IG conducted one inspection and the Air Force  IG conducted two inspections between 2009 and 2011. These  inspections did not overlap with an inspection from another federal  agency. With respect to specific activities in the preparation, execution,  and closeout phases of inspections, entities that experienced DAIG  inspections that overlapped with CDC or APHIS inspections also reported  substantial duplication in inspection activities (see table 4). Among the  inspection activities that tended to be duplicative were arranging staff  availability in the preparation phase, holding entry meetings and escorting  inspectors in the execution phase, and holding exit conferences in the  closeout phase. However, according to DAIG officials, as required by  Army directives and regulations, its biosurety inspections are more- comprehensive than CDC and APHIS inspections, covering biosafety,  biosecurity, biocontainment, personnel reliability program (PRP),  transportation, occupational medical requirements, and emergency  response exercises. While DAIG officials acknowledge a level of overlap  with CDC and APHIS inspections in terms of verifying compliance with  standards, DOD and the Department of the Army have developed specific  requirements to implement those standards. In addition, until recently,  CDC and APHIS inspections did not look at facility or department-specific  requirements such as PRP.
DOD-related select agent entities are also subject to inspections and  reviews from internal organizations, which can overlap with inspections  from CDC, APHIS, and the service IGs. For example, in addition to  service IG inspections, DOD entities undergo Biosurety Management  Reviews (SMR) and Biosurety Staff Assistance Visits (SAV). The SMRs  are meant to verify the entity is managing its biosurety program according  to standards. They allow the command to see how the surety program is  being managed and where there may be deficiencies. According to Army  lab officials, however, these inspections tend to be identical to service IG  biological surety inspections. SAVs are an opportunity for the command  to assist in fixing deficiencies or other lacking areas, and usually take  place before another major inspection. SAVs are not biosurety  inspections, nor are they required under DOD regulations. However,  SAVs tend to be treated similarly to service IG inspections, according to  Army lab officials. While SMRs and SAVs may be handled like an  inspection with written reports of perceived deficiencies for which the  entity makes a formal response and corrections for the identified  deficiencies, DOD officials noted that entities may be self-imposing  requirements or practices not required by regulation or the inspecting  agency. As a result, these internal reviews can represent an additional  area of overlap for DOD-owned and DOD-operated entities. For example,  as shown in table 5, an Army entity underwent eight inspections or  reviews, five of which were for compliance with select agent regulations  between fiscal years 2009 and 2011.
According to an official of this particular Army entity, many of the major  command reviews and visits use up time and resources, fixing issues that  are not value-added, or that are not looked at by other major inspections.  In many cases, inspectors are focusing on minor issues, leaving the  laboratory open for larger deficiencies on the major inspection. According  to the DAIG Chief of Technical Inspections, a major cause of overlapping  internal inspections is that there is no single entity within the DOD  overseeing or coordinating inspections and that each service validates  compliance very differently. The Army requires each major command to  have its own internal biosurety team and, because the entities feel they  should be prepared for higher-level reviews, the teams conduct  inspections in preparation for higher-level inspections. For Army labs, the  DAIG inspects an entity every 2 years, an internal command surety team,  such as ATEC, conducts an SMR every 2 years (alternating with the  DAIG inspection), and SAV reviews may be conducted before any  inspection (DOD or otherwise). In an example of an extreme case, an  Army select agent laboratory that conducts recombinant DNA research  for DHS-funded projects and frequently transfers select agent materials to  collaborators could theoretically be inspected by CDC/APHIS, DAIG, NIH,  DHS, and DOT all within the same year.  This could significantly hinder  critical research productivity at the inspected laboratory because of the  time dedicated to inspections, according to the report of the Working  Group on Strengthening the Biosecurity of the United States. This is  somewhat in contrast with inspections of the two Navy and one Air Force  entities. For example, while the Navy can perform SAVs prior to  inspections, they are not required like Army’s SMRs. In addition, rather  than conducting its own inspection, the Air Force accepts the CDC  inspection results and adds its own PRP review. This limited coordination  among the inspection agencies was noted in the Working Group report.
section_title: While Difficult to Quantify, Overlap Adds to the Cost of Inspections and Can Negatively Impact Laboratory Operations
paragraphs: The costs of overlapping federal inspections and effects on lab operations  are difficult to quantify because (1) agencies and entities generally do not  track the costs or effects of inspections and (2) some costs are not  quantifiable. Nevertheless, the costs and effects on lab operations are  significant when considering (1) the cost of inspections to federal  agencies, (2) both the quantifiable and nonquantifiable costs to the entity,  and (3) surveyed entities’ perceptions of the negative effects of  overlapping inspections on lab operations. Although we could not quantify  the portion of federal and entity costs directly attributable to overlap, we  could quantify the costs of inspections in general. For example, we  estimate that for fiscal years 2010 through August 2011, individual  agencies’ total inspection costs ranged from approximately $22,400 to  over $900,000, according to agency data on the hours spent on  inspection activities, inspector compensation (salaries and benefits) per  labor hour, and travel. The approximate overall federal cost for fiscal year  2010 and 2011 inspections was over $2.1 million dollars. On average, the  entity costs per inspection were nearly $15,000 and 380 hours in staff  time, according to our survey. The quantifiable cost of an inspection to a  select agent entity depends on the number of laboratories and select  agents, the complexity of the entity’s mission, its location, and whether it  has a history of problems or violations of select agent regulations. In  inspections of larger entities, there are higher inspection costs and a  greater likelihood for overlap. The costs of overlap are therefore most  likely higher as well. Entities also reported moderate to significant  nonquantifiable costs of inspections when it comes to loss of productivity  and delays in research. While inspections can help entities correct  deficiencies, improve inventory management and accountability, and  justify the need for resources to improve operations, most surveyed  entities reported that overlapping inspections have negative effects on lab  operations. According to surveyed entities, overlapping inspections  negatively affected lab productivity, staff morale, available time to  complete research, and the research schedule. And according to at least  one-fifth of surveyed entities, overlapping inspections negatively affected  the physical viability of inventory, staff retention, and competitiveness for  research funds. Because many of these entities are federal laboratories  or are funded through federal grants, these quantifiable and  nonquantifiable costs are passed on to the federal government.
subsections: section_title: While the Cost of Multiple Inspections Is Difficult to Quantify, Data from Inspecting Agencies and Entities Indicate They Are Significant
paragraphs: Obtaining an accurate and complete picture of the costs of multiple  inspections is difficult because entities generally do not track the costs of  inspections and some of those costs are nonquantifiable. Nonetheless,  federal agencies do incur quantifiable costs, including salaries, travel, and  training of inspectors, and must purchase inspection equipment and pay  staff to engage in inspection activities as opposed to research or other  routine activities. Entities may also incur nonquantifiable costs of multiple  inspections, such as loss of productivity, delays, and decreased time  available to complete research. In addition, because many of these  entities are federally owned or funded, some portion of this cost is passed  on to the federal government. These costs are affected by the number of  inspectors, the time spent on an inspection, and the size of the entity  being inspected. The larger the entity—in terms of laboratories, staff,  and select agent research—the greater the cost of inspections. Given that  inspections cost more for larger entities and overlap occurs more often for  larger entities, the cost of overlap is greater than it would be if it were  evenly distributed across entities of various sizes.
The approximate direct federal cost for fiscal years 2010 and 2011  inspections was over $2.1 million, ranging from approximately $22,400 at  DOT to over $900,000 at CDC, according to agency data on the hours  spent on inspection activities, inspector compensation (salaries and  benefits) per labor hour, and travel. Specifically, APHIS’s total labor,  travel, and other costs for inspections were $265,792; DOD’s costs were  $697,744; DOT’s costs in 2010 were approximately $22, 444; and CDC’s  costs were $903,475 (see table 6 for agency inspection costs). DHS  estimates their inspection costs at about $250,000 for the 2-year period.  Although we did not estimate indirect costs, the federal government also  incurs costs from inspections because many select agent entities are  either federally owned or funded. Consequently, the costs to entities,  described in the sections below, also accrue to the federal government.
The cost of inspections to entities is also difficult to accurately determine  because entities generally do not track inspection cost. However,  according to focus group participants and entities we surveyed, entities  do incur quantifiable and nonquantifiable costs with each inspection,  some of which can be significant. Focus group participants and surveyed  entities reported quantifiable costs, such as purchasing inspection  equipment and salaries for staff involved in preparing for, carrying out,  and responding to an inspection. Entities also reported less-easily  measured, nonquantifiable costs, such as loss of productivity, delays, and  decreased time to complete research, as well as loss of specimen viability  from repeated thawing and freezing.
Staff time and lab resources are required for inspections, and that burden  is increased with overlapping inspections. The actual cost of an  inspection to a select agent entity will depend on the number of  laboratories and select agents, the complexity of the entity’s mission, its  location, and whether it has a history of problems or violations of select  agent regulations. However, because overlap in inspections tends to  occur more often for larger entities, the overall costs of overlap for these  laboratories are likely higher as well. Entities that experienced an  overlapping inspection spent, on average, 380 hours and nearly $15,000  in staff time to engage in a federal inspection, according to survey data on  the number of hours spent to prepare for, carry out, and close out  inspections and the hourly salaries of laboratory staff involved (see table  7 for average costs across occupational groups). See app. I for an  explanation of how we calculated these costs.
Much of the time spent on inspections takes place in the preparation  phase, according to focus groups of lab staff. For example, staff from  Army laboratories noted that there is a period of 3 to 4 months of intense  preparation for DAIG inspections and months of follow-up. During the  preparation phase, lab staff perform numerous activities, including  updating standard operating procedures and other documents and  records; verifying inspectors’ health records and clearances; turning in  unused equipment; holding meetings to prepare for the inspection;  checking chemicals and agents in inventory; checking lab equipment; and  inspecting, cleaning and painting floors, walls, and desk space. According  to some focus group participants, in the weeks preceding an inspection,  research is suspended and all staff time is directed toward inspection  efforts. Scheduling for inspections can also create conflicts because  experiments must be scheduled around the inspection, and key staff must  be available during the inspection, regardless of personal plans or  schedule.
During the inspection, lab staff are involved in activities that take time and  resources. For example, staff conduct safety training for inspectors (how  to wear safety suits and respirators and blood-borne pathogen and  internal requirement training), issue safety equipment to the inspectors,  clear inspectors through security, and provide escorts for inspectors.  Safety equipment, such as personal protective equipment (PPE), gowns,  and footies are provided to inspectors at the entities’ expense, and these  costs increase with larger inspection parties and when there is overlap  because the equipment can not be shared or reused. Staff time for  escorts can be significant in some cases. For example, Army lab  inspection teams can have as many as 20 personnel and stay as long as  2 weeks. According to one Army laboratory, it spent more than 7,305  labor hours in activities related to preparing for, executing, and  responding to six inspections during 2009 at a cost of $350,640. During  2010, the laboratory estimated it spent 4,082 hours on these activities for  three inspections, at a cost of $195,456. In addition, when joint  inspections take place, the size of the inspection team may be too large  for some entities to manage. For example, according to one survey  respondent, the entity requested DHS and CDC not conduct a joint  inspection because the inspection teams are too large to handle at one  time. According to another survey respondent, the respondent’s  laboratory had to arrange for additional personnel to escort large teams of  inspectors for multiple inspections. Laboratories also spend time and  resources in the closeout phase of the inspection. For example, upper- level staff attend meetings to discuss inspection results, and staff time  and effort are required to respond to oral and written findings that may  change or be added to even after the completion of the inspection. Of  particular concern was that frequent inspections keep lab staff in a  constant mode of preparing for the next inspection while responding to  the last one, inhibiting their ability to respond to inspection findings and  conduct research.
Laboratories also incur nonquantifiable costs from inspections, which can  be exacerbated by overlapping inspections. While these costs tend to be  small, according to most surveyed entities, for some they can be  moderate to significant when it comes to loss of productivity, decreased  time available to complete research, and delays (see table 8). Other costs  noted by surveyed entities include stress and anxiety for lab staff,  contractor and overtime costs, HVAC and equipment testing costs, and  loss of focus and manufacturing time.
Focus group participants noted a variety of nonquantifiable costs, such as  the loss of productivity, recertifying equipment or bringing it offline for  inspection, and loss of agent viability from repeated thawing and freezing  of agents. Although only 5 (11 percent) of surveyed entities reported that  reduced viability of inventory was a moderate to significant cost, a  preliminary study at the Dugway Army Proving Ground on the viability of  agent vials that had undergone multiple inspections found reductions in  agent viability, with a 100 percent loss of viability in a few cases. While  the costs of reduced or lost viability are difficult to determine, a strain that  existed in only a single vial would be impossible to replace. Furthermore,  loss of agent viability can damage research opportunities and  experimental findings.
Focus group participants also noted that multiple inspections reduce an  entity’s competitiveness for grant money, although most surveyed entities  found this a negligible cost. In addition, inspection time must be factored  into grant proposals. And because inspections affect the research cycle,  the time it takes to complete research and the overall costs of the  research increase when entities experience multiple inspections. For  example, some participants noted the costs of their entity’s grant  proposals are higher than others’ because of the “overhead” costs they  must build in for their multiple biosafety and biosecurity inspections. In  addition, the time to conduct research is affected when laboratories have  to suspend research for inspections. Undergoing multiple inspections  exacerbates this problem. Some participants noted that the granting  community knows that if laboratories are frequently inspected, the work  will not get done by the desired deadlines, affecting the communities’  willingness to award grants to these entities.
section_title: While Multiple Inspections Can Improve Lab Operations, They Can Also Negatively Impact Lab Operations
paragraphs: Inspections are essential in ensuring biosafety and biosecurity  requirements are met and SAR regulations are followed. Most surveyed  entities reported that multiple inspections positively affect (1) actions to  correct deficiencies, (2) coverage in helping to identify problems, (3) the  strength of inventory management and accountability, and (4) justification  for additional resources to improve operations. However, many entities  also noted multiple inspections negatively affect lab operations. In  particular, most surveyed entities reported that multiple inspections  negatively affect (1) lab productivity, (2) staff morale, (3) the time to  complete research, and (4) the research schedule. While some entities  reported negative effects to (1) the physical viability of inventory, (2) staff  retention, and (3) staff recruitment, a greater majority of entities reported  these issues were unaffected by multiple inspections (see table 9 for  effects of multiple inspections).
Focus group participants also noted some positive aspects of multiple  inspections, among them having an “extra set of eyes” or a layer of  oversight on laboratory operations to identify areas of need. In addition,  inspection report findings, can provide funding justification for resources  and staff, validate good laboratory practices, and identify needed quality  assurance improvements. However, some focus groups participants also  noted that the amount of time spent on inspections slows down the  science and that while inspections may help get the laboratory renovated,  they do not actually improve the science. Others participants noted that  oversight functions in laboratories, such as quality assurance, are  growing faster than the research community. Finally, one survey  respondent noted that multiple inspections do not allow the laboratory to  have the time necessary to implement lessons learned in a meaningful  time frame.
section_title: A Coordinated Approach with Consistently Applied Standards is Recommended by Earlier Reports and Surveyed Entities
paragraphs: Actions to reduce the costs and negative effects of overlapping and  duplicative inspections include better coordination and greater  consistency in the application of standards, according to various experts  and surveyed entities. Both the HHS-USDA Trans-Federal Task Force  and the Executive Order Working Group on Strengthening the Biosecurity  of the United States recommended enhancing the coordination of  biosafety oversight activities, including inspections. In addition, our earlier  work on select agent laboratories recommended a single coordinating  agency as a means to improve coordination. Accordingly, CDC and  APHIS have taken steps to better coordinate inspections with other  agencies, for example, by increasing the use of joint inspections, signing  MOUs for sharing inspection information with other agencies, establishing  an inspector training program for federal partners, and developing a  common “playbook” for inspection of registered entities. Such  coordination efforts are important steps in reducing overlap and  duplication. However, MOUs and joint inspections may not fully address  the negative effects of overlap and duplication if inspectors are still  applying standards inconsistently and preparing separate reports of  findings. Specifically, according to surveyed entities, standards must be  applied more consistently between agencies and from one inspection to  another. It is this inconsistent application of standards that exacerbates  the negative effects, including costs, of overlapping inspections.  According to surveyed entities, the most effective actions for greater  consistency in the application of standards include (1) ensuring inspectors  are well trained and experienced, (2) establishing a single set of  inspection standards that all agencies accept, (3) providing an opportunity  to discuss, clarify, and rebut inspection findings, and (4) training  inspectors to one set of standards with requirements for noncompliance  findings (see table 10). Well-trained inspectors, who are able to apply  consistent standards, would reduce the negative effects of overlapping  inspections. In particular, they might reduce overlap by allowing federal  agencies to accept each other’s inspection results. Such a result is  facilitated by joint training, according to DOD officials. Without a  consistent standard, however, highly trained inspectors would not be  effective, according to some surveyed entities. In addition, according to  most surveyed entities, a 3-year federal inspection cycle for select agent  entities was reasonable. Focus group participants also suggested  options for minimizing the potential for overlapping or duplicative  inspections and the associated burden, such as having a single  inspecting agency whose findings are accepted by all other agencies and  improving the knowledge and skills of inspectors.
In addition to scaled responses, surveyed entities provided written  suggestions for reducing the negative effects of multiple inspections.  While several entities expressed support for a single inspecting agency or  joint inspections, others felt that addressing overlap alone would be  insufficient when the same agency applies different standards in each  inspection. In support of inspections being conducted by a single agency,  entities noted that at least the inconsistent application of standards across  agencies and frustration over trying to comply with two different agency  regulations would be minimized. Surveyed entities highlighted the  inherent conflict that arises when agencies come to different conclusions  about an entity’s compliance. For example, one noted that because there  is no agency “in complete control,” entities cannot determine which  agency is correct when there is conflict in inspection reports. And a DOD  entity wondered about the implication for CDC—which approved the SAP  registration—if DAIG recommends the shutdown of a lab. While some  thought an independent agency or an inspection czar would be a useful  neutral party, others expressed concern that such an entity would just add  another level of bureaucracy or require unnecessary new legislation.  Because of entities’ familiarity with CDC, one noted that they would not  want to have a single agency unless it was CDC. However, another noted  it was uncertain how useful a single agency with authority to audit would  be because there would be a period of mixed messages while each  agency has its say. And so far, the entity had not seen government  agencies “play well together as a team.”
In addition, while some thought joint inspections might address the  negative effects of overlap, others thought they would exacerbate the  problem because agency coordination—in terms of consistent  interpretations and applications of standards—was still a concern. In  support of joint inspections, some felt they facilitated agency coordination  and reduced duplication. Others noted that joint inspections are a better  use of time and resources, reducing disruption and burden. In addition,  joint inspections could help bring more uniformity and clarity for the  inspection agencies and entities. For example, according to DAIG  officials, they have taken steps to address these issues in joint  inspections with CDC and APHIS. During joint inspections, the agencies  discuss and resolve differences of interpretation so there is one  “inspector” face to the inspected facility, and except for major findings,  DAIG does not duplicate the findings of the other agency. However,  noting recent inspections by two different agencies, one entity reported  that having them at the same time would have made the inspection longer  and more grueling. Joint inspections also require the entity to provide  escorts and enough PPE for everyone entering the facility, a large  expense when considering how many PPEs and escorts are needed for  some joint inspections. In addition, (1) combined inspections may still  result in multiple reports the entity must respond to and (2) having two  teams do what a single team can do is still an additional burden on the  taxpayer. Nonetheless, the burden that results from inconsistent  application of standards could be minimized if agencies (1) conducting  joint inspections issued a single report of findings or (2) instead of  conducting a separate or joint inspection, accepted each others’  inspection findings in lieu of conducting their own inspections.
subsections: section_title: Training to Consistently Apply Standards Is Needed
paragraphs: Regardless of whether a single agency is responsible or joint inspections  are conducted, numerous respondents focused on the need for consistent  application of standards by well-trained inspectors. Respondents’  concerns focused on the inconsistent application of standards that can  occur, not only between agencies, but from one inspection to another.  While the problem may be exacerbated by overlapping inspections, the  burden results from inconsistency, not overlap per se. For example, one  entity noted that the biggest problem with multiple inspections by different  agencies is that each has its own agenda, so establishing inspection  standards that are accepted by all agencies would have the greatest  effect on reducing the negative effects of multiple inspections. Another  noted that a common set of requirements and interpretations would  significantly reduce variation in compliance assessments. Reflecting on  the additional burden to Army laboratories, one entity noted, “Ideally the  set of standards that would be accepted would be the same federal  standard that all organizations, not only DOD organizations, would be  held to. This would not only reduce the burden of multiple inspections but  also level the playing field between DOD and academia which would  increase collaboration between laboratories.” Some noted that this might  facilitate sufficient trust for agencies to accept each other’s inspection  results. If all inspections were based on the same standard, with a  checklist to keep them consistent, and all inspectors were trained  together and required to use the published standard and checklist, the  need for multiple inspections would be decreased or eliminated. Despite  the support for greater consistency, some cautioned against the  application of a “checklist mentality.” For example, one surveyed entity  noted that too many inspectors for their select agent program had been  rigid and unbending, sticking to a checklist rather than common sense.  Training can support efforts to consistently apply standards, while still  allowing the flexibility for inspectors to have essential discussions with  entities to establish rapport, exchange information, and understand why  certain procedures and policies are in place in any given facility.
Surveyed entities offered a variety of other suggestions, such as (1)  limiting the number of inspectors and days for inspections; (2)  streamlining inspections, including the number of inspectors on the team,  and coordinating areas of focus within the team to lessen the burden on  the laboratories; (3) having an advisory group of ROs to give feedback to  the inspection agency, regulatory agency, or GAO; (4) dividing inspection  elements among the inspecting agencies, on the basis of their strengths;  (5) staffing inspection teams for consistency and historical knowledge; (6)  training all inspectors together to help ensure consistent inspections; and  (7) giving entities credit for moving in the right direction. While none of the  written comments suggested reducing the inspection cycle, almost half of  surveyed entities noted that a 3-year federal inspection cycle for select  agent laboratories was reasonable. The agencies have taken some steps  to address these concerns. For example, after we had initiated our work,  CDC and APHIS convened an Interagency Working Group that includes  representatives from DHS and DOD, as well as other federal agencies.  So that effective oversight can be achieved with minimal disruption, the  working group is developing procedures and policies to better coordinate  the inspections of entities that are federally owned or funded, improve  information sharing between agencies, and implement other activities. For  example, the working group has initiated a joint inspection program  through which it has, so far, conducted 24 joint inspections. In addition, it  has MOUs with DHS and DOD to share inspection data and an inspector-  training program to provide the knowledge, skills, and experience to  federal agencies to enable them to conduct “internal” inspections of  registered entities they own or fund or to conduct joint inspections with  CDC or APHIS.
section_title: Conclusions
paragraphs: Inspections are important for safety and compliance and can help  improve laboratory procedures, infrastructure, and security. However, the  value of inspections may be diminished when federal agencies are (1)  expending resources to conduct the same or similar work and (2)  burdening entities with overlapping or duplicative inspections. While one  could argue that more-frequent inspections might be necessary to better  ensure safety—in particular for larger entities—there is no apparent  value-added when specific inspection activities are duplicative and occur,  in some cases, before entities have had time to respond to findings from  a previous inspection. In addition, most surveyed entities reported a  federal inspection schedule of once every 3 years was reasonable; more  frequent—especially duplicative—inspections waste federal dollars and  can negatively affect lab operations. This effect on lab operations also  wastes federal dollars because many of the laboratories are federally  funded through grants or appropriations. To improve interagency  coordination and reduce the potential for overlap and unnecessary  duplication, a single coordinating agency could be helpful. Such a  coordinating agency need not be responsible for conducting all  inspections. Rather, where agencies can demonstrate that they meet  SAR standards for their inspections, their inspections could be used in  lieu of other agency inspections. Currently, the primary agencies with  regulatory authority, CDC and APHIS, have taken steps toward better  coordination through the interagency working group. However, CDC and  APHIS have not been officially charged with overseeing or coordinating  all inspection efforts and the other federal agencies may still inspect an  entity regardless of how recently another agency has conducted an  inspection. Moreover, consistent application of inspection standards— both across and within the agencies—is needed to reduce the negative  effects of multiple inspections on lab operations. Further, cross-agency  training efforts could facilitate consistent learning and application of such  standards. Agencies could then better target their inspection time and  resources, and entities could better prepare for and respond to federal  inspections.
section_title: Recommendations for Executive Action
paragraphs: In order to eliminate overlapping and potentially duplicative inspections,  as well as reduce the burden of such overlap and duplication on select  agent entities, we recommend that CDC and APHIS, as the primary  agencies with regulatory authority, work with DHS and DOD to (1)  coordinate inspections of select agent entities and (2) where possible,  use mechanisms such as (a) joint inspections with a single report of  findings, (b) acceptance of each other’s inspection results rather than  independent inspections, and (c) cross-agency training opportunities to  ensure consistent application of biosafety, biosecurity, and  biocontainment inspection standards.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to HHS, USDA, DOT, DHS, and DOD  for review and comment. DOT did not provide any comments.  In its  written comments, reproduced in appendix III, HHS agreed with our  recommendations. HHS noted that DOT, DHS, and DOD have different  authorities than the federal select agent program (SAP), as we had  outlined in the background section of this report. HHS stated that it  already has a number of activities underway related to reducing overlap  and duplication in inspections. Specifically, through the working group for  Optimizing the Security of Biological Select Agents and Toxins in the  United States, HHS’s Federal Select Agent Program has developed  procedures and policies to better coordinate inspections of federally  owned or funded entities and to improve information sharing between  departments and agencies. HHS initiated three programs to accomplish  these goals: (1) a joint inspection program, (2) inspection information  sharing MOUs, and (3) an inspector training program. HHS has so far  conducted 24 joint inspections, signed inspection information sharing  MOUs with five agencies, and trained five individuals from DHS and four  individuals from DOD.  HHS also plans another training session for spring  2013 to educate federal agencies about recent revisions to select agent  regulations. HHS also provided technical comments, which we  incorporated as appropriate.
In its written comments, reproduced in appendix IV, USDA agreed with  our recommendations. Although USDA noted that it did not believe its  activities were overlapping with DOD and DHS because of differences in  inspection authorities and agency missions, it noted that it had a number  of activities underway to reduce overlap and duplication in inspections.  Along with CDC—USDA’s partner in overseeing the select agent  program—USDA noted that it is conducting joint inspections to minimize  the number of select agent inspections, signing information sharing  MOUs with several agencies to share inspection data, and conducting  inspector training programs to provide knowledge, skills, and experience  to federal partners. USDA also provided technical comments, which we  incorporated as appropriate.
In its written comments, reproduced in appendix V, DHS agreed with our  recommendations.  DHS notes that its Regulatory Compliance Office  collaborates with officials and inspectors at CDC and APHIS to plan joint  inspections, share information, and consult on inspection findings in order  to reduce the burden on inspected institutions. Specifically, DHS signed  an MOU with CDC and APHIS that outlines how the parties will  coordinate joint inspections in order to reduce the burden on entities and  to facilitate coordination of oversight efforts between the agencies. In line  with our recommendations, DHS now accepts the results of inspections  conducted jointly with CDC or APHIS instead of conducting independent  inspections or generating a DHS-specific report of findings. DHS has also  taken advantage of cross-agency training to ensure consistent application  of biosafety, biosecurity, and biocontainment inspection standards.
In its written comments, reproduced in appendix VI, DOD agreed with our  recommendations. DOD noted that certain DOD regulations require  compliance inspections for areas, such as the personnel reliability  program (PRP), that are in addition to the SAR. We recognize some  requirements are agency-specific and do not suggest that they be  eliminated. We do recommend, however, that the burden on entities be  reduced through coordination with other SAR inspection activities.  In line  with our recommendations, we are encouraged that DOD has several  initiatives to coordinate with CDC and APHIS to reduce overlap and  duplication of inspections.  For example, the Army and Navy have signed  information sharing memorandums of agreement and understanding with  CDC and APHIS to coordinate inspections of select agent agencies,  through which they conducted the first CDC, DOD, and Army joint  inspection in spring 2011. Although the Air Force does not have an MOU  with CDC or APHIS for its one facility, it accepts the CDC inspection  results and performs its own PRP inspection. During joint inspections,  DOD notes that it develops an inspection plan with CDC, APHIS, and the  entity, and coordinates with the agencies to minimize entries into  inspected areas by inspecting an area together. While two separate  reports are written, the DAIG notes that it generally does not replicate  CDC findings of noncompliance with SAR standards.  Findings are  discussed to ensure a common understanding of the standards and what  was observed, and to delineate between what is required by the SAR,  DOD, and DAIG.  While DOD notes that it is unclear whether it would or  could accept a CDC or APHIS inspection report in lieu of its own  inspection, these joint efforts are still new and the issue will evolve with  further collaboration. Additionally, DOD is revising its biological and  chemical agent security policies in order to harmonize them with the  recently revised SAR. DOD also noted other steps it has taken to  coordinate oversight efforts though the Working Group, such as  developing the charter and implementation plan for the current federal  “joint” inspections process, and developing tools, such as the inspector  training program and playbook, to administer the joint inspection process.  DOD also made some technical comments, which we incorporated as  appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretaries of Health  and Human Services, Agriculture, Transportation, Homeland Security,  and Defense, the appropriate congressional committees, and other  interested parties. In addition, the report will be available at no charge on  the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-2700 or kingsburyn@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix VII.
section_title: Appendix I: Scope and Methodology
paragraphs: To assess (1) the extent of overlap and potential duplication in federal  agencies’ inspections of entities that work with select agents, (2) the costs  of overlapping federal inspections and effects on laboratory operations,  and (3) actions to reduce the costs and negative effects of overlapping  inspections, we interviewed agency officials, reviewed pertinent  legislation, regulations, and agency documents. Specifically, we spoke  with officials from the Center for Disease Control and Prevention (CDC)  Division of Select Agents and Toxins, the Department of Agriculture’s  Animal and Plant Health Inspection Service (APHIS), the Department of  Transportation’s (DOT) Pipeline and Hazardous Materials Safety  Administration (PHMSA), the Department of Defense (DOD) military  service Inspector General’s offices, including the Department of the Army  Inspector General (DAIG), Navy Inspector General, and Air Force  Inspector General offices. We also spoke with interest groups, such as  the American Biological Safety Association (ABSA), and officials at  various Army, private, academic, and federal select agent entities.
To assess the extent of overlap and potential duplication in federal  agencies’ inspections of entities that work with select agents, we (1)  identified 374 entities registered to work with select agents between fiscal  years 2009 and 2011, (2) analyzed inspection data to identify registered  entities that had been inspected by more than one agency on separate  occasions in a 2-year period (at any point between fiscal years 2009 and  2011), and (3) surveyed those entities to assess the extent of duplication  in inspection activities. In order to operationalize overlap, we relied on our  2011 duplication report, in which we define overlap as multiple agencies  or programs having similar goals, engaging in similar activities or  strategies to achieve them, or targeting similar beneficiaries. CDC,  APHIS, DHS, DOD, and DOT inspections are directed toward a similar  goal—assessing biosafety and biosecurity compliance—and are  accomplished through a similar strategy, the inspection process. When  these inspections targeted the same entities, we counted the inspection  as overlapping. Specifically, we analyzed fiscal year 2009 through July  2011 inspection data from CDC, APHIS, DHS, DOD, and DOT, and  identified any instances where two different agencies had inspected the  same entity on separate occasions within 2 years of each other. We  counted joint inspections as a single inspection in our analysis, regardless  of the agencies involved. For example, joint APHIS/DHS inspections or  CDC/DAIG inspections were counted as one inspection. We chose the 2- year time frame because the Select Agent Regulations (SAR) require  certification renewal every 3 years and it is the policy of CDC and APHIS  to inspect the entity before recertifying, this represents a somewhat  conservative measure of overlap. Because CDC and APHIS manage the  Select Agent Program (SAP) jointly and conduct joint inspections, where  applicable, we collapse these agencies in reporting on overlap to show  where DOT, DHS, or DOD inspections overlapped with either a CDC or  APHIS inspection. We also relied on our 2011 report for our definition of  duplication—when two or more agencies or programs are engaged in the  same activities or provide the same services to the same beneficiaries.  The entities experiencing overlap were identified from the population of  374 entities registered with CDC or APHIS at any point in the 3-year  period. Some entities register and de-register each year, so the number  of entities registered at any point in the 3-year period will differ from a  single point-in-time number of registered entities. Using the number of  entities registered at any point in the 3-year period, the extent of overlap  as we have defined it is 15 percent. To estimate the likelihood that an  entity would experience overlap in federal inspections, we developed a  logistic regression model, using the following entity characteristics: the  number of laboratories, the highest biosafety level (BSL) of laboratories  within an entity, and the number of different select agents the entity works  with. The model did not include the number of staff or the number of  principal investigators, both of which are strongly correlated with the  number of laboratories. We chose to include the number of laboratories in  the model because it was a stronger predictor of the likelihood, on the  basis of the Wald chi-squared test, than these two characteristics.
To assess the extent of duplication in the overlapping inspections, we  surveyed the 55 entities that had been inspected by more than one  agency in a 2-year period (that is, the population experiencing overlap).  The web-based survey of all such entities gathered information on the (1)  extent to which there was duplication in specific inspection activities  related to the preparation, execution, and close-out phases of the two  inspections the entities had undergone, (2) costs and operational effects  of inspections, (3) solutions for mitigating the negative effects of multiple  inspections, and (4) positive and negative aspects of joint inspections  (see app. II for a copy of the survey).The survey was sent to the RO of  each entity, and we received an 86 percent response rate, with 47 of the  55 entities responding. To gather preliminary information and develop our  survey, we conducted focus groups with about 50 laboratory workers from  entities that have experienced federal inspections, including responsible  officials, biosafety officers, principal investigators, and technical staff from  federal, academic, and private entities that work with select agents. To  understand general and agency specific inspection activities, we  interviewed inspectors with CDC, APHIS, DHS, DOD service IGs, and  DOT and examined inspection files. We also spoke with professional  organizations such as the ABSA and the American Society for  Microbiology and experts in the safety of select agent laboratories.
We took several steps to ensure the reliability of survey responses and  the analytical process. To ensure the content and wording of the survey  were clear, accurate, unbiased, and nonburdensome, we solicited  subject-matter expert reviews from affected agencies (CDC, APHIS,  DAIG, and PHMSA), an interest group (ABSA), and an internal (GAO)  survey expert. We also pretested the survey with lab staff from a variety  of laboratories, including an Army lab, an academic lab, a private lab, and  a large federal lab. The survey was deployed through the web, and each  respondent had a unique user identification and password. To increase  the response rate, follow-up e-mail and telephone calls were made to  nonrespondents. Once we had reached an 80 percent response rate, we  reviewed responses for (1) item nonresponse, (2) obvious errors or  outliers in responses, and (3) “no” answers to question 1. We then  followed up with respondents, as necessary, to get additional information  and clarification. Changes identified through this process were recorded,  and appropriate response cleaning was conducted in the analysis. To  eliminate data-processing errors, the computer program that generated  the survey results was independently verified by an internal SAS expert  who was not involved in the engagement.
To assess the costs of overlapping federal inspections and the effects on  laboratory operations, we (1) analyzed agency budget data and (2)  gathered data from focus groups and our survey of entities on the costs  and effects of inspections. Specifically, to assess the costs to the federal  government, we requested information on the government and contracted  staff involved in inspections and their total compensation (salary and  benefits), the number of hours spent on inspection activities, and the  associated travel costs for inspections for fiscal years 2010 and 2011.  DHS was unable to provide detailed cost data as requested because their  inspections of select agent laboratories are provided under a contract that  does not identify labor hours, hourly contract rates, salaries, and benefits.  But DHS did provide an estimate of the cost of inspections for fiscal years  2010 and 2011. This estimate was based on assumptions about typical  staffing levels and costs for the most-common types of inspections,  multiplied by the number of inspections conducted in each fiscal year.  While DHS believes these estimations include travel costs, they do not  include training costs. DOT was not able to provide cost data that include  travel, because their system does not distinguish between inspection  travel and other travel. As a result, DOT inspection amounts reflect only  compensation costs. To assess the reliability of agency data on the costs  of inspections, we provided the agencies with a detailed data-collection  instrument with specific data requests and precalculated formulas. We  reviewed the data for obvious errors, compared these data with cost data  from earlier fiscal years, and followed up with officials to discuss data  reliability, significant changes across fiscal years, obvious errors, or  omissions. We determined these data were sufficiently reliable for our  purposes, which was to provide the approximate aggregate federal costs  of inspections for the four agencies.
To assess the costs to entities, we asked surveyed entities to provide  labor and cost information about the most recent inspection identified in  our overlap analysis (see app. II, question 6, for specific wording).  Specifically, we requested data for five occupational groups that tend to  be involved in federal inspections, including (1) ROs and Alternate  Responsible Officials, (2) Principal Investigators, (3) owners or  controllers, (4) laboratory staff, and (5) support staff (security,  administrative, maintenance, and information technology). We asked  entities to provide the number of staff involved and the total labor hours  spent by each occupational group, as well as the average salary for each  group. We used these data to develop overall averages of the personnel  and labor costs entities experience as a result of inspections by each one  of the five federal agencies in our review. For data reliability purposes, we  checked for outliers or obvious errors and followed up where such issues  were identified. We calculated the average cost of an inspection by  dividing the average (yearly) salary by 2080 to get an hourly rate, and  then multiplied that by the number of labor hours provided. We also  surveyed entities about nonquanitifable costs and the operational effects  of multiple inspections and analyzed comments related to costs from our  focus groups of lab staff.
To assess actions to reduce the costs and negative effects of overlapping  inspections, we interviewed agency officials and interest groups, reviewed  key reports addressing the issue, and analyzed comments related to  solutions from our focus groups of lab staff. In addition, in our survey, we  sought entities’ opinions about solutions for overlapping inspections,  including their experiences with joint inspections, which have been  proposed as a solution for minimizing inspection duplication.
We conducted our work from March 2011 through January 2013 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Survey of Select Agent Entities Experiencing Overlapping Inspections
paragraphs: Welcome to the GAO survey on inspections of Select Agent (SA) entities. To complete this survey, you may  consult with others in your lab, as necessary. Your responses to this web-based survey are essential in order for  us to provide complete and accurate information on these issues to Congress.
We will use your responses, together with the responses of other recipients, to develop aggregate statistics,  observations, and findings. The identities of individual respondents will not be disclosed in our final report.
For questions, contact Jason Fong at fongj@gao.gov or 202-512-xxxx.
If you experience technical problems with this web questionnaire, please contact Rebecca Shea at shear@gao.gov  or 202-512-xxxx.
Important! JavaScript must be enabled on your browser in order to use this web questionnaire.
This survey has 7 main sections covering the following key issues:   1. Inspecting Agencies   2. Inspection Activities:       a. Preparation       b. Execution       c. Close-out/response   3. Quantifiable Costs of Inspections   4. Non-quantifiable Costs of Inspections   5 Effects of Multiple Inspections   6. Solutions   7. Joint Inspections  Click on "Menu" to the left of this screen to display a navigation panel that can be used to move from section to  section.   Click on the "Summary" to the left of this screen to print a copy of the survey.
AAALAC Association for Assessment and Accreditation of Laboratory Animal Care   APHIS Department of Agriculture's Animal and Plant Health Inspection Service   BSC Biosafety cabinet   BSO Biosafety Officer   CDC Department Health and Human Services' Centers for Disease Control and Prevention   DAIG Department of the Army's Inspector General   DHS Department of Homeland Security   DOE Department of Energy   DOT Department of Transportation   EPA Environmental Protection Agency   NIH National Institutes of Health   OSHA Occupational Safety and Health Administration   PI Principle investigator   RO Responsible official (CDC designated)   SA Select Agent   UN certified packaging designed/tested in accordance with specifications of the United Nations Committee of  Experts on the Transport of Dangerous Goods  Unless otherwise specified, the questions in this survey apply only to inspections for biosafety and  biosecurity as required under Select Agent regulations or by those agencies that own or fund your entities'  Select Agent research.
Please check the following information for accuracy. If you need to make changes, please do so in the  appropriate editable field(s) below.
Respondent telephone   1. GAO received inspection data from CDC, APHIS, DHS, DOT, and DAIG, and identified your entity (lab) as  one that has been inspected by more than one of these agencies within the past 2 years. Specifically, these data  indicate your lab was inspected by the following agencies in the past 2 years: ______ in ______ and ______ in  ______.
Is this correct?    1a.  If the information above is not correct, please note the necessary corrections in the space below.   2. States, local governments, accrediting bodies (such as AAALAC and the Joint Commission), military  organizations (such as the Medical Command and Army Material Command), and other federal entities such as  NIH, OSHA, EPA, and DOE might also inspect SA registered labs.   If your lab was inspected in the past 2 years by any other entities (for SA compliance or other reasons),  please provide the name of the inspecting entity and inspection date.    2a.    Other inspecting entity and date of inspection:   2b.    Other inspecting entity and date of inspection:   2c.    Other inspecting entity and date of inspection:   2d.    Other inspecting entity and date of inspection:   2e.  If you would like to describe any areas of overlap between these "other" inspections and the inspections  your lab has received from CDC, APHIS, DAIG, DHS or DOT, please do so in the space below.
Inspections can be seen as taking place in 3 broad phases--(1) preparation, (2) execution, and (3)  closeout/response-- with specific activities occurring in each phase.     3. Thinking about your lab's inspections by ______ in ______ and ______ in ______, please indicate  whether you performed the following preparation activities as part of your routine activities, specifically  for each inspection, or not at all.   Please note: not all listed activities are required as part of routine activities or for inspections.    3a. Update and/or confirm records related to Select Agents and Toxins are current  and accurate (e.g., records for inventory, training, security, biosafety, etc.)
Did not perform   Not applicable   Don't know  3b. Prepare and send requested documents to inspecting agency (hardcopy or electronic)
Don't know  3c. Prepare all needed documentation for inspectors to review during visit (hardcopy or  electronic)
Did not perform   Not applicable   Don't know  3d. Revise and update written safety and security plans (e.g., biosafety,  incident response plans, cyber, personnel and physical security)
Don't know  3i. Decontaminate and prepare labs (i.e., go from a hot lab to a cold lab)
Did not perform   Not applicable   Don't know  3j. Check equipment calibration, and operation (e.g., pressure monitors, autoclaves, BSCs, etc.)
Did not perform   Not applicable   Don't know  3p. Check Hazmat training records/documents concerning transportation responsibilities are  current and available (e.g., general security and safety awareness, function specific)
Don't know  3r. Other (describe below)
Don't know  4b. Train inspectors on the use of Personal Protective Equipment (PPE)
Did not perform   Not applicable   Don't know  4c. Train inspectors on other issues (e.g., entry/exit, lab hazards, decontamination  requirements, emergency procedures, alarm sounds, etc.)
Did not perform   Not applicable   Don't know  4g. Participate in interviews with inspectors (to answer inspector questions and explain lab  procedures and operations)
Did not perform   Not applicable   Don't know  4h. Perform safety or security demonstrations (e.g., emergency response walk-through)
Did not perform   Not applicable   Don't know  4i. Other (describe below)
subsections: section_title: Inspection Activities: Close-out/Response
paragraphs: 5. Thinking about your lab's inspections by ______ in ______ and ______  in ______, please indicate whether you performed the following close- out/response activities specifically for each inspection.   Please note: not all listed activities are required for all inspections. 5a. Hold close out/exit conference with inspecting agency (where agency tells the entity the  results of the inspection in broad terms)
Don't know  5g. Bring equipment back online (e.g., HVAC, biosafety cabinets)
Did not perform   Not applicable   Don't know  5h. Other (describe below)
6a. Responsible Official (RO) and Alternate Responsible Official (ARO)
section_title: 6b. Principal Investigators (PI)
subsections: section_title: (1) Number of PIs involved
section_title: 6c. Owner/Controller
subsections: section_title: (1) Number of owner/controllers involved
section_title: 6d. Laboratorians
subsections: section_title: (1) Number of laboratorians involved
section_title: 6e. Support Staff (e.g., IT, Security, Admin, Animal Care, Maintenance, Janitorial)
subsections: section_title: (1) Number of support staff involved
section_title: Non-quantifiable Costs of Inspections
subsections: section_title: 7. In your opinion, how significant a cost, if any, did your lab incur in the
paragraphs: following areas as a result of the inspection by ______ in ______?  7a. Loss of productivity   7b. Delays in funded research   7c. Dollars lost in funded research   7d. Decreased time to complete research   7e. Reduced viability of inventory   7f. Reduced competitiveness for research  7g. Other costs 1 (describe below)   7h. Other costs 2 (describe below)   7i. Other costs 3 (describe below)
Other costs of inspections #1     Other costs of inspections #2     Other costs of inspections #3   8. If you would like to provide additional information about the costs of  inspections, or context for your responses, please do so in the space  provided below.   9. Thinking about the overall impact of having more than one federal inspection within the  past 2 years, has having multiple federal inspections positively or negatively affected your  lab operations in the following areas?  9a. Competitiveness for research funding   9c. Time to complete research   9e. Physical viability of inventory (i.e.,  resilience of the biological sample)   9i. Strength of inventory management and  9j. Actions to correct deficiencies   9k. Justification for additional resources to  9l. Ability to have specialized inspections in  which agencies focus on different areas  (e.g., biosafety, security, animal  husbandry, SA transport)   9m. "Coverage" in helping to identify  problems (i.e., more than one set of "eyes  on the problem")   9n. Other effect (describe below)   9o. Other--describe    10.
If you would like to provide context for your responses or additional  information about the positive or negative effects of multiple federal  inspections, please do so in the space provided below.   11. In your opinion, what is a reasonable federal inspection cycle for select  agent laboratories?   1.  2.  3.  4.  5.  6.
More than once per year   Once per year   Every other year   Every three years   As needed   Other  11a. Please explain your response.   12. In your opinion, how effective would the following actions be in reducing the negative  effects of multiple federal inspections? 12a. Conducting joint inspections   12b. Establishing an inspection czar (i.e., an  individual who directs and coordinates  SA inspection programs and strategy)   12c. Designating a single agency with  12d. Establishing a single set of inspection  standards that all agencies accept (in  areas where authorities overlap)   12e. Inspecting to requirements rather than  12f. Training inspectors to one set of  standards with requirements for non- compliance findings   12g. Ensuring inspectors are well  12h. Training lab staff about the different  agency missions and purposes for  inspecting   12i. Providing an opportunity to discuss,  clarify, and rebut inspection findings   12j. Other (describe)   12k. Other--describe   13. Please explain why you think the actions above would or would not be  effective at reducing the negative effects of multiple federal inspections.  14. If you would like to provide additional information about solutions for  reducing the costs of multiple inspections, please do so in the space  provided below.   15. Has your lab received a joint inspection (e.g., CDC/APHIS, CDC/DAIG,  CDC/DHS, APHIS/DHS inspect concurrently)?  1.  2.  3.
Yes   No (GO TO QUESTION 16)   Don't know (GO TO QUESTION 16)
section_title: 15a. If so, what are the positive aspects of joint inspections compared to
paragraphs: single agency inspections?   15b. What are the negative aspects of joint inspections compared to  single agency inspections?   16. If you would like to clarify any of your responses to this survey, or comment  on any other related topic, please do so in the space below.  17. If you have completed the questions in this survey, please move the check to  the "Completed" button below. (Your answers will not be used until you have  checked "Completed."   1.  2.
section_title: Getting a Copy of Your Responses
paragraphs: You may view and print your completed survey by clicking on the Summary  link in the menu to the left. When you are done, click on the "Exit" button below  to exit the survey and send your responses to GAO.
Thank you for your help.
section_title: Appendix III: Comments from the Department of Health and Human Services
section_title: Appendix IV: Comments from the Department of Agriculture
section_title: Appendix V: Comments from the Department of Homeland Security
section_title: Appendix VI: Comments from the Department of Defense
section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Sushil Sharma, Assistant  Director; Rebecca Shea; Jason Fong; Elaine Vaurio; Jim Ashley; and  Laurel Rabin made key contributions to this report.