FBI Laboratory: Chemistry and Trace Evidence Units Generally Adhere to Quality Standards, but Could Review More Examiner Testimonies
Published: 2017-06-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The FBI Laboratory, within the Department of Justice (DOJ), is responsible for analysis of forensic evidence for the FBI, other parts of DOJ, and domestic law enforcement agencies, among others.
GAO was asked to examine how the FBI Laboratory ensures the reliability of its forensic examinations, in particular within its Chemistry and Trace Evidence Units. For these two units, this report addresses (1) how the FBI Laboratory works to ensure quality in conducting forensic examinations, and (2) the extent to which it has taken steps to ensure adherence to the FBI Laboratory's quality standards.
GAO reviewed policies and procedures of the FBI Laboratory and its Chemistry and Trace Evidence Units; audit and accreditation reports from 2008, when the Laboratory was accredited to international standards, through 2015, the most recent available; the training records of all 47 staff who conducted casework in these two units from fiscal year 2011 to July 2016, the most recent available; and evaluation records for examiner testimonies and related laboratory reports in these two units from fiscal years 2011 to 2015, the 5 fiscal years prior to this review. GAO also independently sought to obtain testimony transcripts the FBI was unable to obtain for this period.
section_title: What GAO Found
paragraphs: The Federal Bureau of Investigation (FBI) Laboratory has a framework in place to help ensure quality in its forensic examinations of chemical and trace evidence. Based on accreditation results and GAO's review, the framework meets international and accreditation standards. The FBI Laboratory quality assurance framework consists of policies and procedures, quality assurance mechanisms, corrective actions, and training requirements that are designed to ensure quality in its forensic examinations and related activities (see figure). The framework includes policies, procedures, and training specific to each unit of the Laboratory, such as the Chemistry and Trace Evidence Units.
GAO found that the FBI Laboratory generally ensures the Chemistry and Trace Evidence Units adhere to quality standards for conducting forensic examinations, including conducting audits, implementing corrective actions, ensuring staff have appropriate training, and reviewing laboratory reports. However, the Laboratory's program to review examiner testimonies to ensure they are accurate and within the scientific limits of the given forensic discipline is limited by difficulties in acquiring testimony transcripts. Specifically, the Laboratory did not acquire transcripts and conduct internal evaluations for nearly half of the testimonies (78 of 164) given by Chemistry and Trace Evidence Unit examiners from 2011 through 2015, citing difficulties in locating transcripts and lack of response from courts. To better understand these factors, GAO sought and obtained almost half of the 78 transcripts (36 of 78). While attempting to obtain the remainder, GAO confirmed some of the difficulties identified by the FBI. Consistent with internal control standards, the FBI Laboratory could better ensure it obtains more transcripts for review by routinely capturing and using additional information that is critical to transcript acquisition, such as court jurisdiction and points of contact. Obtaining additional transcripts could help the FBI Laboratory expand its monitoring of examiner testimonies to help ensure the testimonies are accurate and within scientific limits, as defined by FBI and accreditation standards.
section_title: What GAO Recommends
paragraphs: GAO recommends that the FBI Laboratory's transcript acquisition procedure routinely capture and use additional information critical to transcript acquisition. The FBI concurred with our recommendation and described planned actions for implementation.

section_title: Letter
paragraphs: Forensic science plays a vital role in the United States criminal justice  system at the local, state, and federal levels. The Federal Bureau of  Investigation Laboratory (FBI Laboratory), within the U.S. Department of  Justice (DOJ), uses a variety of scientific and forensic techniques to  analyze evidence to help solve crimes.
The quality of the FBI Laboratory’s forensic work is important because  forensic evidence can influence whether a criminal defendant is acquitted,  convicted, or even charged. To ensure that justice is served, evidence  presented by the FBI Laboratory must be reliable and objective. However,  from 2009 to 2012, DNA testing revealed that three men had been  wrongfully convicted based at least in part on overstated FBI examiner  testimony that exceeded the limits of science. As a result, in 2012, the  FBI together with DOJ and in partnership with the National Association of  Criminal Defense Lawyers and the Innocence Project initiated a review of  FBI microscopic hair comparison analysis cases conducted prior to  2000. As of April 2015, the review found that at least 90 percent of trial  transcripts the FBI analyzed as part of the review contained erroneous  statements. Further, the review found that 26 of 28 FBI agents/analysts  provided either testimony with erroneous statements or submitted  laboratory reports with erroneous statements.
You asked us to examine how the FBI Laboratory ensures the reliability of  its forensic examinations, in particular within its Chemistry and Trace  Evidence Units. This report addresses (1) how the FBI Laboratory works  to ensure quality in conducting forensic examinations of chemical and  trace evidence, and (2) the extent to which the FBI Laboratory has taken  steps to ensure that the Chemistry and Trace Evidence Units adhere to  the Laboratory’s quality standards.
To address how the FBI Laboratory works to ensure quality in conducting  forensic examinations of chemical and trace evidence, we identified the  standards, policies, and procedures to which the FBI Laboratory, as a  whole, and the Chemistry and Trace Evidence Units, in particular, must  adhere. Specifically, we reviewed FBI policies and procedures related to  the Laboratory’s quality assurance and corrective action processes, such  as the FBI Laboratory Quality Assurance Manual and Laboratory  Operations Manual. We also reviewed the laboratory standards issued  by the International Organization for Standardization and the International  Electrotechnical Commission (ISO/IEC), and by the accrediting bodies for  the FBI Laboratory and Chemistry and Trace Evidence Units—the  American Society of Crime Laboratory Directors/Laboratory Accreditation  Board (ASCLD/LAB) and the American Board of Forensic Toxicology.
To assess the extent to which the FBI Laboratory has taken steps to  ensure that the Chemistry and Trace Evidence Units adhere to the  Laboratory’s quality standards, we examined reports of all internal audits,  external audits, and corrective actions issued by the Chemistry and Trace  Evidence Units and accrediting bodies from calendar years 2008 through  2015 to identify trends and determine the effectiveness of the quality  assurance process. We also examined the individual training records of  all 47 examiners and technicians that conducted casework or provided  testimony in the disciplines practiced by the Chemistry and Trace  Evidence Units from fiscal year 2011 to July 2016 to assess the extent to  which the staff have been trained and qualified to conduct and report the  results of forensic examinations in accordance with FBI policies and  accreditation standards.
We further examined the FBI Laboratory’s review process and testimony  monitoring program to determine the extent to which it has taken steps to  ensure that laboratory reports and testimony adhere to FBI standards for  quality. For all 164 testimonies provided by FBI Laboratory Chemistry  and Trace Evidence Unit examiners from fiscal years 2011 through 2015,  we reviewed related laboratory reports and available evaluations of the  testimonies to ensure that they had been reviewed in accordance with  FBI Laboratory and accreditation standards. Specifically, we reviewed  the laboratory reports to ensure that they had undergone the proper  technical and supervisory reviews. We further reviewed evaluations of  Chemistry and Trace Evidence Unit examiner testimonies to understand  the breadth and use of evaluations provided by external parties, such as  court officials, and internal evaluations of testimony transcripts conducted  by FBI Laboratory management and subject matter experts. We also  determined whether the Laboratory had obtained all transcripts for  testimony provided by Chemistry and Trace Evidence Unit examiners during this period, and whether Laboratory officials had reviewed the  obtained transcripts according to FBI policies and accreditation  standards. Furthermore, we attempted to obtain any transcripts that the  FBI Laboratory was unable to obtain by searching the Public Access to  Court Electronic Records system (PACER) and contacting the courts or  court reporters directly.
To inform our review for both objectives, we conducted interviews with  FBI officials responsible for overseeing the Chemistry and Trace  Evidence Units and quality assurance processes, and we visited the FBI  Laboratory at Quantico, Virginia, to observe examples of examination  methods used by the Chemistry and Trace Evidence Units. We also met  with a representative from ASCLD/LAB to discuss the organization’s  accreditation requirements and methodology for assessing the FBI  Laboratory for accreditation. In addition, for background purposes, we  met with officials from the Department of Commerce’s National Institute of  Standards and Technology (NIST) to discuss ongoing forensic science  efforts.
We conducted this performance audit from October 2015 to June 2017 in  accordance with generally accepted government auditing standards.  These standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: FBI Laboratory
paragraphs: The FBI Laboratory, located in Quantico, Virginia, provides a full range of  forensic services including forensic examinations, technical support, and  expert witness testimony. As of fiscal year 2015, the FBI Laboratory had  an operational budget of approximately $102.4 million and a staffing  budget of approximately $97.9 million. The FBI Laboratory employed  651 full-time staff at the end of fiscal year 2015 across five sections:  Biometrics Analysis, Forensic Response, Forensic Science Support,  Scientific Analysis, and the Terrorist Explosive Device Analytical Center.
The units discussed in this report, the Chemistry and Trace Evidence  Units, are both located within the Scientific Analysis Section.
The Chemistry Unit conducts scientific analyses in four disciplines:  general chemistry, toxicology, paints and polymers, and metallurgy. The  Chemistry Unit had an operational budget of approximately $807,500 and  a personnel budget of approximately $3.2 million in fiscal year 2015, and  employed 15 caseworking staff. The Trace Evidence Unit conducts  scientific analyses in three disciplines: hair and fiber, geology and  mineralogy, and forensic anthropology. The Trace Evidence Unit had an  operational budget of approximately $122,700 and personnel budget of  approximately $2.6 million in fiscal year 2015, and employed 16  caseworking staff. The combined operational budgets of the Chemistry  and Trace Evidence Units accounted for about 20 percent of the Scientific  Analysis Section’s fiscal year 2015 operational budget and 1 percent of  the FBI Laboratory’s overall operational budget.
In fiscal year 2015, the Chemistry Unit received 324 requests for  examinations, and the Trace Evidence Unit received 637 requests for  examinations. From fiscal years 2011 to 2015, the total number of  requests for examinations submitted to the Chemistry and Trace  Evidence Units declined from 1,159 requests to 961 requests.
section_title: International Standards and Accreditation
paragraphs: A testing and calibration laboratory may opt to follow international  standards to ensure that it is technically competent, has a management  structure, and is able to generate technically valid results. The  International Organization for Standardization (ISO) and the International  Electrotechnical Commission (IEC) develop and publish standards for  various industries worldwide, including laboratories. Laboratories can  use the ISO/IEC 17025:2005 General Requirements for Competence of  Testing and Calibration Laboratories standard to govern operations.
Accreditation bodies can use the ISO/IEC 17025:2005 standard as the  basis to accredit participating laboratories. The ISO/IEC 17025:2005  standard specifies the general requirements for laboratories to  competently carry out tests, including sampling. The standard specifies,  among other things, that a laboratory shall have a quality manual and a  quality policy statement, define the roles and responsibilities of  management, and have policies and procedures that institute many parts  of a quality assurance framework. For example, the standard requires  laboratories to have policies and procedures to handle nonconformities in  their work, including a corrective action process; conduct periodic internal  audits and management reviews; and ensure that staff performing testing  are qualified on the basis of appropriate education, training, experience,  and/or demonstrated skills.
For the period of our review’s focus from 2008 to 2015, the American  Society of Crime Laboratory Directors/Laboratory Accreditation Board  (ASCLD/LAB) accredited the FBI Laboratory to demonstrate that it met  the ISO/IEC 17025:2005 requirements, as well as additional ASCLD/LAB  requirements specific to forensic science laboratories, or the  ASCLD/LAB-International Supplemental Requirements for the   Accreditation of Forensic Science Testing Laboratories. ASCLD/LAB  accredits forensic science laboratories in specific disciplines.
ASCLD/LAB accredits laboratories on a 5-year cycle. The accreditation  process begins with ASCLD/LAB ensuring that the policies and  procedures of a laboratory meet the ISO/IEC and ASCLD/LAB standards.  The laboratory undergoes a full accreditation assessment consisting of a  document review of policies and procedures, meetings with the laboratory  director and other appointed individuals, an on-site assessment, and  corrective actions to address identified nonconformities. ASCLD/LAB then  provides an assessment report and certifies that the laboratory is  accredited, if applicable. Following the full on-site assessment, the  laboratory must provide a performance declaration to ASCLD/LAB  periodically, participate in and abide by the rules of the ASCLD/LAB  proficiency testing program, disclose all significant events and  nonconformities, and undergo scheduled surveillance visits.  ASCLD/LAB determines the frequency of performance declaration  submissions based on the performance of the laboratory during the  previous accreditation cycle. One scenario is the submission of a  performance declaration only when a laboratory does not undergo an on- site visit.
section_title: Legal Standards for Assessing the Reliability of Forensic Evidence in Court
paragraphs: Federal jurisdictions and states use different standards for determining  the reliability of expert testimony in court. Under the Federal Rules of  Evidence, Rule 702, an expert witness is considered qualified to testify if,  among other things, the testimony is the product of reliable principles and  methods. The 1993 Supreme Court case, Daubert v. Merrell Dow  Pharmaceuticals, Inc. (509 U.S. 579), significantly changed the  assessment of reliability of scientific evidence for federal trial courts,  making trial judges responsible for acting as gatekeepers to exclude  unreliable scientific expert testimony. The Daubert case listed factors for  judges to use in assessing the reliability of scientific expert testimony,  including (1) whether the expert’s technique or theory can be or has been  tested, (2) whether the technique or theory has been subjected to peer  review, (3) the known or potential rate of error of the technique or theory  when applied, (4) the existence and maintenance of standards and  controls, and (5) whether the technique or theory has been generally  accepted in the relevant scientific community. The Daubert factors are not  meant to be exclusive and other courts have found additional factors  relevant in determining the reliability of expert testimony. Most states  generally follow the Daubert standard, while some use the Frye test,  developed by the Court of Appeals of the District of Colombia in 1923  (Frye v. U.S., 293 F. 1013). The Frye test requires that the scientific  principle be generally accepted in the relevant scientific community. A  minority of states have not adopted either the Daubert factors or Frye test  and have their own rules for admitting scientific evidence into court.
section_title: Challenges With Certain Forensic Standards, Methods, and Practices
paragraphs: A method is the course of action or technique  followed in conducting a specific analysis or  comparison leading to an analytical result,  such as microscopic hair comparison.
Since the publication of the 2009 National Academy of Sciences (NAS)  report titled Strengthening Forensic Science in the United States: A Path  Forward, the standards, methods, and practices of the forensic science  community have been the subject of considerable discussion. The NAS  report highlighted that there is wide variability across forensic science  disciplines with regard to techniques, methodologies, reliability, types and  numbers of potential errors, research, general acceptability, and  published material. Further, while some of the disciplines and methods  used by the forensic science community—such as DNA analysis,  serology, toxicology, and chemical analysis—are built on solid bases of  theory and research, many of the pattern-based disciplines—such as  microscopic hair comparison, latent fingerprints, and tool marks—were  developed based on observation, experience, and reasoning. As such,  the scientific rigor, accuracy, and reliability of certain methods and the  specificity with which examiners may report opinions and conclusions in  laboratory reports and court testimony vary substantially by forensic  discipline and method.
The forensic science community has initiated a number of efforts to  strengthen the forensic sciences. For example, in February 2013, DOJ, in  partnership with the Department of Commerce’s National Institute of  Standards and Technology (NIST), established the National Commission  on Forensic Science to enhance the practice and improve the reliability of  forensic science. The Commission included federal, state and local  forensic science service providers; research scientists and academics;  law enforcement officials; prosecutors, defense attorneys, and judges;  and other stakeholders from across the country. Further, in February  2014, NIST, with support from DOJ, established the Organization of  Scientific Area Committees (OSAC) for Forensic Science with the stated  goal of creating an infrastructure that produces consensus-based  documentary standards and guidelines for forensic science. OSAC is  designed to provide leadership in the following areas: (1) facilitating the  development and promulgation of consensus-based documentary  standards and guidelines for forensic science, (2) promoting standards  and guidelines that are fit-for-purpose and based on sound scientific  principles, (3) promoting the use of OSAC standards and guidelines by  accreditation and certification bodies, and (4) establishing and  maintaining working relationships with similar organizations. The  organization is a collaborative body of more than 500 forensic science  practitioners and other experts who represent local, state, and federal  agencies; academia; and industry.
section_title: FBI Has a Framework In Place To Help Ensure Quality In Its Forensic Examinations of Chemical and Trace Evidence
paragraphs: The FBI Laboratory has established a quality assurance framework  through its policies and procedures to help ensure quality in its forensic  examinations of chemical and trace evidence. According to the FBI, the  policies and procedures that establish the quality assurance framework  are designed to adhere to ISO/IEC and ASCLD/LAB standards.  ASCLD/LAB accreditation results and our review of the framework found  it to meet international and accreditation standards.
Ensuring that forensic examiners produce scientifically sound, valid, and  reliable examination results is a stated goal of the FBI Laboratory quality  system. The FBI Laboratory’s quality assurance framework consists of  several components: policies and procedures, quality assurance  mechanisms, corrective actions to address nonconformities, and training  requirements designed to ensure quality in its forensic examinations. The  quality assurance mechanisms are programs designed to ensure that the  policies and procedures are followed correctly in the Laboratory. The  corrective actions are designed to address nonconformities and correct  issues within the Laboratory. Based on the results of corrective actions,  the FBI Laboratory updates policies and procedures as needed. Overall,  the framework is also reliant on a structure of training, continuing  education, and proficiency testing to ensure forensic examiners and  technicians are qualified to conduct casework. Figure 1 depicts the  continuous nature of the FBI Laboratory’s quality assurance framework.
The two main documents establishing the policies and procedures under  the FBI Laboratory’s quality assurance framework are the Laboratory  Quality Assurance Manual and the Laboratory Operations Manual. The  Quality Assurance Manual contains the policies, practices, and  procedures that all units are to follow to ensure technical competence and  valid forensic examination. The Laboratory Operations Manual is a  collection of Laboratory-wide quality assurance practices that outline how  to implement portions of the Quality Assurance Manual, as well as  generalized Laboratory practices such as the security of evidence storage  rooms and the handling of different types of evidence.
The individual caseworking and quality assurance units of the Laboratory  supplement the Laboratory Quality Assurance Manual and Laboratory  Operations Manual with their own unit-specific quality assurance  manuals. The unit-specific quality assurance manuals include standard  operating procedures that apply to a specific unit, but they do not  supersede the Laboratory-wide manuals. The Chemistry Unit and Trace  Evidence Unit quality assurance manuals provide each unit’s mission  statement, scope of work, and administrative structure, as well as unit- specific guidance on evidence handling, validation of methods, training,  and proficiency testing, among other areas.
In addition to unit-specific quality assurance manuals, every caseworking  and quality assurance unit in the Laboratory has a collection of standard  operating procedures for each forensic method practiced within the unit.  For example, there are standard operating procedures that describe how  to analyze opioids in hair and how to conduct forensic hair examinations.  The standard operating procedures also cover other aspects of daily work  such as report writing procedures and FBI approved standards for  scientific testimony and report language, specific to the disciplines  practiced by the unit.
Every FBI Laboratory unit also has a collection of unit-specific training  manuals and equipment manuals. The training manuals provide the basic  training procedures for the forensic disciplines, methods, and pieces of  equipment used in each unit. The equipment manuals detail the use,  care, and maintenance of every piece of equipment in the Laboratory.  Furthermore, all Laboratory quality assurance documents undergo  document control procedures to ensure that they are adequate, approved  for use, and that only the current versions are in use. Figure 2 shows  the relationship of the FBI Laboratory quality assurance policies and  procedures.
section_title: FBI Laboratory Generally Ensures Chemistry and Trace Evidence Units Adhere to Quality Standards, but Could Review More Testimonies
paragraphs: We found that the FBI Laboratory generally ensures the Chemistry and  Trace Evidence Units adhere to a variety of quality standards, including  conducting audits, implementing corrective actions, ensuring staff have  appropriate training, and reviewing laboratory reports. However, the  Laboratory’s testimony monitoring program is limited by difficulties in  acquiring testimony transcripts.
subsections: section_title: Internal Audits of the Chemistry and Trace Evidence Units Complied with FBI Policies and International Standards; External Audits of These Units Confirmed Compliance with Accreditation Requirements
paragraphs: The FBI Laboratory has mechanisms that are intended to ensure  adherence to its quality assurance framework. These mechanisms  include internal audits, external audits, internally reported  nonconformities, and management reviews and a quality assurance  working group.
subsections: section_title: Internal Audits
paragraphs: The FBI Laboratory’s Forensic Analysis Support Unit manages the  internal audit program for the entire Laboratory. The internal audit  program consists of at least eight annual audits for each caseworking unit  in the Laboratory. Currently, the FBI Laboratory requires that every unit  complete audits of the following eight areas: case file, evidence security  and seal, proficiency tests, training records and continuing education,  instrument calibration and maintenance, purchasing services and  supplies, document control, and court testimony monitoring. The  Laboratory may then conduct additional audits as it sees fit. To conduct  internal audits, auditors are to review records, interview personnel, and  observe operations and facilities. The auditor is to complete a checklist of  requirements that are to be met, and, once the audit is complete, the  auditors are to produce an audit report that lists all audit outcomes and  any corrective actions. When the unit has resolved all corrective actions  from an audit, the audit is closed.
Based on our review of internal audit reports from 2008 to 2015 we found  that all of the internal audits of the Chemistry and Trace Evidence Units  functioned in accordance with FBI policies and procedures, and  international standards. The program identified nonconformities and  followed the required corrective action process to correct the  nonconformities, and no trends were identified to signal possible  problems in a specific discipline or procedure. We noted two instances of  issues in record keeping. In one instance, we found that a notation in an  internal audit checklist was not noted in the audit report. The problem  pertained to equipment calibration documentation and did not have any  negative impact on the Laboratory’s performance. In the other instance,  the Laboratory was unable to provide us a 2013 audit report upon  request. To provide the audit report, the Laboratory had to reissue the  report with signatures of the current supervisors. We determined that  neither of these record keeping issues was detrimental to the audit  program or any of the casework conducted by the units.
section_title: External Audits
paragraphs: The FBI Laboratory contracts with accreditation bodies to conduct  external audits of the FBI Laboratory at least once every 2 to 5 years.  These audits range from the accreditation of the Laboratory’s quality  assurance process and overall management systems to an assessment  of how Laboratory staff carry out specific methods and follow standard  operating procedures.
As noted above, ASCLD/LAB is the main accreditation organization that  the FBI Laboratory utilizes. In addition to the full on-site assessment of  the Laboratory every 5 years, ASCLD/LAB also conducts abbreviated  reviews of specific Laboratory processes and records annually. According  to an ASCLD/LAB representative, ASCLD/LAB provides an audit team  made up of a lead assessor, employed by ASCLD/LAB, and subject  matter experts from other accredited laboratories for on-site  assessments. The subject matter experts represent every forensic  discipline in which the Laboratory seeks accreditation. The subject matter  experts participate on a voluntary basis and receive training from  ASCLD/LAB. ASCLD/LAB first accredited the FBI Laboratory in 1998,  and then later accredited it to the international program (meeting the  ISO/IEC 17025:2005 standard) in 2008. ASCLD/LAB reassessed the FBI  Laboratory in 2013 to maintain the accreditation. ASCLD/LAB also  conducted surveillance visits of the FBI Laboratory in 2009, 2010, 2011,  and 2015; and off-site document reviews in 2012 and 2014. On the basis  of our review of ASCLD/LAB documents from 2008 through 2015, the FBI  Laboratory generally met accreditation requirements in each instance.  The ASCLD/LAB representative stated that the FBI Laboratory will  undergo its next full assessment for accreditation in April 2018.
In addition, the American Board of Forensic Toxicology conducts on-site  audits of the toxicology subunit, within the FBI Laboratory Chemistry Unit,  every 2 years and off-site reviews in the intervening years. The audits  examine the toxicology subunit based in part on the Society of Forensic  Toxicology/American Academy of Forensic Sciences standards that go  beyond ASCLD/LAB standards, and according to an FBI official, are more  specific to the subunit’s work. The American Board of Forensic  Toxicology has accredited the toxicology subunit of the FBI Laboratory  since 2007, and the subunit has met accreditation requirements in each  instance.
section_title: Internally Reported Nonconformities
paragraphs: The FBI Laboratory has a process for staff to internally report any issues  that may appear in Laboratory work or the day-to-day administration of  the Laboratory. The process allows any employee to report  nonconformities that occurred during Laboratory work, improvements to  processes that they identify, or mistakes they may have made. According  to FBI Laboratory officials, the internal process allows employees the  ability to correct and improve Laboratory processes before they are  identified through the internal or external audits. Once the issues are  reported to management through the internal reporting process, they are  assessed through the Laboratory’s corrective action process, which is  described in more detail below.
section_title: Management Reviews and Quality Assurance Working Group
paragraphs: The FBI Laboratory uses management reviews and a quality assurance  working group as additional quality assurance mechanisms. The  management review is a synthesis by the FBI Laboratory’s Forensic  Analysis Support Unit of all quality assurance actions taken by the  Laboratory over the previous year. It identifies the Laboratory’s  adherence to international and ASCLD/LAB standards, outcomes of any  internal and external audits, the number and nature of external complaints  about the Laboratory, recommendations for improvements, and the  overall workload of the Laboratory. The review results in an annual report  for Laboratory management and ASCLD/LAB. According to the FBI  Laboratory, the review keeps upper level management engaged in the  detailed aspects of the Laboratory and provides management with insight  into trends within the Laboratory’s workload.
The quality assurance working group is a group of representatives from  every caseworking unit within the Laboratory who meet monthly to  discuss ongoing quality assurance issues within the Laboratory.  According to the FBI Laboratory, the group examines improvements that  can be made to the quality assurance framework and examines case  studies of useful quality assurance practices from around the Laboratory.  FBI Laboratory officials told us the monthly meetings typically involve  ensuring consistency of language across all quality assurance documents  and a presentation from one of the unit representatives on handling a  recent nonconformity. Officials maintain agendas and meeting minutes for  future review by management, quality assurance representatives, or  accrediting bodies.
section_title: FBI Has Fully Addressed All But One Corrective Action for the Chemistry and Trace Evidence Units from 2008 to 2015
paragraphs: The FBI uses a corrective action process to address nonconformities that  have been identified through internal and external audits and through  internally reported nonconformities. Regardless of how the nonconformity  is identified, the Laboratory uses the same corrective action process.  According to FBI Laboratory officials, the quality assurance manager and  unit management evaluate the reported nonconformity to determine how  to address it. The Laboratory, following ISO/IEC standards, uses different  types of corrective actions based on the nonconformity’s possible impact  on Laboratory casework, as shown in table 1.
According to FBI Laboratory policy, level 1 corrective actions deal with  situations or conditions that directly affect the quality of Laboratory work.  For example, a level 1 corrective action in the Trace Evidence Unit may  be used to address the possible contamination of evidence. Possible  contamination happens when evidence from two different sources is  processed in the same room on the same day.
According to FBI policies, the first step of addressing a nonconformity is  to notify the appropriate technical manager and quality manager, if  necessary, of the nonconformity and document the issue in the relevant  database or forms. If the nonconformity requires a level 1 or 2 corrective  action, the nonconformity is to undergo a root cause analysis to  determine why the nonconformity occurred. Then quality assurance  personnel or the technical manager are to ensure that the appropriate  action steps are taken to correct the nonconformity. Depending on the  nature of the nonconformity, the action steps may include the following:  notifying the contributor of the evidence of the corrective action, reviewing  and correcting any previous findings, issuing amended reports,  reassigning casework duties, remedial training, revising standard  operating procedures, adopting additional quality control measures, or  indefinitely removing an employee from casework. Once the action steps  are completed, the appropriate personnel must verify the effectiveness of  the corrective action and close the corrective action, as appropriate.
We examined all 57 corrective actions produced by the Laboratory for the  Chemistry and Trace Evidence Units from calendar years 2008 to 2015  as a result of external audits, internal audits, and internally reported  nonconformities. Our analysis of the corrective actions issued by the  Laboratory during this period found that all nonconformities except one  have been addressed, reviewed, and verified in accordance with the FBI  Laboratory’s corrective actions process. In the one case, the FBI  Laboratory issued a level 1 corrective action in September 2015 resulting  from its ongoing review of cases involving microscopic hair comparison  examinations conducted by the FBI Laboratory from the mid-1970s  through 1999. In December 2015 and January 2016, the Laboratory  convened a root cause panel to begin gathering evidence and mapping  possible causes that contributed to examiners making erroneous  statements in laboratory reports and testimonies. In February 2017, the  FBI posted a request for proposal for a third-party contractor to conduct a  full root cause analysis. The FBI Laboratory in its correspondence with  ASCLD/LAB noted that until the root cause analysis is completed, the  corrective action will remain open.
The internal audits, external audits, and internally reported  nonconformities all produced corrective actions. The number and types of  corrective actions varied based on the originating mechanism. For  example, the internally reported nonconformities resulted in 7 level 1  corrective actions and 14 level 2 corrective actions for these two units.  The internal audits produced 1 level 1 corrective action and 25 level 2  corrective actions. In a separate case, demonstrating the effectiveness of  the corrective action process, the Laboratory issued a level 2 corrective  action concerning the handling of chemical materials, based on an  internal audit finding in 2011. The Laboratory then reviewed the  effectiveness of the corrective action in 2012, through the normal quality  assurance mechanisms, and discovered that the original issue persisted.  The Laboratory issued another corrective action through its internally  reported process to supersede the previous action and corrected the  issue.
Overall, the FBI Laboratory’s corrective action process serves to identify  areas for improvement and results in changes in operating procedures  intended to improve the FBI Laboratory’s quality assurance framework.
section_title: Chemistry and Trace Evidence Unit Staff Met Annual Proficiency Testing Requirements and Their Training Files Were Generally Complete and Met FBI and Accreditation Requirements
paragraphs: The FBI Laboratory has developed a training approach to help ensure all  caseworking employees are trained and qualified to conduct forensic  casework. According to ASCLD/LAB, the training program’s requirements  are consistent with international and accreditation standards for personnel  conducting casework in forensic laboratories.
Forensic examiners and technicians are the two primary types of  employees who conduct casework within the FBI Laboratory. In  accordance with the FBI Laboratory’s Quality Assurance Manual, all FBI  examiners and technicians are to complete an initial training program in  which they must satisfactorily demonstrate competency in the relevant  discipline(s) or categor(y/ies) of testing they will practice prior to assuming  independent casework responsibilities. To help ensure competency  prior to assuming casework responsibilities, the Laboratory’s training  approach employs unit-specific training. For example, the Chemistry and  Trace Evidence Units have each developed unit-specific training  programs and manuals to train staff on their specific disciplines and  positions. While training and testing requirements are specific to the  discipline(s) and role the employee is to perform, every employee must  pass competency tests and oral board examinations. In addition, forensic  examiners, who provide testimony in court proceedings, must also  complete moot court and admissibility hearing exercises, such as a  Daubert hearing. Successful completion of all training requirements is to be documented in a “qualification and authorization” memorandum  prepared by the unit chief or designee. Table 2 provides additional  information on the initial training and qualification requirements for  examiners and technicians in the Chemistry and Trace Evidence Units.
After completing the initial training program, caseworking employees must  complete continued education and proficiency test requirements each  year. For example, Chemistry and Trace Evidence Unit employees must  complete a minimum of 15 hours of continued education related to job  requirements or job performance each year. Chemistry Unit policy states  that the continued education should focus on maintaining technical skills  and expertise. In addition, each examiner and technician must complete  an annual proficiency test in each category of testing that he or she  performs. According to accreditation and FBI requirements, the  Laboratory must use proficiency tests developed by an external test  provider where available. On the basis of ASCLD/LAB assessments, all  FBI Laboratory staff conducting casework or providing testimony in  disciplines in which the Chemistry and Trace Evidence Units are  accredited had successfully met annual proficiency testing requirements  in their respective discipline(s) from 2011 to 2015 as required by  accreditation standards.
Additionally, we observed that for the forensic anthropology discipline,  wherein ASCLD/LAB does not provide accreditation, the two Trace  Evidence Unit examiners who conducted casework in forensic  anthropology completed annual proficiency testing in forensic  anthropology from 2011 to 2015. Furthermore, the FBI Laboratory  requires that all staff conducting casework in forensic anthropology obtain  certification by the American Board of Forensic Anthropology. According  to the Board, these two examiners were board-certified in 2012 and 2013,  respectively, and have maintained certification.
The combination of training, continued education, and annual proficiency  testing are intended to help provide reasonable assurance that the FBI  Laboratory’s caseworking staff are appropriately trained and qualified. We  reviewed the initial training records for the 47 employees who conducted  casework in the Chemistry and Trace Evidence Units or provided court  testimony related to chemical or trace evidence from fiscal year 2011 to  July 2016 and found that the records were generally complete and met  FBI requirements.
section_title: FBI Laboratory Has Processes to Review Laboratory Reports and Testimony, but Could Review More Examiner Testimonies
paragraphs: The FBI Laboratory implemented a monitoring program to help ensure  that the results of its forensic examinations and any related examiner  testimony are presented consistently with what is known about each  forensic method. However, the testimony monitoring program is limited by  difficulties in acquiring testimony transcripts.
FBI Approved Standards for Scientific  Testimony and Report Language  These standards document examples of the  statements the FBI Laboratory has  determined to be scientifically-supported and  approved for its examiners to use when  reporting examination conclusions and expert  opinion in laboratory reports and court  testimony for certain methods. practices, and unit procedures. Our review of the 166 laboratory reports  produced by the Chemistry and Trace Evidence Units and used as the  basis of testimonies from fiscal years 2011 through 2015 found that all  had been technically and administratively reviewed in accordance with  FBI requirements.
For example, according to the standards, FBI  fiber examiners may state or imply that a  textile fiber is natural or manufactured (man- made) and its type (e.g., cotton, wool,  polyester). However, the examiner may not  state or imply that a fiber came from a  particular source to the exclusion of all others.  The examiner also may not state or imply a  statistical weight or probability to a conclusion  or provide a likelihood that the questioned  fiber originated from a particular source. The  FBI Laboratory began developing and  implementing these standards in 2014.
The court testimony monitoring program, in contrast, includes both  internal and external reviews. Specifically, the court testimony monitoring  program’s two key monitoring and oversight components include the  following: (1) obtaining and reviewing transcripts of examiner testimonies  to determine their compliance with FBI standards, referred to as internal  evaluations; and (2) obtaining feedback on the quality of examiners’  testimonies directly from court officials, referred to as external  evaluations. The FBI’s internal evaluation required that the reviewer— an appropriate manager and, when necessary, with assistance from a  subject matter expert in the same area as the individual testifying— assess the examiner’s testimony against seven technical criteria. These  include whether the examiner (1) testified accurately; (2) testified within  the scope of his or her expertise; and (3) declined to answer questions  beyond his or her expertise or beyond scientific limitations for the  discipline, among other things. External evaluations, on the other hand,  rely on the feedback of court officials who are not scientific or technical  experts and who are asked to evaluate factors such as an examiner’s  appearance, demeanor, promptness, ability to maintain composure as an  expert witness, and ability to communicate results. See appendix I for  further details about the FBI Laboratory’s practices for internal and  external evaluation of testimony.
From fiscal years 2011 through 2015, a total of 22 forensic examiners in  the FBI Laboratory’s Chemistry and Trace Evidence Units produced a  total of 164 testimonies. However, the FBI Laboratory did not acquire  transcripts and conduct internal evaluations for nearly half of these  testimonies (78 of 164). As such, the 78 testimonies were not reviewed  by a subject matter expert or against technical discipline-specific criteria,  as provided for by internal evaluations.
The FBI obtained transcripts for 86 of the 164 testimonies and conducted  an internal evaluation of the 86 transcripts. Additionally, of the 164  testimonies, the FBI obtained an external evaluation of the examiner’s  performance for 99 testimonies. All of the internal evaluations of  examiner testimonies that we reviewed received satisfactory  assessments from an FBI manager. Also, in conformance with  accreditation and FBI requirements and on the basis of our review of  ASCLD/LAB reports, the FBI Laboratory met the accreditation  requirement for testimony monitoring during this period.
While the FBI Laboratory’s testimony monitoring program meets current  accreditation requirements, it could be improved by enhancing its  testimony acquisition approach to allow it to conduct additional internal  evaluations. Subjecting testimonies to an internal evaluation provides the  strongest basis for understanding the extent to which examiners are  adhering to FBI technical and reporting standards while also ensuring that  examiner testimony is factually accurate. Furthermore, in March 2017, the  merged accrediting body issued updated testimony monitoring  requirements requiring that examiner testimony in each discipline be  reviewed by an individual that has been competency tested in the task(s)  that the review is encompassing. To meet the revised requirement, the  FBI Laboratory will need to conduct an internal evaluation of examiner  testimony. In other words, external evaluations by court officials, alone,  will no longer satisfy the requirement for accreditation.
In explaining why the 78 testimony transcripts were not acquired, FBI  Laboratory officials told us that because they could not compel courts to  provide them with responses or transcripts, they have no way of knowing  exactly why courts or court reporters did not provide specific transcripts.  Anecdotally, they provided us with the following challenges and limitations  they often encounter in their efforts to obtain transcripts:
There is no clear point of contact, or there is lack of responsiveness  from points of contact, such as prosecutors or court reporters;
There is a lack of uniform computer and business systems across the  separate federal, state and local courts in which they testify;
Some courts store their transcripts manually and lack electronic  databases, making it difficult to locate and acquire transcripts;  In some cases, transcripts may not be accessible because they have  been sealed or may not have been generated.
While FBI cannot compel courts to provide transcripts in all instances,  there are opportunities for the FBI to improve its ability to obtain more  transcripts. To better understand the factors cited by the FBI Laboratory,  we sought to obtain the remaining 78 transcripts directly from the courts.  During this effort, we obtained 36 of the 78 transcripts (46 percent) that  the FBI Laboratory did not obtain. Of the 36, we obtained:
16 transcripts by searching the Public Access to Court Electronic  Records (PACER), an electronic public access service that allows  users to obtain case information from federal appellate, district, and  bankruptcy courts. This did not require personal contact with any court  officials involved with the case;
9 transcripts by directly contacting and coordinating with the court or
11 transcripts through third-party transcription services or by directly  visiting the court.
We were unable to obtain the remaining 42 of 78 transcripts for various  reasons. In several cases, the information provided by the FBI was  incomplete or additional information would be needed to locate the case  and associated transcript. However, while attempting to obtain the  outstanding transcripts, we also confirmed some of the reasons the FBI  identified for not acquiring transcripts. For example, after identifying the  court reporter, sometimes the reporter did not respond to our request(s)  or provide us with the documentation necessary to request the transcript.  Further, some testimonies could not be released because the cases were  sealed. Lastly, some testimonies were not transcribed at the time of our  review.
To develop better controls in its testimony monitoring program activities,  in May 2013, the FBI Laboratory implemented a Testimony Tracker  System (Tracker) to capture and monitor information on internal and  external evaluation of examiner testimony and to help ensure that  transcripts are requested, obtained, and reviewed. As of May 2013, FBI  policy requires that appropriate data regarding the monitoring and  evaluation of examiner testimonies—including when a testimony  occurred, whether a transcript was requested, whether follow-up  occurred, and when a transcript was received and reviewed, among other  things—be recorded in the Tracker system. In addition, in December  2016, the FBI updated its policy to require that a unit chief or designee  check the Testimony Tracker System monthly to ensure testifying  individuals are entering required information. However, we found that  the Tracker is not currently set up to routinely capture certain additional  information that could help examiners conduct more effective follow up.  For example, the Tracker does not capture the specific reason that a  transcript has not been acquired. If a court reporter indicates to an  examiner that a transcript will not be available for 1 year, being able to  note the reason and the availability date or timeframe in the Tracker  would better enable the examiner to follow up at the specified time. Or, if  a specific transcript will not be available until after the case is completed,  configuring the Tracker or revising the testimony monitoring policy to  require examiners to record this information would better enable them to  follow up on the transcript at a future date. Other information that could be helpful in obtaining a transcript includes the court’s jurisdiction, address,  and a point of contact for the transcript. FBI officials confirmed that the  Tracker is not currently configured to routinely record this type of  information but did acknowledge that some information would be useful. A  senior FBI Laboratory official also stated that recording more information,  such as point of contact information, could help the FBI obtain more  transcripts.
Obtaining transcripts and subjecting them to an internal evaluation is a  key control in FBI’s efforts to ensure the quality of examiner testimonies.  To facilitate this, each time an examiner testifies, FBI policy requires the  examiner to request a transcript of the testimony from the respective court  official so that it may be reviewed by an appropriate FBI manager or other  subject matter expert. Further, Standards for Internal Control in the  Federal Government calls for, among other things, controls to be  designed to assure that ongoing monitoring occurs. The standards also  state that relevant, reliable, and timely information is needed throughout  an agency to control operations, achieve its objectives, and carry out its  oversight responsibilities. Consistent with internal control standards, the  FBI Laboratory could better ensure that it obtains more transcripts by  routinely capturing and using additional information and data that are  critical to their acquisition, such as the reason a transcript is unavailable,  when it is expected to be available, the court jurisdiction, and a point of  contact for the transcript. Increasing the number of transcripts acquired  could help the FBI Laboratory expand its monitoring of examiners’  testimonies, which helps ensure that they are accurate, supported by the  underlying analyses, and within the scientific limits of the given forensic  discipline, as defined by FBI and accreditation standards.
section_title: Conclusions
paragraphs: The FBI Laboratory’s forensic work is used to support law enforcement  across all levels of government. Forensic evidence can help solve crimes  and influence whether a criminal defendant is acquitted, convicted, or  even charged. Ensuring that forensic examiners produce scientifically  valid and reliable examinations, laboratory reports, and testimonies for  such cases is a stated goal of the FBI Laboratory quality system. The FBI  Laboratory has a quality assurance framework that meets international  and accreditation standards, including those for monitoring examiner  testimonies. For example, the FBI Laboratory has established standards  for scientific testimony and report language, employs training to help  ensure its examiners adhere to the standards, and uses laboratory report  and testimony review processes to help ensure examiners adhere to the  standards. We found that the FBI Laboratory generally ensures that the  Chemistry and Trace Evidence Units adhere to a variety of quality  standards, including conducting audits, implementing corrective actions,  ensuring staff have appropriate training, and reviewing laboratory reports.
The Laboratory’s internal evaluation of examiner testimony provides the  strongest basis for understanding the extent to which examiners are  adhering to FBI technical and reporting standards while also ensuring that  examiner testimony is factually accurate. We found, however, that the FBI  Laboratory could obtain more transcripts by enhancing the current  procedure for acquiring testimony transcripts. Specifically, the Laboratory  could ensure that it captures and uses information that facilitates the  retrieval of a transcript, such as the reason the transcript may not be  available and its court jurisdiction. Routinely capturing additional  information could help the FBI Laboratory successfully obtain more  transcripts to review, which can help better ensure that its examiners are  providing testimony in court that is accurate, supported by the underlying  analyses, and within the scientific limits of the given forensic discipline, as  defined by FBI and accreditation standards.
section_title: Recommendation for Executive Action
paragraphs: To better ensure that the FBI Laboratory obtains additional transcripts,  the FBI Director should require that the FBI Laboratory’s procedure for  tracking and obtaining transcripts routinely captures and uses additional  information and data critical to transcript acquisition, such as the reason a  transcript is unavailable, when it is expected to be available, the court  jurisdiction, and a point of contact for the transcript.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Departments of Justice and  Commerce. In response, the FBI provided written comments, which are  reproduced in full in appendix II. FBI concurred with the recommendation,  and described actions planned to address it. Specifically, FBI concurred  with GAO that the use of additional information and data is helpful in  obtaining testimony transcripts. As such, the FBI Laboratory plans to  modify its Testimony Tracker System accordingly to add additional  information and available data for transcript acquisitions. The Department  of Commerce did not provide written comments on our draft report. The  FBI and the Department of Commerce also provided technical comments,  which we incorporated as appropriate.
We are sending copies of this report to the Attorney General and the  Secretary of Commerce and appropriate congressional committees, and  other interested parties. In addition, the report is available at no charge on  the GAO website at http://www.gao.gov.
If you or your staff have any questions, please contact Diana Maurer at  (202) 512-8777 or maurerd@gao.gov. Contact points for our Offices of  Congressional Relations Public Affairs may be found on the last page of  this report. GAO staff that made significant contributions to this report are  listed in appendix III.
section_title: Appendix I: FBI Laboratory Practices for Internal and External Evaluations of Court Testimony
paragraphs: Listed below are the key steps involved in each of the internal and  external evaluations of examiner testimonies delivered during court  proceedings.
subsections: section_title: Internal Evaluations
paragraphs: 1.  Each time the examiner testifies he or she is responsible for  requesting a transcript of the testimony from the court reporter(s),  prosecutor, or other appropriate individual so it may be reviewed by  an appropriate FBI manager or other subject matter expert.  2.  If a copy of the transcript is not received after the initial request, the  examiner is required to follow-up at least twice within 6 months of his  or her testimony.  3.  If a transcript is received, within 45 days of receipt, an appropriate FBI  manager is required to review it and assess whether the examiner’s  statements comply with the seven criteria in the Internal Evaluation of  Testimony (7-256).  4.  If the manager is not a subject matter expert, then a subject matter  expert is to assist the manager in the evaluation.  5.  After the review is complete, the manager and subject matter expert, if  appropriate, who completed the Internal Evaluation of Testimony (7- 256) is/are to review the completed evaluation with the examiner  within 45 days of receiving the transcript, and all parties are to  document this review in the form.
section_title: External Evaluation
paragraphs: 1.  After each testimony, examiners are required to provide court officials,  such as a prosecutor or defense attorney, with an External Evaluation  of Testimony (7-257) to provide feedback on the examiner’s  testimony.  2.  The manager and subject matter expert, if appropriate, who received  the External Evaluation of Testimony (7-257) is/are to review the  completed evaluation with the examiner within 45 days of receipt, and  all parties are to document this review in the form.
section_title: Appendix II: Comments from the Federal Bureau of Investigation
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Joseph P. Cruz (Assistant  Director), Hazel Bailey, Amy Bowser, Jennifer Bryant, Clifton Douglas,  Michele Fejfar, April Gamble, Susan Hsu, Kyle Maksuta, Claire Peachey,  Lerone Reid, Sushil Sharma, Jack Sheehan, and Helina Wong made key  contributions to this report.