FBI Laboratory: Chemistry and Trace Evidence Units Generally Adhere to Quality Standards, but Could Review More Examiner Testimonies
Released: 2017-06-28T00:00:00

Letter

Background

FBI Laboratory

The FBI Laboratory, located in Quantico, Virginia, provides a full range of forensic services including forensic examinations, technical support, and expert witness testimony. As of fiscal year 2015, the FBI Laboratory had an operational budget of approximately $102.4 million and a staffing budget of approximately $97.9 million. The FBI Laboratory employed 651 full-time staff at the end of fiscal year 2015 across five sections: Biometrics Analysis, Forensic Response, Forensic Science Support, Scientific Analysis, and the Terrorist Explosive Device Analytical Center.

The units discussed in this report, the Chemistry and Trace Evidence Units, are both located within the Scientific Analysis Section.

The Chemistry Unit conducts scientific analyses in four disciplines: general chemistry, toxicology, paints and polymers, and metallurgy. The Chemistry Unit had an operational budget of approximately $807,500 and a personnel budget of approximately $3.2 million in fiscal year 2015, and employed 15 caseworking staff. The Trace Evidence Unit conducts scientific analyses in three disciplines: hair and fiber, geology and mineralogy, and forensic anthropology. The Trace Evidence Unit had an operational budget of approximately $122,700 and personnel budget of approximately $2.6 million in fiscal year 2015, and employed 16 caseworking staff. The combined operational budgets of the Chemistry and Trace Evidence Units accounted for about 20 percent of the Scientific Analysis Section’s fiscal year 2015 operational budget and 1 percent of the FBI Laboratory’s overall operational budget.

In fiscal year 2015, the Chemistry Unit received 324 requests for examinations, and the Trace Evidence Unit received 637 requests for examinations. From fiscal years 2011 to 2015, the total number of requests for examinations submitted to the Chemistry and Trace Evidence Units declined from 1,159 requests to 961 requests.

International Standards and Accreditation

A testing and calibration laboratory may opt to follow international standards to ensure that it is technically competent, has a management structure, and is able to generate technically valid results. The International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) develop and publish standards for various industries worldwide, including laboratories. Laboratories can use the ISO/IEC 17025:2005 General Requirements for Competence of Testing and Calibration Laboratories standard to govern operations.

Accreditation bodies can use the ISO/IEC 17025:2005 standard as the basis to accredit participating laboratories. The ISO/IEC 17025:2005 standard specifies the general requirements for laboratories to competently carry out tests, including sampling. The standard specifies, among other things, that a laboratory shall have a quality manual and a quality policy statement, define the roles and responsibilities of management, and have policies and procedures that institute many parts of a quality assurance framework. For example, the standard requires laboratories to have policies and procedures to handle nonconformities in their work, including a corrective action process; conduct periodic internal audits and management reviews; and ensure that staff performing testing are qualified on the basis of appropriate education, training, experience, and/or demonstrated skills.

For the period of our review’s focus from 2008 to 2015, the American Society of Crime Laboratory Directors/Laboratory Accreditation Board (ASCLD/LAB) accredited the FBI Laboratory to demonstrate that it met the ISO/IEC 17025:2005 requirements, as well as additional ASCLD/LAB requirements specific to forensic science laboratories, or the ASCLD/LAB-International Supplemental Requirements for the Accreditation of Forensic Science Testing Laboratories. ASCLD/LAB accredits forensic science laboratories in specific disciplines.

ASCLD/LAB accredits laboratories on a 5-year cycle. The accreditation process begins with ASCLD/LAB ensuring that the policies and procedures of a laboratory meet the ISO/IEC and ASCLD/LAB standards. The laboratory undergoes a full accreditation assessment consisting of a document review of policies and procedures, meetings with the laboratory director and other appointed individuals, an on-site assessment, and corrective actions to address identified nonconformities. ASCLD/LAB then provides an assessment report and certifies that the laboratory is accredited, if applicable. Following the full on-site assessment, the laboratory must provide a performance declaration to ASCLD/LAB periodically, participate in and abide by the rules of the ASCLD/LAB proficiency testing program, disclose all significant events and nonconformities, and undergo scheduled surveillance visits. ASCLD/LAB determines the frequency of performance declaration submissions based on the performance of the laboratory during the previous accreditation cycle. One scenario is the submission of a performance declaration only when a laboratory does not undergo an on- site visit.

Legal Standards for Assessing the Reliability of Forensic Evidence in Court

Federal jurisdictions and states use different standards for determining the reliability of expert testimony in court. Under the Federal Rules of Evidence, Rule 702, an expert witness is considered qualified to testify if, among other things, the testimony is the product of reliable principles and methods. The 1993 Supreme Court case, Daubert v. Merrell Dow Pharmaceuticals, Inc. (509 U.S. 579), significantly changed the assessment of reliability of scientific evidence for federal trial courts, making trial judges responsible for acting as gatekeepers to exclude unreliable scientific expert testimony. The Daubert case listed factors for judges to use in assessing the reliability of scientific expert testimony, including (1) whether the expert’s technique or theory can be or has been tested, (2) whether the technique or theory has been subjected to peer review, (3) the known or potential rate of error of the technique or theory when applied, (4) the existence and maintenance of standards and controls, and (5) whether the technique or theory has been generally accepted in the relevant scientific community. The Daubert factors are not meant to be exclusive and other courts have found additional factors relevant in determining the reliability of expert testimony. Most states generally follow the Daubert standard, while some use the Frye test, developed by the Court of Appeals of the District of Colombia in 1923 (Frye v. U.S., 293 F. 1013). The Frye test requires that the scientific principle be generally accepted in the relevant scientific community. A minority of states have not adopted either the Daubert factors or Frye test and have their own rules for admitting scientific evidence into court.

Challenges With Certain Forensic Standards, Methods, and Practices

A method is the course of action or technique followed in conducting a specific analysis or comparison leading to an analytical result, such as microscopic hair comparison.

Since the publication of the 2009 National Academy of Sciences (NAS) report titled Strengthening Forensic Science in the United States: A Path Forward, the standards, methods, and practices of the forensic science community have been the subject of considerable discussion. The NAS report highlighted that there is wide variability across forensic science disciplines with regard to techniques, methodologies, reliability, types and numbers of potential errors, research, general acceptability, and published material. Further, while some of the disciplines and methods used by the forensic science community—such as DNA analysis, serology, toxicology, and chemical analysis—are built on solid bases of theory and research, many of the pattern-based disciplines—such as microscopic hair comparison, latent fingerprints, and tool marks—were developed based on observation, experience, and reasoning. As such, the scientific rigor, accuracy, and reliability of certain methods and the specificity with which examiners may report opinions and conclusions in laboratory reports and court testimony vary substantially by forensic discipline and method.

The forensic science community has initiated a number of efforts to strengthen the forensic sciences. For example, in February 2013, DOJ, in partnership with the Department of Commerce’s National Institute of Standards and Technology (NIST), established the National Commission on Forensic Science to enhance the practice and improve the reliability of forensic science. The Commission included federal, state and local forensic science service providers; research scientists and academics; law enforcement officials; prosecutors, defense attorneys, and judges; and other stakeholders from across the country. Further, in February 2014, NIST, with support from DOJ, established the Organization of Scientific Area Committees (OSAC) for Forensic Science with the stated goal of creating an infrastructure that produces consensus-based documentary standards and guidelines for forensic science. OSAC is designed to provide leadership in the following areas: (1) facilitating the development and promulgation of consensus-based documentary standards and guidelines for forensic science, (2) promoting standards and guidelines that are fit-for-purpose and based on sound scientific principles, (3) promoting the use of OSAC standards and guidelines by accreditation and certification bodies, and (4) establishing and maintaining working relationships with similar organizations. The organization is a collaborative body of more than 500 forensic science practitioners and other experts who represent local, state, and federal agencies; academia; and industry.

FBI Has a Framework In Place To Help Ensure Quality In Its Forensic Examinations of Chemical and Trace Evidence

The FBI Laboratory has established a quality assurance framework through its policies and procedures to help ensure quality in its forensic examinations of chemical and trace evidence. According to the FBI, the policies and procedures that establish the quality assurance framework are designed to adhere to ISO/IEC and ASCLD/LAB standards. ASCLD/LAB accreditation results and our review of the framework found it to meet international and accreditation standards.

Ensuring that forensic examiners produce scientifically sound, valid, and reliable examination results is a stated goal of the FBI Laboratory quality system. The FBI Laboratory’s quality assurance framework consists of several components: policies and procedures, quality assurance mechanisms, corrective actions to address nonconformities, and training requirements designed to ensure quality in its forensic examinations. The quality assurance mechanisms are programs designed to ensure that the policies and procedures are followed correctly in the Laboratory. The corrective actions are designed to address nonconformities and correct issues within the Laboratory. Based on the results of corrective actions, the FBI Laboratory updates policies and procedures as needed. Overall, the framework is also reliant on a structure of training, continuing education, and proficiency testing to ensure forensic examiners and technicians are qualified to conduct casework. Figure 1 depicts the continuous nature of the FBI Laboratory’s quality assurance framework.

The two main documents establishing the policies and procedures under the FBI Laboratory’s quality assurance framework are the Laboratory Quality Assurance Manual and the Laboratory Operations Manual. The Quality Assurance Manual contains the policies, practices, and procedures that all units are to follow to ensure technical competence and valid forensic examination. The Laboratory Operations Manual is a collection of Laboratory-wide quality assurance practices that outline how to implement portions of the Quality Assurance Manual, as well as generalized Laboratory practices such as the security of evidence storage rooms and the handling of different types of evidence.

The individual caseworking and quality assurance units of the Laboratory supplement the Laboratory Quality Assurance Manual and Laboratory Operations Manual with their own unit-specific quality assurance manuals. The unit-specific quality assurance manuals include standard operating procedures that apply to a specific unit, but they do not supersede the Laboratory-wide manuals. The Chemistry Unit and Trace Evidence Unit quality assurance manuals provide each unit’s mission statement, scope of work, and administrative structure, as well as unit- specific guidance on evidence handling, validation of methods, training, and proficiency testing, among other areas.

In addition to unit-specific quality assurance manuals, every caseworking and quality assurance unit in the Laboratory has a collection of standard operating procedures for each forensic method practiced within the unit. For example, there are standard operating procedures that describe how to analyze opioids in hair and how to conduct forensic hair examinations. The standard operating procedures also cover other aspects of daily work such as report writing procedures and FBI approved standards for scientific testimony and report language, specific to the disciplines practiced by the unit.

Every FBI Laboratory unit also has a collection of unit-specific training manuals and equipment manuals. The training manuals provide the basic training procedures for the forensic disciplines, methods, and pieces of equipment used in each unit. The equipment manuals detail the use, care, and maintenance of every piece of equipment in the Laboratory. Furthermore, all Laboratory quality assurance documents undergo document control procedures to ensure that they are adequate, approved for use, and that only the current versions are in use. Figure 2 shows the relationship of the FBI Laboratory quality assurance policies and procedures.

FBI Laboratory Generally Ensures Chemistry and Trace Evidence Units Adhere to Quality Standards, but Could Review More Testimonies

We found that the FBI Laboratory generally ensures the Chemistry and Trace Evidence Units adhere to a variety of quality standards, including conducting audits, implementing corrective actions, ensuring staff have appropriate training, and reviewing laboratory reports. However, the Laboratory’s testimony monitoring program is limited by difficulties in acquiring testimony transcripts.

Internal Audits of the Chemistry and Trace Evidence Units Complied with FBI Policies and International Standards; External Audits of These Units Confirmed Compliance with Accreditation Requirements

The FBI Laboratory has mechanisms that are intended to ensure adherence to its quality assurance framework. These mechanisms include internal audits, external audits, internally reported nonconformities, and management reviews and a quality assurance working group.

Internal Audits

The FBI Laboratory’s Forensic Analysis Support Unit manages the internal audit program for the entire Laboratory. The internal audit program consists of at least eight annual audits for each caseworking unit in the Laboratory. Currently, the FBI Laboratory requires that every unit complete audits of the following eight areas: case file, evidence security and seal, proficiency tests, training records and continuing education, instrument calibration and maintenance, purchasing services and supplies, document control, and court testimony monitoring. The Laboratory may then conduct additional audits as it sees fit. To conduct internal audits, auditors are to review records, interview personnel, and observe operations and facilities. The auditor is to complete a checklist of requirements that are to be met, and, once the audit is complete, the auditors are to produce an audit report that lists all audit outcomes and any corrective actions. When the unit has resolved all corrective actions from an audit, the audit is closed.

Based on our review of internal audit reports from 2008 to 2015 we found that all of the internal audits of the Chemistry and Trace Evidence Units functioned in accordance with FBI policies and procedures, and international standards. The program identified nonconformities and followed the required corrective action process to correct the nonconformities, and no trends were identified to signal possible problems in a specific discipline or procedure. We noted two instances of issues in record keeping. In one instance, we found that a notation in an internal audit checklist was not noted in the audit report. The problem pertained to equipment calibration documentation and did not have any negative impact on the Laboratory’s performance. In the other instance, the Laboratory was unable to provide us a 2013 audit report upon request. To provide the audit report, the Laboratory had to reissue the report with signatures of the current supervisors. We determined that neither of these record keeping issues was detrimental to the audit program or any of the casework conducted by the units.

External Audits

The FBI Laboratory contracts with accreditation bodies to conduct external audits of the FBI Laboratory at least once every 2 to 5 years. These audits range from the accreditation of the Laboratory’s quality assurance process and overall management systems to an assessment of how Laboratory staff carry out specific methods and follow standard operating procedures.

As noted above, ASCLD/LAB is the main accreditation organization that the FBI Laboratory utilizes. In addition to the full on-site assessment of the Laboratory every 5 years, ASCLD/LAB also conducts abbreviated reviews of specific Laboratory processes and records annually. According to an ASCLD/LAB representative, ASCLD/LAB provides an audit team made up of a lead assessor, employed by ASCLD/LAB, and subject matter experts from other accredited laboratories for on-site assessments. The subject matter experts represent every forensic discipline in which the Laboratory seeks accreditation. The subject matter experts participate on a voluntary basis and receive training from ASCLD/LAB. ASCLD/LAB first accredited the FBI Laboratory in 1998, and then later accredited it to the international program (meeting the ISO/IEC 17025:2005 standard) in 2008. ASCLD/LAB reassessed the FBI Laboratory in 2013 to maintain the accreditation. ASCLD/LAB also conducted surveillance visits of the FBI Laboratory in 2009, 2010, 2011, and 2015; and off-site document reviews in 2012 and 2014. On the basis of our review of ASCLD/LAB documents from 2008 through 2015, the FBI Laboratory generally met accreditation requirements in each instance. The ASCLD/LAB representative stated that the FBI Laboratory will undergo its next full assessment for accreditation in April 2018.

In addition, the American Board of Forensic Toxicology conducts on-site audits of the toxicology subunit, within the FBI Laboratory Chemistry Unit, every 2 years and off-site reviews in the intervening years. The audits examine the toxicology subunit based in part on the Society of Forensic Toxicology/American Academy of Forensic Sciences standards that go beyond ASCLD/LAB standards, and according to an FBI official, are more specific to the subunit’s work. The American Board of Forensic Toxicology has accredited the toxicology subunit of the FBI Laboratory since 2007, and the subunit has met accreditation requirements in each instance.

Internally Reported Nonconformities

The FBI Laboratory has a process for staff to internally report any issues that may appear in Laboratory work or the day-to-day administration of the Laboratory. The process allows any employee to report nonconformities that occurred during Laboratory work, improvements to processes that they identify, or mistakes they may have made. According to FBI Laboratory officials, the internal process allows employees the ability to correct and improve Laboratory processes before they are identified through the internal or external audits. Once the issues are reported to management through the internal reporting process, they are assessed through the Laboratory’s corrective action process, which is described in more detail below.

Management Reviews and Quality Assurance Working Group

The FBI Laboratory uses management reviews and a quality assurance working group as additional quality assurance mechanisms. The management review is a synthesis by the FBI Laboratory’s Forensic Analysis Support Unit of all quality assurance actions taken by the Laboratory over the previous year. It identifies the Laboratory’s adherence to international and ASCLD/LAB standards, outcomes of any internal and external audits, the number and nature of external complaints about the Laboratory, recommendations for improvements, and the overall workload of the Laboratory. The review results in an annual report for Laboratory management and ASCLD/LAB. According to the FBI Laboratory, the review keeps upper level management engaged in the detailed aspects of the Laboratory and provides management with insight into trends within the Laboratory’s workload.

The quality assurance working group is a group of representatives from every caseworking unit within the Laboratory who meet monthly to discuss ongoing quality assurance issues within the Laboratory. According to the FBI Laboratory, the group examines improvements that can be made to the quality assurance framework and examines case studies of useful quality assurance practices from around the Laboratory. FBI Laboratory officials told us the monthly meetings typically involve ensuring consistency of language across all quality assurance documents and a presentation from one of the unit representatives on handling a recent nonconformity. Officials maintain agendas and meeting minutes for future review by management, quality assurance representatives, or accrediting bodies.

FBI Has Fully Addressed All But One Corrective Action for the Chemistry and Trace Evidence Units from 2008 to 2015

The FBI uses a corrective action process to address nonconformities that have been identified through internal and external audits and through internally reported nonconformities. Regardless of how the nonconformity is identified, the Laboratory uses the same corrective action process. According to FBI Laboratory officials, the quality assurance manager and unit management evaluate the reported nonconformity to determine how to address it. The Laboratory, following ISO/IEC standards, uses different types of corrective actions based on the nonconformity’s possible impact on Laboratory casework, as shown in table 1.

According to FBI Laboratory policy, level 1 corrective actions deal with situations or conditions that directly affect the quality of Laboratory work. For example, a level 1 corrective action in the Trace Evidence Unit may be used to address the possible contamination of evidence. Possible contamination happens when evidence from two different sources is processed in the same room on the same day.

According to FBI policies, the first step of addressing a nonconformity is to notify the appropriate technical manager and quality manager, if necessary, of the nonconformity and document the issue in the relevant database or forms. If the nonconformity requires a level 1 or 2 corrective action, the nonconformity is to undergo a root cause analysis to determine why the nonconformity occurred. Then quality assurance personnel or the technical manager are to ensure that the appropriate action steps are taken to correct the nonconformity. Depending on the nature of the nonconformity, the action steps may include the following: notifying the contributor of the evidence of the corrective action, reviewing and correcting any previous findings, issuing amended reports, reassigning casework duties, remedial training, revising standard operating procedures, adopting additional quality control measures, or indefinitely removing an employee from casework. Once the action steps are completed, the appropriate personnel must verify the effectiveness of the corrective action and close the corrective action, as appropriate.

We examined all 57 corrective actions produced by the Laboratory for the Chemistry and Trace Evidence Units from calendar years 2008 to 2015 as a result of external audits, internal audits, and internally reported nonconformities. Our analysis of the corrective actions issued by the Laboratory during this period found that all nonconformities except one have been addressed, reviewed, and verified in accordance with the FBI Laboratory’s corrective actions process. In the one case, the FBI Laboratory issued a level 1 corrective action in September 2015 resulting from its ongoing review of cases involving microscopic hair comparison examinations conducted by the FBI Laboratory from the mid-1970s through 1999. In December 2015 and January 2016, the Laboratory convened a root cause panel to begin gathering evidence and mapping possible causes that contributed to examiners making erroneous statements in laboratory reports and testimonies. In February 2017, the FBI posted a request for proposal for a third-party contractor to conduct a full root cause analysis. The FBI Laboratory in its correspondence with ASCLD/LAB noted that until the root cause analysis is completed, the corrective action will remain open.

The internal audits, external audits, and internally reported nonconformities all produced corrective actions. The number and types of corrective actions varied based on the originating mechanism. For example, the internally reported nonconformities resulted in 7 level 1 corrective actions and 14 level 2 corrective actions for these two units. The internal audits produced 1 level 1 corrective action and 25 level 2 corrective actions. In a separate case, demonstrating the effectiveness of the corrective action process, the Laboratory issued a level 2 corrective action concerning the handling of chemical materials, based on an internal audit finding in 2011. The Laboratory then reviewed the effectiveness of the corrective action in 2012, through the normal quality assurance mechanisms, and discovered that the original issue persisted. The Laboratory issued another corrective action through its internally reported process to supersede the previous action and corrected the issue.

Overall, the FBI Laboratory’s corrective action process serves to identify areas for improvement and results in changes in operating procedures intended to improve the FBI Laboratory’s quality assurance framework.

Chemistry and Trace Evidence Unit Staff Met Annual Proficiency Testing Requirements and Their Training Files Were Generally Complete and Met FBI and Accreditation Requirements

The FBI Laboratory has developed a training approach to help ensure all caseworking employees are trained and qualified to conduct forensic casework. According to ASCLD/LAB, the training program’s requirements are consistent with international and accreditation standards for personnel conducting casework in forensic laboratories.

Forensic examiners and technicians are the two primary types of employees who conduct casework within the FBI Laboratory. In accordance with the FBI Laboratory’s Quality Assurance Manual, all FBI examiners and technicians are to complete an initial training program in which they must satisfactorily demonstrate competency in the relevant discipline(s) or categor(y/ies) of testing they will practice prior to assuming independent casework responsibilities. To help ensure competency prior to assuming casework responsibilities, the Laboratory’s training approach employs unit-specific training. For example, the Chemistry and Trace Evidence Units have each developed unit-specific training programs and manuals to train staff on their specific disciplines and positions. While training and testing requirements are specific to the discipline(s) and role the employee is to perform, every employee must pass competency tests and oral board examinations. In addition, forensic examiners, who provide testimony in court proceedings, must also complete moot court and admissibility hearing exercises, such as a Daubert hearing. Successful completion of all training requirements is to be documented in a “qualification and authorization” memorandum prepared by the unit chief or designee. Table 2 provides additional information on the initial training and qualification requirements for examiners and technicians in the Chemistry and Trace Evidence Units.

After completing the initial training program, caseworking employees must complete continued education and proficiency test requirements each year. For example, Chemistry and Trace Evidence Unit employees must complete a minimum of 15 hours of continued education related to job requirements or job performance each year. Chemistry Unit policy states that the continued education should focus on maintaining technical skills and expertise. In addition, each examiner and technician must complete an annual proficiency test in each category of testing that he or she performs. According to accreditation and FBI requirements, the Laboratory must use proficiency tests developed by an external test provider where available. On the basis of ASCLD/LAB assessments, all FBI Laboratory staff conducting casework or providing testimony in disciplines in which the Chemistry and Trace Evidence Units are accredited had successfully met annual proficiency testing requirements in their respective discipline(s) from 2011 to 2015 as required by accreditation standards.

Additionally, we observed that for the forensic anthropology discipline, wherein ASCLD/LAB does not provide accreditation, the two Trace Evidence Unit examiners who conducted casework in forensic anthropology completed annual proficiency testing in forensic anthropology from 2011 to 2015. Furthermore, the FBI Laboratory requires that all staff conducting casework in forensic anthropology obtain certification by the American Board of Forensic Anthropology. According to the Board, these two examiners were board-certified in 2012 and 2013, respectively, and have maintained certification.

The combination of training, continued education, and annual proficiency testing are intended to help provide reasonable assurance that the FBI Laboratory’s caseworking staff are appropriately trained and qualified. We reviewed the initial training records for the 47 employees who conducted casework in the Chemistry and Trace Evidence Units or provided court testimony related to chemical or trace evidence from fiscal year 2011 to July 2016 and found that the records were generally complete and met FBI requirements.

FBI Laboratory Has Processes to Review Laboratory Reports and Testimony, but Could Review More Examiner Testimonies

The FBI Laboratory implemented a monitoring program to help ensure that the results of its forensic examinations and any related examiner testimony are presented consistently with what is known about each forensic method. However, the testimony monitoring program is limited by difficulties in acquiring testimony transcripts.

FBI Approved Standards for Scientific Testimony and Report Language These standards document examples of the statements the FBI Laboratory has determined to be scientifically-supported and approved for its examiners to use when reporting examination conclusions and expert opinion in laboratory reports and court testimony for certain methods. practices, and unit procedures. Our review of the 166 laboratory reports produced by the Chemistry and Trace Evidence Units and used as the basis of testimonies from fiscal years 2011 through 2015 found that all had been technically and administratively reviewed in accordance with FBI requirements.

For example, according to the standards, FBI fiber examiners may state or imply that a textile fiber is natural or manufactured (man- made) and its type (e.g., cotton, wool, polyester). However, the examiner may not state or imply that a fiber came from a particular source to the exclusion of all others. The examiner also may not state or imply a statistical weight or probability to a conclusion or provide a likelihood that the questioned fiber originated from a particular source. The FBI Laboratory began developing and implementing these standards in 2014.

The court testimony monitoring program, in contrast, includes both internal and external reviews. Specifically, the court testimony monitoring program’s two key monitoring and oversight components include the following: (1) obtaining and reviewing transcripts of examiner testimonies to determine their compliance with FBI standards, referred to as internal evaluations; and (2) obtaining feedback on the quality of examiners’ testimonies directly from court officials, referred to as external evaluations. The FBI’s internal evaluation required that the reviewer— an appropriate manager and, when necessary, with assistance from a subject matter expert in the same area as the individual testifying— assess the examiner’s testimony against seven technical criteria. These include whether the examiner (1) testified accurately; (2) testified within the scope of his or her expertise; and (3) declined to answer questions beyond his or her expertise or beyond scientific limitations for the discipline, among other things. External evaluations, on the other hand, rely on the feedback of court officials who are not scientific or technical experts and who are asked to evaluate factors such as an examiner’s appearance, demeanor, promptness, ability to maintain composure as an expert witness, and ability to communicate results. See appendix I for further details about the FBI Laboratory’s practices for internal and external evaluation of testimony.

From fiscal years 2011 through 2015, a total of 22 forensic examiners in the FBI Laboratory’s Chemistry and Trace Evidence Units produced a total of 164 testimonies. However, the FBI Laboratory did not acquire transcripts and conduct internal evaluations for nearly half of these testimonies (78 of 164). As such, the 78 testimonies were not reviewed by a subject matter expert or against technical discipline-specific criteria, as provided for by internal evaluations.

The FBI obtained transcripts for 86 of the 164 testimonies and conducted an internal evaluation of the 86 transcripts. Additionally, of the 164 testimonies, the FBI obtained an external evaluation of the examiner’s performance for 99 testimonies. All of the internal evaluations of examiner testimonies that we reviewed received satisfactory assessments from an FBI manager. Also, in conformance with accreditation and FBI requirements and on the basis of our review of ASCLD/LAB reports, the FBI Laboratory met the accreditation requirement for testimony monitoring during this period.

While the FBI Laboratory’s testimony monitoring program meets current accreditation requirements, it could be improved by enhancing its testimony acquisition approach to allow it to conduct additional internal evaluations. Subjecting testimonies to an internal evaluation provides the strongest basis for understanding the extent to which examiners are adhering to FBI technical and reporting standards while also ensuring that examiner testimony is factually accurate. Furthermore, in March 2017, the merged accrediting body issued updated testimony monitoring requirements requiring that examiner testimony in each discipline be reviewed by an individual that has been competency tested in the task(s) that the review is encompassing. To meet the revised requirement, the FBI Laboratory will need to conduct an internal evaluation of examiner testimony. In other words, external evaluations by court officials, alone, will no longer satisfy the requirement for accreditation.

In explaining why the 78 testimony transcripts were not acquired, FBI Laboratory officials told us that because they could not compel courts to provide them with responses or transcripts, they have no way of knowing exactly why courts or court reporters did not provide specific transcripts. Anecdotally, they provided us with the following challenges and limitations they often encounter in their efforts to obtain transcripts:

There is no clear point of contact, or there is lack of responsiveness from points of contact, such as prosecutors or court reporters;

There is a lack of uniform computer and business systems across the separate federal, state and local courts in which they testify;

Some courts store their transcripts manually and lack electronic databases, making it difficult to locate and acquire transcripts; In some cases, transcripts may not be accessible because they have been sealed or may not have been generated.

While FBI cannot compel courts to provide transcripts in all instances, there are opportunities for the FBI to improve its ability to obtain more transcripts. To better understand the factors cited by the FBI Laboratory, we sought to obtain the remaining 78 transcripts directly from the courts. During this effort, we obtained 36 of the 78 transcripts (46 percent) that the FBI Laboratory did not obtain. Of the 36, we obtained:

16 transcripts by searching the Public Access to Court Electronic Records (PACER), an electronic public access service that allows users to obtain case information from federal appellate, district, and bankruptcy courts. This did not require personal contact with any court officials involved with the case;

9 transcripts by directly contacting and coordinating with the court or

11 transcripts through third-party transcription services or by directly visiting the court.

We were unable to obtain the remaining 42 of 78 transcripts for various reasons. In several cases, the information provided by the FBI was incomplete or additional information would be needed to locate the case and associated transcript. However, while attempting to obtain the outstanding transcripts, we also confirmed some of the reasons the FBI identified for not acquiring transcripts. For example, after identifying the court reporter, sometimes the reporter did not respond to our request(s) or provide us with the documentation necessary to request the transcript. Further, some testimonies could not be released because the cases were sealed. Lastly, some testimonies were not transcribed at the time of our review.

To develop better controls in its testimony monitoring program activities, in May 2013, the FBI Laboratory implemented a Testimony Tracker System (Tracker) to capture and monitor information on internal and external evaluation of examiner testimony and to help ensure that transcripts are requested, obtained, and reviewed. As of May 2013, FBI policy requires that appropriate data regarding the monitoring and evaluation of examiner testimonies—including when a testimony occurred, whether a transcript was requested, whether follow-up occurred, and when a transcript was received and reviewed, among other things—be recorded in the Tracker system. In addition, in December 2016, the FBI updated its policy to require that a unit chief or designee check the Testimony Tracker System monthly to ensure testifying individuals are entering required information. However, we found that the Tracker is not currently set up to routinely capture certain additional information that could help examiners conduct more effective follow up. For example, the Tracker does not capture the specific reason that a transcript has not been acquired. If a court reporter indicates to an examiner that a transcript will not be available for 1 year, being able to note the reason and the availability date or timeframe in the Tracker would better enable the examiner to follow up at the specified time. Or, if a specific transcript will not be available until after the case is completed, configuring the Tracker or revising the testimony monitoring policy to require examiners to record this information would better enable them to follow up on the transcript at a future date. Other information that could be helpful in obtaining a transcript includes the court’s jurisdiction, address, and a point of contact for the transcript. FBI officials confirmed that the Tracker is not currently configured to routinely record this type of information but did acknowledge that some information would be useful. A senior FBI Laboratory official also stated that recording more information, such as point of contact information, could help the FBI obtain more transcripts.

Obtaining transcripts and subjecting them to an internal evaluation is a key control in FBI’s efforts to ensure the quality of examiner testimonies. To facilitate this, each time an examiner testifies, FBI policy requires the examiner to request a transcript of the testimony from the respective court official so that it may be reviewed by an appropriate FBI manager or other subject matter expert. Further, Standards for Internal Control in the Federal Government calls for, among other things, controls to be designed to assure that ongoing monitoring occurs. The standards also state that relevant, reliable, and timely information is needed throughout an agency to control operations, achieve its objectives, and carry out its oversight responsibilities. Consistent with internal control standards, the FBI Laboratory could better ensure that it obtains more transcripts by routinely capturing and using additional information and data that are critical to their acquisition, such as the reason a transcript is unavailable, when it is expected to be available, the court jurisdiction, and a point of contact for the transcript. Increasing the number of transcripts acquired could help the FBI Laboratory expand its monitoring of examiners’ testimonies, which helps ensure that they are accurate, supported by the underlying analyses, and within the scientific limits of the given forensic discipline, as defined by FBI and accreditation standards.

Conclusions

The FBI Laboratory’s forensic work is used to support law enforcement across all levels of government. Forensic evidence can help solve crimes and influence whether a criminal defendant is acquitted, convicted, or even charged. Ensuring that forensic examiners produce scientifically valid and reliable examinations, laboratory reports, and testimonies for such cases is a stated goal of the FBI Laboratory quality system. The FBI Laboratory has a quality assurance framework that meets international and accreditation standards, including those for monitoring examiner testimonies. For example, the FBI Laboratory has established standards for scientific testimony and report language, employs training to help ensure its examiners adhere to the standards, and uses laboratory report and testimony review processes to help ensure examiners adhere to the standards. We found that the FBI Laboratory generally ensures that the Chemistry and Trace Evidence Units adhere to a variety of quality standards, including conducting audits, implementing corrective actions, ensuring staff have appropriate training, and reviewing laboratory reports.

The Laboratory’s internal evaluation of examiner testimony provides the strongest basis for understanding the extent to which examiners are adhering to FBI technical and reporting standards while also ensuring that examiner testimony is factually accurate. We found, however, that the FBI Laboratory could obtain more transcripts by enhancing the current procedure for acquiring testimony transcripts. Specifically, the Laboratory could ensure that it captures and uses information that facilitates the retrieval of a transcript, such as the reason the transcript may not be available and its court jurisdiction. Routinely capturing additional information could help the FBI Laboratory successfully obtain more transcripts to review, which can help better ensure that its examiners are providing testimony in court that is accurate, supported by the underlying analyses, and within the scientific limits of the given forensic discipline, as defined by FBI and accreditation standards.

Recommendation for Executive Action

To better ensure that the FBI Laboratory obtains additional transcripts, the FBI Director should require that the FBI Laboratory’s procedure for tracking and obtaining transcripts routinely captures and uses additional information and data critical to transcript acquisition, such as the reason a transcript is unavailable, when it is expected to be available, the court jurisdiction, and a point of contact for the transcript.

Agency Comments

We provided a draft of this report to the Departments of Justice and Commerce. In response, the FBI provided written comments, which are reproduced in full in appendix II. FBI concurred with the recommendation, and described actions planned to address it. Specifically, FBI concurred with GAO that the use of additional information and data is helpful in obtaining testimony transcripts. As such, the FBI Laboratory plans to modify its Testimony Tracker System accordingly to add additional information and available data for transcript acquisitions. The Department of Commerce did not provide written comments on our draft report. The FBI and the Department of Commerce also provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the Attorney General and the Secretary of Commerce and appropriate congressional committees, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions, please contact Diana Maurer at (202) 512-8777 or maurerd@gao.gov. Contact points for our Offices of Congressional Relations Public Affairs may be found on the last page of this report. GAO staff that made significant contributions to this report are listed in appendix III.

Appendix I: FBI Laboratory Practices for Internal and External Evaluations of Court Testimony

Listed below are the key steps involved in each of the internal and external evaluations of examiner testimonies delivered during court proceedings.

Internal Evaluations

1. Each time the examiner testifies he or she is responsible for requesting a transcript of the testimony from the court reporter(s), prosecutor, or other appropriate individual so it may be reviewed by an appropriate FBI manager or other subject matter expert. 2. If a copy of the transcript is not received after the initial request, the examiner is required to follow-up at least twice within 6 months of his or her testimony. 3. If a transcript is received, within 45 days of receipt, an appropriate FBI manager is required to review it and assess whether the examiner’s statements comply with the seven criteria in the Internal Evaluation of Testimony (7-256). 4. If the manager is not a subject matter expert, then a subject matter expert is to assist the manager in the evaluation. 5. After the review is complete, the manager and subject matter expert, if appropriate, who completed the Internal Evaluation of Testimony (7- 256) is/are to review the completed evaluation with the examiner within 45 days of receiving the transcript, and all parties are to document this review in the form.

External Evaluation

1. After each testimony, examiners are required to provide court officials, such as a prosecutor or defense attorney, with an External Evaluation of Testimony (7-257) to provide feedback on the examiner’s testimony. 2. The manager and subject matter expert, if appropriate, who received the External Evaluation of Testimony (7-257) is/are to review the completed evaluation with the examiner within 45 days of receipt, and all parties are to document this review in the form.

Appendix II: Comments from the Federal Bureau of Investigation

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Joseph P. Cruz (Assistant Director), Hazel Bailey, Amy Bowser, Jennifer Bryant, Clifton Douglas, Michele Fejfar, April Gamble, Susan Hsu, Kyle Maksuta, Claire Peachey, Lerone Reid, Sushil Sharma, Jack Sheehan, and Helina Wong made key contributions to this report.