Scientific & Technical Review

Panel Final Report for

OSAC 2021-S-0021

Forensic Autosomal STR DNA

Statistical Analyses - General

Protocol, Protocol Verification,

and Case Record Requirements

Organization of Scientific Area Committees (OSAC) for Forensic Science
1

STRP Final Report

OSAC 2021-S-0021

Forensic Autosomal STR DNA Statistical

Analyses - General Protocol, Protocol

Verification, and Case Record

Requirements

Organization of Scientific Area Committees (OSAC) for Forensics Science

August 18, 2022

Disclaimer:

This report was produced by an independent Scientific and Technical Review Panel (STRP). The

views expressed in the report do not necessarily reflect the views or policies of the U.S.

Government. Visit the OSAC website for more information on OSAC’s STRP process.

Scientific & Technical Review Panel Members

• Keith Inman, California State University East Bay

• Marla Kaplan, Oregon State Police - Portland Metro Crime Laboratory •

Pamela King, Minnesota Judicial Branch

• Adele Quigley-McBride, Duke Law, Wilson Center for Science and Justice •

Gary Shutler, Independent Forensic DNA Consultant

• Carl Sobieralski, Indiana State Police Laboratory

• Melissa Valadez, Texas Department of Public Safety

• Bruce Weir, University of Washington

• Sandy Zabell, Northwestern University
2

Report Summary:

The Scientific and Technical Review Panel (STRP) for “Forensic Autosomal STR DNA

Statistical Analyses - General Protocol, Protocol Verification, and Case Record Requirements” is

an independent panel appointed by the National Institute of Standards and Technology (NIST). A

STRP is established with a range of experts to consider how well a standard meets the needs of

the forensic science, law enforcement, and legal communities, and to recommend improvements

to the standards under review. The STRP appreciates the efforts of Michael Coble, Human

Forensic Biology Subcommittee affiliate, while serving as the subcommittee liaison to this STRP

during the review process.

The STRP began its review process with a kickoff meeting on July 8, 2021, and concluded with

this STRP final report. The panel reviewed the draft standard and prepared comments for the

Human Forensic Biology Subcommittee.

Report Components:

The STRP reviewed this draft standard against OSAC’s STRP Instructions for Review which

include the following content areas: scientific and technical merit, human factors, quality

assurance, scope and purpose, terminology, method description and reporting results. The details

below contain a brief description of each reviewed content area and the STRP’s assessment of

how that content was addressed in the Draft OSAC Proposed Standard.

1. Scientific and Technical Merit: OSAC-approved standards must have strong scientific

foundations so that the methods practitioners employ are scientifically valid, and the

resulting claims are trustworthy. In addition, standards for methods or interpretation of

results must include the expression and communication of the uncertainties in measurements

or other results.

1.1 Consensus View – The STRP recognizes that the proposed standard, for the use of

the calculations commonly employed for statistical analysis protocols of DNA

evidence, is based on valid principles in published literature. It contains relevant

references and general guidance about the approach for using each of the

protocols; the performance measures needed to verify the use of these protocols;
and for materials to be included in the case record. Documented verification of the

statistical analysis protocol used is a requirement for the laboratory. Uncertainties

are defined within the statistical approach taken and is dependent on the protocol

used. The document contains an informative Annex with basic information on the

three (3) major statistical approaches currently used in practice. The STRP

believes that including standards for CPI/CPE is important as outlined in the

footnote to “Annex A”. However, the document should strongly advocate that

laboratories adopt the use of likelihood ratios for the interpretation of evidence,

whether they use binary, semi-continuous or fully continuous methods, taking

3

advantage of the many options (open source and commercial) and opportunities

for training available. The OSAC criterion for technical merit is that a standard

has a strong scientific foundation. CPE/CPI is too controversial to have earned

that distinction. Although common in paternity testing, no research has ever

undertaken a published validation study, it arguably does not answer a legally

relevant question, and can potentially mislead a decision maker. Only one paper

(Bieber et al BMC Genetics, 2016, 17:125) describes a forensic casework method

and a basis for its use. The 2016 report by the President's Council and Advisors

on Science and Technology (PCAST) found “the interpretation of complex DNA

mixtures with the CPI statistic has been an inadequately specified—and thus

inappropriately subjective—method. As such, the method is clearly not

foundationally valid.” PCAST did agree that the rules specified in Bieber et al

were necessary, but due to the late release of the paper, they did not conduct a

review, nor did they take a position of whether reliability was addressed.

1.2 Minority View – NA

2. Human Factors: All forensic science methods rely on human performance in acquiring,

examining, reporting, and testifying to the results. In the examination phase, some standards

rely heavily on human judgment, whereas others rely more on properly maintained and

calibrated instruments and statistical analysis of data.

2.1. Consensus View – The STRP believes that this proposed standard adequately

addresses some of the issues related to human factors and performance.

Specifically, there is a verification requirement for evaluating consistency

among analysts in the laboratory, and the standards include a requirement to

define the acceptable range of variability. However, there are human factor

issues that are not yet addressed in this proposed standard and should be

considered when the document is next reviewed. For example, a high priority for

attention would be the Appendix A reference to propositions in LR calculations

that states “may be referred to as prosecution/defense.” It unfortunately does not

point out that the continued use of such terminology perpetuates arguments of

cognitive bias. During the next review, the subcommittee is also urged to
address the following matters:

2.1.1 Increasing transparency: More transparency in decision-making has many

benefits. This document begins to introduce documentation requirements

for certain aspects of the process, but there are other aspects that labs or

analysts are still left to determine, either in a general sense (e.g., creating a

policy) or on a case-by-case basis. Laboratory or analyst decisions and the

underlying reasons for those decisions (e.g., if there is research to support

that decision, if it is lab policy to make this decision, if there is a technical

reason that the decision is being made) should be documented fully in the

4

examiner’s notes with a summary in the report to inform the reader that

judgement was required in their decision.

2.1.2 Standardizing procedures: In several places throughout the document,

there are protocols and procedures that are described in a vague,

permissive manner so that laboratories can decide how to deal with those

matters. While this might make sense in some instances, each individual

lab will do things slightly differently, and sometimes laboratories will

implement procedures that are lower quality than desired. At a minimum,

providing options that are acceptable or being more specific about what

the laboratory policy must include, and what is less consequential, should

be explained in these clauses when this proposed standard is reviewed in

the future.

2.1.3 Increasing comprehension: Including examples and simple ways of

presenting information are immensely helpful to analysts and legal

professionals. Maybe these things are not appropriate to have in the

proposed standard itself, but tools and materials could be provided on an

open-source website (e.g., Open Science Framework) and the link

included in the proposed standard to help analysts and those who need to

understand the reports make sense of the analyses.

2.2. Minority View – NA

3. Quality Assurance: Quality assurance covers a broad range of topics. For example, a

method must include quality assurance procedures to ensure that sufficiently similar results

will be obtained when the methodology is properly followed by different users in different

facilities.

3.1. Consensus View – The STRP believes that quality assurance topics are

appropriately covered in this proposed standard. Requirements for robust

protocols are included that define when specific methods, options, and/or

assumptions are deployed to develop consistency within a laboratory.

Limitations of the statistical methods are also included. Evaluating consistency
among analysts in the laboratory and defining the acceptable range of variability

is a component of quality assurance. Evaluating consistency within the

laboratory’s statistical calculations utilizing probabilistic genotyping software

having an element of randomness to the results is also covered. Laboratories

would benefit from specific examples being incorporated into the proposed

standard to explain complex concepts as discussed in the comments on Human

Factors.

3.2. Minority View – NA

4. Scope and Purpose: Standards should have a short statement of their scope and purpose.

They should list the topics that they address and the related topics that they do not address.

5

Requirements, recommendations, or statements of what is permitted or prohibited do not

belong in this section.

4.1. Consensus View – The STRP considers the statements of scope and purpose to

be adequate. The wording in the proposed standard is compliant with its scope

and purpose.

4.2. Minority View – NA

5. Terminology: Standards should define terms that have specialized meanings. Only rarely

should they give a highly restricted or specialized meaning to a term in common use among

the general public.

5.1. Consensus View – The STRP finds that the proposed standard defines

appropriate terms that are specific to a statistical analysis work process. The

document avoids defining commonly used terminology.

5.2. Minority View – NA

6. Method Description: There is no rule as to the necessary level of detail in the description of

the method. Some parts of the method may be performed in alternative ways without

affecting the quality and consistency of the results. Standards should focus on standardizing

steps that must be performed consistently across organizations to ensure equivalent results.

Alternatively, standards can define specific performance criteria that are required to be

demonstrated and met rather than specifying the exact way a task must be done. For example,

it may be enough to specify the lower limit for detecting a substance without specifying the

equipment or method for achieving this limit of detection.

6.1. Consensus View –The STRP considers the proposed standard to meet the

Method Description requirement. This is based on inclusion of all three (3)

major statistical calculations encompassing the legacy CPI/CPE approach and

the more developed approaches of RMP and LR. The core requirements for each
approach are documented to perform the statistical analysis correctly. The

laboratory maintains the ability to choose the statistical approach to meet their

capabilities or if there is no other statistical calculation available. The

information shared with the STRP that ANSI/ASB Standard 018, First Edition

2020, Standard for Validation of Probabilistic Genotyping Systems addresses

the validation of probabilistic genotyping software and the acceptable degree of

variability for randomness of results was considered in this deliberation.

6.2. Minority View – NA

7. Reporting Results: Methods must not only be well described, scientifically sound, and

comprehensive but also lead to reported results that are within the scope of the standard,

appropriately caveated, and not overreaching.

6

7.1. Consensus View – The STRP recognizes that the proposed standard applies to

performing calculations resulting from the comparison of DNA profiles, but

reporting results is not within the scope of this document. Propositions and

conditioning used to calculate a LR does carryover into reporting and is within

the scope of this document. The STRP is of the opinion that the guidance

provided for the use of propositions and conditioning with the calculations are

adequately described to avoid misinterpretation and is not overreaching.

7.2. Minority View – NA
7