1

Scientific & Technical Review

Panel Final Report for

2021-S-0004

Standard Practices for

Evaluating Measurement

Uncertainty of Quantitative

Measurements in Forensic

Toxicology

Organization of Scientific Area Committees (OSAC) for Forensic Science
2

STRP Final Report

2021-S-0004

Standard Practices for Evaluating

Measurement Uncertainty of Quantitative

Measurements in Forensic Toxicology

Organization of Scientific Area Committees (OSAC) for Forensics Science

September 21, 2021

Disclaimer:

This report was produced by an independent Scientific and Technical Review Panel (STRP). The

views expressed in the report do not necessarily reflect the views or policies of the U.S.

Government. Visit the OSAC website for more information on OSAC’s STRP process.

Scientific & Technical Review Panel Members

• Brigitte Desharnais, Laboratoire de sciences judiciaires et de médecine légale

• Jeff Kukucka, Towson University

• Dayong Lee, Houston Forensic Science Center

• Barry Logan, NMS Labs

• Paul Neuharth, Paul Neuharth, Jr. APC/Criminal Defense

• Lori Nix, Georgia Bureau of Investigation

• Mike Smith, FBI Laboratory

• Blaza Toman, National Institute of Standards and Technology (NIST)
3

Report Summary:

The Scientific and Technical Review Panel (STRP) for “Standard Practices for Evaluating

Measurement Uncertainty of Quantitative Measurements in Forensic Toxicology” is an

independent panel appointed by the National Institute of Standards and Technology (NIST). A

STRP is established with a range of experts to consider how well a standard meets the needs of

the forensic science, law enforcement, and legal communities, and to recommend improvements

to the standards under review. The STRP appreciates the efforts of Tate Yeatman, Forensic

Toxicology Subcommittee Vice Chair, while serving as the subcommittee liaison to this STRP

during the review process.

The STRP began its review process with a kickoff meeting on February 4, 2021 and concluded

with this STRP final report. The panel reviewed the draft standard and prepared comments for

the OSAC Forensic Toxicology Subcommittee.

Report Components:

The STRP reviewed this draft standard against OSAC’s STRP Instructions for Review which

include the following content areas: scientific and technical merit, human factors, quality

assurance, scope and purpose, terminology, method description and reporting results. The details

below contain a brief description of each reviewed content area and the STRP’s assessment of

how that content was addressed in the Draft OSAC Proposed Standard.

1. Scientific and Technical Merit: OSAC-approved standards must have strong scientific

foundations so that the methods practitioners employ are scientifically valid, and the

resulting claims are trustworthy. In addition, standards for methods or interpretation of

results must include the expression and communication of the uncertainties in measurements

or other results.

1.1. Consensus View – The STRP observed that the proposed standard adopts its

basic approach to calculating measurement uncertainty from the National

Institute of Standards and Technology, SOP 29-Standard Operating Procedure

for the Assignment of Uncertainty, Joint Committee for Guides in Metrology

(JCGM) Evaluation of Measurement Data-Guide to the Expression of

Uncertainty in Measurement (GUM) and similarly authoritative documents.

The standard is thorough in its approach and well-written. It contains suitable

references and clear guidance on the approach to calculating measurement

uncertainty. Clear guidance is supplied on reporting the results. The document

contains a number of useful example problems with detailed calculations. These

should prove useful to users seeking to implement the approach and reviewers

seeking to further evaluate the approach advocated by the document.

1.2. Minority View – None
4

2. Human Factors: All forensic science methods rely on human performance in acquiring,

examining, reporting, and testifying to the results. In the examination phase, some standards

rely heavily on human judgment, whereas others rely more on properly maintained and

calibrated instruments and statistical analysis of data.

2.1. Consensus View – The STRP believes that this draft adequately addresses issues

related to human factors and performance. Of principal importance, the

standard explicitly recognizes the relevance of human factors such as

experience, training, and fatigue in the quantification of measurement

uncertainty. Moreover, the computation of measurement uncertainty

incorporates reproducibility data from multiple analysts performing the same

measurement over time so as to account for both inter- and intra-examiner

variability, and the standard includes numerous informative annexes that

illustrate the application of this practice.

2.2. Minority View – None

3. Quality Assurance: Quality assurance covers a broad range of topics. For example, a

method must include quality assurance procedures to ensure that sufficiently similar results

will be obtained when the methodology is properly followed by different users in different

facilities.

3.1. Consensus View – The STRP believes that quality assurance topics are properly

covered in this draft standard. Measurement uncertainty, the topic of this

standard, is an inherent part of quality assurance. Furthermore, other related

items, such as metrological traceability, quality control, and method

validation are properly included and referenced throughout the draft standard.

3.2. Minority View – None

4. Scope and Purpose: Standards should have a short statement of their scope and purpose.

They should list the topics that they address and the related topics that they do not address.

Requirements, recommendations, or statements of what is permitted or prohibited do not

belong in this section.

4.1. Consensus View – Upon review, the STRP sees no reason to modify the

proposed language defining "Scope and Purpose" as currently set forth in the

document. The document also complies with the proposed language in that the

terminology is appropriate and the draft standard's scope and purpose are

appropriate.

4.2. Minority View – None
5

5. Terminology: Standards should define terms that have specialized meanings. Only rarely

should they give a highly restricted or specialized meaning to a term in common use among

the general public.

5.1. Consensus View – The STRP finds that the draft standard defines appropriate

terms with specialized meaning within the metrology application of uncertainty

of measurement, specifically to toxicology. The document balances the need for

definitions while avoiding defining commonly used terminology. The STRP

recommends using OSAC preferred terms, when available.

5.2. Minority View – None

6. Method Description: There is no rule as to the necessary level of detail in the description of

the method. Some parts of the method may be performed in alternative ways without

affecting the quality and consistency of the results. Standards should focus on standardizing

steps that must be performed consistently across organizations to ensure equivalent results.

Alternatively, standards can define specific performance criteria that are required to be

demonstrated and met rather than specifying the exact way a task must be done. For example,

it may be enough to specify the lower limit for detecting a substance without specifying the

equipment or method for achieving this limit of detection.

6.1. Consensus View – The STRP considers that the proposed standard meets the

Method Description requirement. This opinion is based on the fact that the

standard provides standardizing steps (i.e., eight steps on how to evaluate and

report measurement uncertainty) with specific criteria when applicable (e.g.,

4.2.4.3 – minimum requirements for Type B evaluations) and describes

alternative ways to evaluate uncertainty depending on the conditions of

difference methods (e.g., 4.2.4.2.1.2 – whether a method shows constant or

different variance across the calibration range). The STRP verified that the

standard minimizes ambiguity by presenting the eight steps in detail and

providing examples (e.g., 4.2.2) throughout the main text as well as via four

annexes; lists specific factors to include for accurate calculation of uncertainty

(i.e., 4.2.3); and specifies when and how to adjust a method's procedure if a

limitation is found during the uncertainty evaluation process (e.g., 4.2.6.1 –

evaluation of bias).

6.2. Minority View – None
6

7. Reporting Results: Methods must not only be well described, scientifically sound, and

comprehensive but also lead to reported results that are within the scope of the standard,

appropriately caveated, and not overreaching.

7.1. Consensus View – The STRP believes that the standard adequately describes

how the results should be presented considering what is known of their

reliability and accuracy. It provides clear instruction on how estimates of

uncertainties are to be calculated and reported. The worked examples in the

standard are helpful in this regard.

7.2. Minority View – None