FINAL by: OSAC Program Office; Version 2, January 13, 2020 1

OSAC RESEARCH NEEDS ASSESSMENT FORM

Title of research need: Examiner Reliability Study: Black/White Box Study on Footwear and Tire 

Examiners

Describe

the need:

Evaluate the diversity of the interpretations provided by trained footwear/tire examiners 

when conducting comparisons of footwear/tire impression evidence test samples. 

Additionally, evaluate the diversity of the conclusions provided by trained footwear/tire 

examiners versus those provided by laypersons when conducting comparisons of 

footwear/tire impression test samples. Note: This portion of the study should only be 

performed after OSAC publishes a new articulation standard. The research outcomes 

should do some or all of the following: (1.) Quantify intra-examiner, inter-examiner and 

examiner-layperson variability as a function of the quality/quantity of the evidence 

provided, (2.) Quantify intra-examiner, inter-examiner and examiner-layperson variability 

as a function of the test taker’s education and discipline-specific training and experience, 

(3.) Identify aspects of the exam process and evidence that are sources of consistency in 

reporting conclusions, (4.) Identify aspects of the exam process and evidence that are 

sources of variability in reporting conclusions, (5.) Elucidate the process by which 

examiners assess and interpret footwear/tire impression evidence (e.g., quality, 

sufficiency, etc.). Note: Practitioner involvement in providing subject matter expertise 

during the planning phase of this research is highly encouraged in order to ensure that the 

research outcomes have applicability to casework, and the test samples are as realistic as 

possible under the research constraints.

Keyword(s): Footwear, Tires, Conclusions, Variability, Accuracy, Error

Submitting subcommittee(s): Footwear & Tire Subcommittee Date Approved: February 23, 2021

(If SAC review identifies additional subcommittees, add them to the box above.)

Background Information:

1. Does this research need address a gap(s) in a current or planned standard? (ex.: Field identification system 

for on scene opioid detection and confirmation)

2. Are you aware of any ongoing research that may address this research need that has not yet been published 

(e.g., research presented in conference proceedings, studies that you or a colleague have participated in but 

have yet to be published)?

FBI Footwear Black Box Study
FINAL by: OSAC Program Office; Version 2, January 13, 2020 2

3. Key bibliographic references relating to this research need: (ex.: Toll, L., Standifer, K. M., Massotte, D., eds. 

(2019). Current Topics in Opioid Research. Lausanne: Frontiers Media SA. doi: 10.3389/978-2-88963-180-3)

• Jacqueline A. Speir, Nicole Richetelli, Lesley Hammer. Forensic Footwear Reliability: Part I—Participant 

Demographics and Examiner Agreement. J Forensic Sci, Vol. 65, No. 6 (2020). doi: 10.1111/1556-4029.14553

• Nicole Richetelli, Lesley Hammer, and Jacqueline A. Speir. Forensic Footwear Reliability: Part II—Range

 of Conclusions, Accuracy, and Consensus. J Forensic Sci, Vol. 65, No. 6 (2020). doi: 10.1111/1556-4029.14551

• Nicole Richetelli, Lesley Hammer, and Jacqueline A. Speir. Forensic Footwear Reliability: Part III—Positive

 Predictive Value, Error Rates, and Inter-Rater Reliability. J Forensic Sci, Vol. 65, No. 6 (2020). doi: 10.1111/1556- 

 4029.14552

• L. Hammer, K. Duffy, J. Fraser, N. Nic Daeid. A study on the variability in footwear impression comparison 

conclusions. Journal of Forensic Identification. Vol 63, No. 2 (2013). pp. 205-218.

• B. T. Ulery, R. A. Hicklin, J. Buscaglia, M. A. Roberts, Accuracy and reliability of forensic latent fingerprint decisions. 

Proceedings of the National Academy of Sciences of the United States of America. Vol. 108, No. 19 (2011). 

pp.7733–7738.

• B. T. Ulery, R. A. Hicklin, M. A. Roberts, J. Buscaglia. Measuring what latent fingerprint examiners consider 

sufficient information for individualization determinations. PLOS ONE. Vol. 9, No. 11(2014). e110179 pp. 1-

 16.

• R. A. Hicklin, J. Buscaglia, M.A. Roberts, S. B. Meagher, W. Fellner, M. J. Burge, M. Monaco, D. Vera, L.R. Pantzer, 

C.C. Yeung,T. N. Unnikumaran. Latent Fingerprint Quality: A Survey of Examiners. Journal of Forensic 

Identification.Vol. 61, No. 4(2011). pp. 385-418.

• G. M. Langenburg. A Performance Study of the ACE-V Process: A Pilot Study to Measure the Accuracy, Precision, 

Reproducibility, Repeatability, and Biasability of Conclusions Resulting from the ACE--V Process. Journal of 

Forensic Identification.Vol. 59, No. 2(2009). pp. 219-256.

• H. Majamaa, & A. Ytti. A Survey of the Conclusions Drawn of Similar Footwear Cases in Various Crime 

Laboratories. Forensic Science International. Vol. 82, No. 1 (1996). pp.109-120.

• National Research Council. Strengthening Forensic Science in the United States: A Path Forward; The National 

Academies Press: Washington, D.C. (2009). Chapter 5, p.148.

4. Review the annual operational/research needs published by the National Institute of Justice (NIJ) at 

https://nij.ojp.gov/topics/articles/forensic-science-research-and-development-technology-working-groupoperational#latest? Is your research need identified by NIJ?

Yes.

5. In what ways would the research results improve current laboratory capabilities?

The results of this research would be considered by the footwear/tire examiner community, laboratories and 

accrediting bodies in order to implement necessary changes to the methodology, standard operation procedures, 

training programs and other quality assurance practices to reduce examiner error and minimize intra-/inter-examiner 

variation in reporting conclusions.

6. In what ways would the research results improve understanding of the scientific basis for the 

subcommittee(s)?

Examinations of footwear/tire impression can be extremely complex, and the factors influencing a comparison are not 

static from case to case. The variables at play are the value of the evidence (i.e., quality, quantity, clarity and 
FINAL by: OSAC Program Office; Version 2, January 13, 2020 3

limitations) and the examiner’s education, training and experience. As such, a degree of variation in opinion is 

expected. The results of this research would provide a better understanding of factors that influence the comparative 

and decision-making process. Those factors include: the quality of the evidence; class characteristics and randomly 

acquired characteristics (quality, quantity, clarity, complexity, number and size); the examiner’s education, training 

and experience; examiner certification and laboratory accreditation; and peer review.

7. In what ways would the research results improve services to the criminal justice system?

The research results would provide the ability to understand (and if possible, quantify) the conditions/factors that 

influence the examiner’s ability to analyze footwear/tire impression evidence and accurately interpret their findings. 

Further, the results would be extremely valuable in promoting transparency, objectivity, and the communication 

between experts and laypersons, particularly within the criminal justice system.

8. Status assessment (I, II, III, or IV): II

Major gap in 

current 

knowledge

Minor gap in 

current 

knowledge

No or limited

current research is 

being conducted I III

Existing current 

research is being 

conducted II IV

This research need has been identified by one or more subcommittees of OSAC and is being provided as an 

informational resource to the community.