1

Report of the Digital Evidence Task 

Group Quality Study 

Prepared for 

The Organization of Scientific Area Committees for Forensic Science (OSAC) 

Prepared by: 

OSAC Task Group on Quality Practices in Digital Forensics Laboratories 

Date November 2022

https://doi.org/10.29325/OSAC.DE.0001
Table of Contents

EXECUTIVE SUMMARY 1

PROJECT OVERVIEW AND GOALS 1

PROJECT METHODOLOGY 2

DESCRIPTION OF INTERVIEWEES 3

SELECTION BIAS OF INTERVIEWEES 7

ORGANIZATION OF THE REPORT 7

RESULTS: LABORATORY PERSONNEL 8

1. DOCUMENTATION: HOW DOES YOUR LABORATORY VIEW AND USE DOCUMENTATION? 8

2. TOOL VALIDATION AND TESTING: DOES YOUR LABORATORY CONDUCT TOOL VALIDATION AND TESTING?

WHY OR WHY NOT? 10

3. MANAGEMENT REVIEWS: HOW DOES YOUR LABORATORY UTILIZE MANAGEMENT REVIEWS AS PART OF A 

QUALITY MANAGEMENT SYSTEM? 12

4. PEER REVIEW: WHAT IS THE VALUE OF PEER/TECHNICAL REVIEWS FOR FORENSIC REPORTS? 13

5. AUDIT: HOW DOES YOUR LABORATORY ADDRESS AUDITS AND ASSESSMENTS? 15

6. TESTIMONY MONITORING: DOES YOUR LABORATORY CONDUCT TESTIMONY MONITORING? WHAT IS THE 

VALUE IN TERMS OF QUALITY FOR THIS ELEMENT? 16

7. PERSONNEL: WHAT ARE KEY ELEMENTS OF COMPETENT FORENSIC LABORATORY PERSONNEL IN YOUR 

LABORATORY? 17

8. QUALITY MANAGER: WHAT IS THE VALUE OF A QUALITY MANAGER IN YOUR LABORATORY AND WHAT IS 

THIS ROLE’S EFFECT ON THE QUALITY OF YOUR LABORATORY’S WORK? 18

HIGHER LEVEL LOOK AT QUALITY PROCESSES AND ACCREDITATION 19

SUMMARY OF LABORATORY INTERVIEWS 20

RESULTS: CUSTOMER INTERVIEWS 21

WHAT DO CUSTOMERS VALUE IN A LABORATORY? 22

WHAT ARE THE MAIN PROBLEMS THEY HAVE EXPERIENCED? 23

WHAT DO CUSTOMERS WANT MOVING FORWARD? 24

SUMMARY OF CUSTOMER INTERVIEWS 25
FINDINGS AND RECOMMENDATIONS 25

FINDINGS 26

RECOMMENDATIONS 27

APPENDIX A: SURVEY QUESTIONS (LABORATORY PERSONNEL) 29

APPENDIX B: SURVEY QUESTIONS (CUSTOMERS) 32

APPENDIX C: QUALITY MANAGEMENT SYSTEMS 34
1

Executive Summary

In 2020, the Organization of Scientific Area Committees (OSAC) for Forensic Science’s Digital 

Evidence Subcommittee formed a task group to identify the quality practices and management 

systems that are most effective within digital forensics laboratories. The task group identified 

contemporary quality management practices used in forensic labs and interviewed laboratory 

management, senior staff and digital evidence customers (e.g., law enforcement (LE), 

attorneys, judges). This report is based on the interviewees’ experiences with quality practices 

and management systems. 

The study found significant problems with how quality management systems (QMS) are 

implemented within digital forensics laboratories generally, though it did not uncover problems 

with the accuracy of any specific report. The study also found problems with timeliness and the 

ability of consumers of laboratory products to understand and make use of reports generated by 

digital forensic laboratories. The study found that accreditation under ISO 17025:2017 and ISO 

17020:2012 and formal QMSs are not having the desired outcome of enhancing the accuracy, 

timeliness, and clarity of reports. Current efforts to improve the processes are not resulting in 

significant gains and are, in many cases, reducing quality. Some quality elements took 

significant resources with no benefit and others actually reduced timeliness and clarity. 

However, this study documented many examples where the implementation of a QMS had 

significant value. In most of these cases, the QMS had been optimized for digital forensics.

Based on these findings, the study recommends the development of a QMS optimized 

specifically for digital evidence processing. This new system should be tailored to maximize the 

improvement of quality that is most important to digital evidence. A digital forensics QMS should 

focus on quality elements relevant to digital evidence processing rather than shoehorning digital 

processing into existing frameworks. Since this is a significant undertaking, the task group also 

recommends some incremental improvements. 

Project Overview and Goals

The 2009 National Research Council report, Strengthening Forensic Science in the United 

States: A Path Forward and the 2016 President’s Council of Advisors on Science and 

Technology (PCAST) report on Forensic Science in Criminal Courts: Ensuring Scientific Validity 

of Feature-Comparison Methods recommended significant changes to the practice of forensics

necessary to ensure the delivery of accurate, trustworthy, understandable results to 

downstream forensic science “customers” (e.g., lawyers, judges, investigators) in a reasonable 

timeframe. This objective is central to the administration of justice, particularly in criminal 

matters, including a criminal defendant’s Constitutional rights to due process and access to 

potentially exculpatory information. To date, efforts to accomplish this objective across forensic 
2

disciplines have included a focus on quality management including training, standardized 

methodologies, and consistency of forensic testing and analysis.

As with other forensics disciplines, quality management is essential to the field of digital 

forensics. However, there has never been a study of contemporary quality management 

practices in the field of digital forensics. Today, there are a wide range of approaches, from 

formal accreditation to informal protocols. And, critically, there is a significant disparity of opinion 

in what might be the proper path for the future of digital forensic quality management. 

Project Methodology

The task group collected information relating to current quality management practices in several 

different forensic laboratory environments through interviews with individuals from local, state, 

and federal law enforcement, private organizations, and digital forensic science practitioners. All 

but three of the participants worked in the United States. Information was also collected from the 

“customers” of the output from digital evidence examinations, including attorneys and members 

of the judiciary.

This research underwent human subject protection review and was approved by the NIST IRB 

(ITL-2020-0245). Interviewees were recruited by announcements at digital forensics 

conferences and forums and by personal outreach from the task group. 

The interviews included questions tailored to an individual interviewee’s status as either 

laboratory personnel or a customer. The laboratory management survey included questions 

regarding quality management elements such as the role of standard operating procedures 

(SOPs) and tool testing in improving laboratory processes as well as additional questions 

related to accreditation and quality management. The survey for customers focused on their 

experience working with digital forensics laboratories. The surveys are provided in Appendices 

A and B. Task group members were allowed to ask follow-up questions. Interviewees were 

encouraged to provide details or examples to support their discussion points. Most interviews 

had two or three members of the task group in attendance; one had only a single interviewer.

Many of the interviewees were either not familiar with some of the quality elements that were 

the focus of the laboratory survey or were using the term based on how their laboratory 

operated. (A description of the quality elements is provided in Appendix C.) For example, 

laboratories had different meanings for many elements, including tool validation and testing, 

auditing, and management reviews. If an interviewee asked for clarification, it was provided, but 

the task group allowed interviewees to base their answer on their understanding or 

interpretation of the quality element being discussed. 

The task group consisted of members and affiliates of the OSAC Digital Evidence, 

Video/Imaging & Analysis, and Anthropology Subcommittees along with members of the 

Forensic Science Standards Board’s Human Factors and Legal Task Groups. The Task Group

Members performed this study in their capacity as members of the Organization of Scientific 
3

Area Committees for Forensic Science. The views and opinions expressed in this document do 

not necessarily reflect the official policy or position of the author’s employers, NIST, the 

Department of Commerce or the U.S. government. The members of the Task Group are listed 

below. The interviewers are designated with an asterisk (*).

Task Group Member OSAC Subcommittee 

William Eber Affiliate, Digital Evidence 

Brandon Epstein* Member, Video/Imaging Technology & Analysis 

Sabrina Fève Member, Legal Task Group/Digital Evidence

Daren Ford Member, Digital Evidence

Barbara Guttman* Vice Chair, Digital Evidence

John Hollway Member, Human Factors Task Group/Digital Evidence 

Christopher Kelly* Member, Digital Evidence

Troy Lawrence* Member, Digital Evidence

Jim Lyle Affiliate, Digital Evidence

Kelly Sauerwein Member, Human Factors Task Group/Anthropology

Shaun Vodde* Member, Digital Evidence

Tracy Walraven* Chair, Digital Evidence

Description of Interviewees

The task group conducted 31 interviews consisting of 17 laboratory personnel and 14 legal 

customers. The demographics are displayed below, however, some individuals self-identified 

themselves or their laboratory as having multiple roles but are listed according to the role used 
4

to answer the questions. Note: In the charts below, DE is Digital Evidence and CJ is Criminal 

Justice.
5

The tables below provide a breakdown of the demographics based on each interviewee. Many 

interviewees had worked in multiple locations or were retired. The demographic is given for their 

primary affiliation. When another affiliation was significant, this is noted in the Additional 

Information column. International interviewees are categorized based on their similarity to a 

federal or state/local type of laboratory. Answers provided by the interviewees are used to 

support the conclusions drawn, however, these are not direct quotes from the interviewee, but 

instead are summarized from the interviewers’ notes. Interviewers collected separate 

summaries of responses, which were consistent.

Interviewee # 

(Laboratory 

Personnel)

Primary Role Laboratory Type Additional Roles

Interviewee 1 Management Medium state/local 

laboratory

Interviewee 2 Management Medium state/local 

laboratory

Also ANAB assessor

Interviewee 3 Management Small private 

laboratory

Also trainer

Interviewee 4 Management Medium state/local 

laboratory

Interviewee 5 Examiner Small state/local 

laboratory

Interviewee 6 Quality management 

and examiner

Large state/local 

laboratory

Interviewee 7 Management Medium state/local 

laboratory

Interviewee 8 Examiner Small laboratory 

local lab

Interviewee 9 Trainer Large federal 

laboratory

Interviewee 10 Management Medium state/local 

laboratory
6

Interviewee 11 Examiner Medium state/local 

laboratory

Interviewee 12 Management Medium federal 

laboratory

Interviewee 13 Management Large federal 

laboratory

Interviewee 14 Management Private laboratory Also an academic

Interviewee 15 Quality management Large state/local 

laboratory

Interviewee 16 Management Small private 

laboratory

Interviewee 17 Management Large federal 

laboratory

Interviewee #

(Customer)

Primary Role Used to 

Answer Interview Questions

Additional 

Roles

Interviewee 18 Prosecutor

Interviewee 19 Prosecutor Also a trainer, previously defense 

attorney

Interviewee 20 Police chief

Interviewee 21 Prosecutor

Interviewee 22 Prosecutor

Interviewee 23 Prosecutor Also a trainer
7

Interviewee 24 Prosecutor

Interviewee 25 Prosecutor

Interviewee 26 Prosecutor

Interviewee 27 Prosecutor Currently a judge

Interviewee 28 Defense attorney

Interviewee 29 Defense attorney

Interviewee 30 Defense attorney

Interviewee 31 Defense attorney

Selection Bias of Interviewees

Interviewees were recruited through the use of announcements at digital forensics conferences 

and forums and by personal outreach from the task group. Additional participants were recruited 

with assistance from study interviewees. The task group continued to recruit new interviewees 

for interviews until consistent patterns emerged, stopping when interviewees were no longer 

introducing new information or patterns. 

Organization of the Report

The analysis of the interview results is organized according to interviewee type (i.e., laboratory 

personnel or customer) with each interview topic discussed. The responses for each interview 

topic are summarized and grouped into three or four themes that emerged from the interviews. 

Summaries from the interviewers’ notes were utilized to highlight and support each of the 

themes with additional analysis provided. While the interviewers’ notes are not direct quotes 

made by the interviewee, they do reflect what the interviewees said. A summary provides key 
8

takeaways from the analysis of each quality element. Overall findings are discussed and key 

recommendations for improving the quality management process are provided.

Results: Laboratory Personnel 

The survey included questions covering eight topics addressing common elements of quality 

management systems. The elements were drawn from common quality practices. Appendix C 

provides additional references about quality elements and quality systems. The quality elements 

covered were:

1. documentation, 

2. tool validation and testing, 

3. management and peer reviews, 

4. audits, 

5. testimony monitoring, 

6. personnel qualifications, 

7. presence of a quality manager, and 

8. accreditation. 

Experiences about strengths, weaknesses, and efficacy of these systems was also asked of the 

interviewees (see Appendix A for interview questions). Each of these elements is described and 

analyzed in detail below. The task group asked interviewees to address quality in terms of 

accuracy, completeness, and timeliness.

While this study was not designed to evaluate the correctness of any specific reports, all

laboratory interviewees felt that their laboratories were generating accurate reports and work 

product. By contrast, timeliness – the ability to generate reports within a reasonable time frame 

for downstream constituents and the efficient flow of the justice system – was viewed as an 

endemic problem. Overall, interviewees expressed a commitment to quality but the techniques 

they were using were often ineffective to achieving their stated goals, mostly by creating 

inefficiencies, and not focusing on higher-level quality goals. 

A summary of all laboratory questions asked is provided at the end of this section.

1. Documentation: How does your laboratory view and use 

documentation?

Overall, documentation was considered useful in the majority of laboratories, but few 

interviewees could explain documentation’s role in a quality management system. (See 

Appendix C for references about the quality elements.) A lot of the documentation discussion 

used standardized terms (from ISO 17025) without demonstrating an understanding of how this 

related to elements of quality. However, what people considered to be “documentation” varied 

considerably, ranging from SOPs and chain of custody to case notes. The interviewees found 

that chain of custody and case note SOPs were the most useful types of documentation.
9

Responses to the survey fell into four themes: 

⚫ Theme 1: Documentation was considered critical with a focus on laboratory operations 

including managing customer requests, maintaining chain of custody, and tracking of 

various laboratory operational tasks such as tool validation testing or proficiency testing.

Interviewees generally valued documentation for managing expectations and managing 

general operations within the laboratory. Several interviewees saw a connection to 

improved laboratory operations, which increased efficiency.1

 Interviewee 14 noted that documentation was good for three things: 

1) efficiency and helping customers know what they are going to get. 2) accuracy 

and more dependable products. 3) morale because examiners are happy to 

know that they are working to an acceptable level.

 Interviewee 15 introduced streamlined forensic reporting instead of each person 

doing their own report style. The reporting structure has two phases (a basic 

report and a more in depth one that is only produced if requested) for increased 

efficiency. Other parts of their crime laboratory have picked up this approach and 

all of them are generating reports faster.

⚫ Theme 2: Documentation was considered critical but was focused only on case work to 

include case notes and a detailed description of analysis. Interviewees highlighted the 

need for the consistency of process between examiners. Case notes were deemed 

critical to assist the examiner in the recollection of the examination for testimony 

purposes. Documentation, particularly SOPs, was used to provide structure to the 

laboratory and organizations, and for training new examiners. It was also noted that 

laboratory SOPs were consistently written as high-level overviews to avoid being too 

prescriptive, since digital examinations often require flexibility. Interviewees noted the 

tension between standardization and flexibility.

 Interviewee 4 described documentation about multiple laboratory processes such 

as authorizing tools land basic workflows. Interviewee thought this was useful for 

training and testimony reference.

 Interviewee 5 described documentation as more of a guideline. The interviewee 

did not think documentation improved quality in terms of accurate reporting but 

felt that it might be helpful for training new examiners in the laboratory.

⚫ Theme 3: Documentation was considered critical for potential challenges in court.

Several interviewees stated that documentation can be used in court to show that the 

laboratory is following best practices or standardized procedures.

 Interviewee 6 described the utility of case notes for testimony, for communicating 

with customers about what is in reports, and for determining time management of 

tasks as well.

 Interviewee 2 stated that they can't live without documentation because 

prosecutors and defense ask for it in court.

1 Reminder: the bullets are the interviewers’ notes summarizing the interviewee’s answers and are not direct quotes from the 

interviewee.
10

⚫ Theme 4: Documentation was poorly regarded. Several interviewees thought that most, 

but not all, documentation was either a waste of time or actually made reports worse. 

Reports were more bureaucratic and harder for the customers to understand.

 Interviewee 1 thought that the documentation required for accreditation had no 

impact on quality - just made the reports harder to read for the customer. One of 

the changes the laboratory made "simplified" reports but left out useful material.

 Interviewee 10 described how at their accredited laboratory, the paperwork was 

challenging, not practical and slowed the process down considerably. Digital 

forensics laboratories were being pulled under the same umbrella as other 

forensic sciences causing duplication of work and inefficiency. They pulled out of 

the accreditation umbrella but streamlined procedures with chain of custody 

documentation remaining in place because it is critical. The essential piece was 

that the examiners used good forensic practices and procedures.

 Interviewee 8 stated that they didn’t have SOPs because the interviewee thought 

most SOPs were a waste of time. The Interviewee did have enough case notes 

to recount the reasons as to why actions were performed in an examination.

Summary: Targeted forms of documentation, especially for chain of custody, case notes, and 

customer interactions helped laboratories improve their quality. Other forms, especially 

extensive SOPs, were less useful, except as training aids.

2. Tool Validation and Testing: Does your laboratory conduct tool 

validation and testing? Why or Why not? 

In general, testing tools prior to use (often referred to as validation) was viewed as resourceintensive and was seen as taking away from the primary mission of the laboratory, even though 

laboratory personnel described it as important. As with other quality elements, there was 

significant misunderstanding about the element itself. (See Appendix C for references about the 

quality elements.) This may be partially based on a lack of knowledge about how to perform 

meaningful testing, which might require a fairly in-depth knowledge of computer engineering, file 

systems, operating systems, database design, or other knowledge of software engineering.

It was concerning that there were several statements from laboratory professionals that showed 

a true misunderstanding of how to perform testing and the benefits and limitations of validation 

and testing.

The comments generally fell into 4 themes:

⚫ Theme 1: Interviewees stated that they valued tool testing but were doing only minimal 

testing. Of these, some felt the concept was important, but they were not able to 

effectively implement a testing program. 

 Interviewee 12 says that they never identified software that was working 

incorrectly, but that testing did reveal unexpected observations, which the 
11

Interviewee thought were presentation issues. Interviewee said it was almost 

impossible to do validation for all laboratory tools.

 Interviewee 9 thinks that validation as it is done now is not good. The interviewee 

thought testing needed more transparency, shared testing, standard datasets,

and a system where the testing load is shared between vendors and a 3rd party. 

The Interviewee thought this would result in greater laboratory efficiency (since 

tool testing takes time away from examinations) and increased testing, enabling 

newer versions of tools to be used in the labs rather than using older versions. 

 Interviewee 1 recounted having to explain to their quality assurance team why 

digital forensic software is not like biology tools that resulted in added work the 

interviewee considered unnecessary. 

⚫ Theme 2: The majority of interviewees said validation testing made their lives worse 

because the value did not justify the time commitment involved in proper tool testing and 

validation.

 Interviewees 9 and 14 thought that testing was too resource intensive. 

Interviewee 9 even stated that the poor testing of tools was “a low point for the 

community.” 

 Interviewee 12 described that the testing done was quite minimal and was limited 

to checking whether the tool in use obtained the same answer as another tool. 

Interviewee described the difficulty of keeping up with tool updates.

 Interviewee 14 described a time when an error in the tool causing mix-ups 

between time and date stamps on text messages was caught, not with testing, 

but during an examination. This resulted in the vendor releasing a patch. 

⚫ Theme 3: A few interviewees were able to describe actual benefits of tool testing.

 Interviewee 15 described that their laboratory has a technical lead assigned 

specifically for validation due to the volume of work caused by changing 

technology and the constant need for validation of methods. Having a person 

who did this helped with making the approach to testing more consistent and 

resulted in tool limitations being clearly defined. This has led to greater 

consistency across a large laboratory system. While this process was viewed as 

onerous, the interviewee felt that standardization was helping balance the 

workload. 

 Interviewee 7 said that performance checks made the tools better, but it takes 

longer. They described catching the tools not parsing data completely in mobile 

devices and that performance checks added more confidence that the report 

going out the door was complete.

Summary: The practice of tool validation and testing is inefficient and poorly understood by 

many laboratories and they do not take into account best practices for software testing. The 

inefficient and ineffective testing results in significant costs to the laboratory. These include:

⚫ Inability to update tools to take advantage of new versions (which are generally better 

than old versions).

⚫ Wasted time.
12

⚫ Misplaced confidence. Several interviewees felt better about their tools after having 

performed tests that were not rigorous enough to justify their findings.

⚫ Lack of attention to the human factors involved with tool usage.

3. Management Reviews: How does your laboratory utilize 

management reviews as part of a quality management system?

Very few of the interviewees were using management reviews as an opportunity for process 

review and improvement. Differences among laboratories were quite stark. A few were using

them to meet other non-quality related objectives. A small number of the interviewees had 

thoughtfully optimized management reviews for the digital forensic process, but most of the 

other laboratories that performed management reviews seem to be merely “checking a box.” In 

addition, laboratories had differing interpretations of what constituted a management review. 

Accredited laboratories defined management reviews as the annual meeting to discuss trends, 

deviations, errors, personnel, training, and overall work of the digital laboratory. Non-accredited 

laboratories defined management reviews ranging from a management review of technical work 

product to a weekly meeting with management to discuss issues. Some non-accredited 

laboratories recognized how management review could benefit the laboratories but were mostly 

performing a more administrative review. 

Interviewee comments generally fell into 3 themes:

⚫ Theme 1: Many of the laboratory personnel saw management reviews as mostly 

nitpicking by their management. They saw only minimal value in the process, which was 

mostly focused on failure to follow various procedures. These interviewees did generally 

see some value in the procedure, but they did not see it as worth the effort. 

 Interviewee 13 described a focus on small details with no tie to quality.

 Interviewee 9 had only done one of these with someone who had digital forensic 

(DF) experience but found that situation to be the only time that such a review 

had value, in their opinion. Other such exercises were viewed as “rubber 

stamping.” The person with DF experience had subject-matter experience and 

could provide that helpful feedback

⚫ Theme 2: Many laboratories used the management review as a way to communicate 

with management about activities in the digital forensics section and to ask for 

resources. Many of the interviewees were from general purpose forensics laboratories 

and had management that was unaware of the specifics of digital forensic operations. 

 Interviewee 10 described their laboratory’s review process of people, oversight of 

the unit, and the work product, including both in-house management and 

onsite/response work. They emphasized that reviewing work products (reports 

and output) was helpful for monitoring how much work was being done as well as

checking on the well-being and mental health (crimes against children in 

particular) of their employees.

⚫ Theme 3: A few of the interviewees were using management reviews as an opportunity 

for process improvement.
13

 Interviewee 2 described using the process to improve evidence handling, intake 

and archiving. Examiners didn’t think that their evidence was as important as 

DNA and didn't think evidence could intermingle/contaminate. This required 

some effort since the concept of contamination is quite different for DNA versus 

digital evidence.

 Interviewee 14 described management reviews as part of a continuous 

improvement process. They sit down every 12 months and look at what has 

changed, why it has changed, and whether this is explained to staff. Interviewee 

14 stated that it is important and makes sure processes are fit for purpose. This 

helps keep SOPs up to date and accurate

 Interviewee 15 described a specific improvement in which management reviews 

identified a bottleneck with peer reviews and enabled a productive change to that 

process.

Summary: Management reviews can be used for process improvement, but very few 

laboratories were using them this way. The disconnect between the potential benefits of 

management reviews and their implementation was particularly striking; this included accredited 

laboratories which should have a quality management process in place to use this element 

effectively.

4. Peer Review: What is the value of peer/technical reviews for 

forensic reports?

Peer review, along with having competent personnel, was the most highly regarded quality 

control element. Peer review had multiple positive impacts on overall quality including helping 

examiners learn from each other, streamlining operations beyond the scope of a single review, 

and reviewing the work performed. Peer review programs that optimized the collaborative 

aspect were viewed much more favorably. There was a general trend suggesting that the 

formality of the peer review process decreased its effectiveness.

The comments generally fell into three themes:

⚫ Theme 1: Peer review was well-regarded and integrated into laboratory operations. Peer 

review was often used as a means of information sharing and process improvement.

 Interviewee 9 stated that peer reviews “improve the report, improve the reviewer, 

improve the laboratory.”

 Interviewee 10’s laboratory had specialists in various areas (Mac forensics, cell 

phones, etc.) and reviews were done informally. Interviewee 10 thought that 

more informal peer reviews were helpful and more productive than when they are 

performed under the accreditation umbrella. 

 Interviewee 13 said that peer reviews help to ensure the laboratory meets quality 

standards. Furthermore, reviews lead to analysts talking with one another and 

personal advancement. Interviewee 13 also reported that peer reviews created 

competition amongst analysts which upped the value of work and helped inform 

adjustments to the quality manual. 
14

 Interviewee 17 reported that peer reviews were performed in 100% of cases and 

were helpful for the purposes of developing efficiency, investigative methods, 

learning, etc. They also were looking for things that could be problematic at trial. 

Overall, peer reviews were more of a learning experience than an anomalyfinding experience.

⚫ Theme 2: Peer reviews were well-regarded but described as a task with no apparent 

benefit to quality. Interviewees described finding no or only minor (e.g., grammatical) 

errors. It was accepted as a part of quality management, but people couldn’t say why. 

This is part of a general trend observed about the adoption of quality management 

techniques without regard to actual quality improvement.

 Interviewee 1 described peer review as catching small errors such as copy and 

paste errors, grammatical, and administrative errors, or missed items based on 

the SOPs, but did not find many technical errors.

 Interviewee 4 also described not finding anything substantial but thought peer 

review was important because “we don't know what we are missing.”

 Interviewee 15 described how this helped build trust with customers, especially if 

a case goes to court. They also said that the technical reviews haven't caught 

much and have never led to rework. 

⚫ Theme 3: Peer reviews were viewed as problematic. A few of the interviewees 

addressed problems with peer review including delays, potential negative impact on 

morale and, quite distressingly, a significant specific negative impact. In one case, the 

requirement for 100% peer review led to a delay in providing information needed for a 

crime in progress. The Interviewee did report that a new procedure was put in place for 

urgent situations.

 Interviewee 2 described how a requirement for peer review for all cases led to 

not providing information in a timely manner for an urgent case (PD needed a 

phone number related to a crime in progress). The interviewee called this the 

“price of an accredited laboratory.” It makes the work harder. (Note the laboratory 

did update its SOPs to allow for release of information for urgent cases.)

 Interviewee 12 said that peer review was not done regularly - only when a 

problem was identified. However, Interviewee 12 did say that examiners often 

sought advice from each other. Interviewee 12 was concerned that double 

checking people’s work could negatively affect morale and that the resulting 

friction could actually cause mistakes and create a toxic work environment. They 

hoped that framing this as a quality review instead of checking on people could 

help.

Summary: A collaborative peer review process can be a very effective part of a quality system 

for digital forensics. When properly implemented, it has significant benefits for information 

sharing, staff development, and morale. As with other quality elements, many laboratories 

viewed peer review as simply an editorial review or (even worse) a “gotcha” environment in 

which reviewers were deemed to be actively seeking mistakes from the report writers.
15

5. Audit: How does your laboratory address audits and 

assessments?

Interviewees’ laboratories rarely used audits to improve quality. This was true for both 

accredited and unaccredited laboratories. The strong negative experience with auditors is 

unfortunate and suggests that improvements are needed in how audits are conducted. Overall, 

Interviewees talked much less about their experience with audits than with other techniques. 

The most beneficial experiences were with fairly specific areas like evidence management 

rather than the overall quality process. Interviewees answered based on different types of 

audits, both external (assessments) and internal audits.

Responses generally fell into 3 themes:

⚫ Theme 1: Most interviewees saw little value in audits and thought they focused on 

finding something just to find something or had findings not relevant to digital forensics.

 Interviewee 13 stated that if the approach of the auditors was to coach and help, 

rather than promote competition and try to “ding” the laboratory, there would be 

far more value to audits.

 Interviewee 1 described having to explain something at every audit because their 

laboratory does not have a digital quality person. Policy changes usually result 

after each audit, but they do not affect the quality of the work product.

⚫ Theme 2: Some interviewees found that the audits identified processes that should be 

improved.

 Interviewee 10 described that under accreditation, the QA team did audits 

frequently and that it was overkill. The process in other disciplines did not apply 

well to digital forensics, but quality assurance audits applied to everything, which 

was also too much. Outside of accreditation, their audits were more directed to 

specific processes with different timeframes, and this was more helpful. The 

focus was on good forensic practices and personnel. They removed some 

procedures and added others to streamline the process.

 Interviewee 14 used spot checks which helped identify chain of custody issues

for processing multiple phones.

⚫ Theme 3: One interviewee found the audits useful for internal morale and for marketing 

to customers.

 Interviewee 13 stated that audits helped with both the laboratory’s reputation and 

with laboratory personnel’s esprit de corps and pride.

Summary: Auditing, as practiced, has not been seen as an effective method to improve quality. 

The lack of understanding of digital forensics processes has led to a focus on less relevant 

processes and created a “nitpicking” culture. A few interviewees showed that audits can be 

beneficial if they are practiced with an understanding of digital forensics and a focus on 

improvement.
16

6. Testimony Monitoring: Does your laboratory conduct testimony 

monitoring? What is the value in terms of quality for this element? 

Most laboratories surveyed did not monitor the testimony of digital forensic examiners as a 

quality management practice. Interviewees often discussed general training for how to testify or 

other feedback mechanisms. Of those laboratories that actually did go to court to monitor 

testimony, such monitoring was generally limited to testimony from new examiners.

Responses generally fell into 4 themes:

⚫ Theme 1: No testimony monitoring was done. Most interviewees did not use testimony 

monitoring.

⚫ Theme 2: Testimony monitoring was useful for training, especially of new staff.

 Interviewee 10 said they did testimony monitoring as part of their accreditation

because there was an annual requirement. When not a part of an accredited 

laboratory, the interviewee would often go to watch testimony for personal 

interest, not for the purpose of monitoring. Instead, they would do some 

testimony reviews primarily for newer examiners.

 Interviewee 6 stated that testimony monitoring helped examiners learn strategies

and tactics for use in court, such as improving reporting and communication, and 

how to avoid open-ended questions or situations where people feel compelled to 

expand further on answers

 Interviewee 14 noted that testimony monitoring could be useful, although it was 

not done in their laboratory due to lack of resources to develop and implement a 

policy for it. They would like to implement a “buddy system” to support testimony, 

provide practice and feedback to examiners, and ultimately result in better 

testimony.

⚫ Theme 3: Testimony monitoring was not useful and done only to meet an external 

requirement.

 Interviewee 15 described having to do this because it is required as part of their 

accreditation. Interviewee described only finding minor things, nothing critical. 

This interviewee thought training was far more valuable.

⚫ Theme 4: Laboratories used alternative methods for testimony monitoring. Several 

interviewees described using feedback from the court to learn about the testimony of 

digital examiners.

 Interviewee 17 described using an “officer of the court” feedback form for both 

defense attorneys and prosecutors to provide feedback, noting that the defense 

attorneys were the most helpful. The laboratory was very happy with this 

approach.

 Interviewee 12 described using several techniques, such as getting feedback 

from prosecutors, examining transcripts, and having senior staff sit in. The 

interviewee couldn’t recall any times things went wrong.
17

Summary: The practice of laboratory staff performing testimony monitoring does not appear to 

be widely used or effective. The alternate technique of getting feedback from customers 

(described by Interviewee 17) suggests that more effective practices could be used. Initial 

training on how to testify was seen as valuable.

7. Personnel: What are key elements of competent forensic 

laboratory personnel in your laboratory? 

Having competent personnel was regarded as the single most important quality element, 

eclipsing all the other elements combined. The overarching theme in personnel is the 

inconsistency of training, education, and required knowledge/skills; the skill of the person 

determines a successful examiner overall. Laboratory programs were quite varied in assessing 

personnel. In two extreme examples, one laboratory took two weeks to qualify someone to do 

independent casework and another took three years. Many laboratories lacked formal training 

programs, although most laboratories had some training for DE and laboratory procedures. 

There was significant variation even in the more formal programs. The variation suggests that 

laboratories are unsure what the best methods are to address this critical quality element. 

Responses generally fell into three themes:

⚫ Theme 1: Personnel requirements were critically important.

 Interviewee 13 stated that having the right personnel was more important than 

SOPs and rigid workflows, that one needed to be able to trust staff to do the job. 

Interviewee 13 thought that certifications and competency are not emphasized 

enough in the accreditation process.

⚫ Theme 2: Various personnel requirements are useful but not sufficient. Management or 

senior staff assessment of capabilities is needed to see if staff are actually good 

examiners. There were concerns about training and certifications, primarily cost and 

perceived value.

 Interviewee 3 wanted the emphasis to be on both problem solving and applied 

skills. Interviewee 3 felt that applied training and certifications are needed but felt 

the more elusive problem-solving skill was hard to assess and didn’t correlate 

with a college degree.

 Interviewee 9 considered training to be the biggest problem in the industry 

because it is very expensive so only people already in a DE examiner position 

can afford the training. This makes it hard for newcomers to break in. It would be 

good for the industry if we had people with different backgrounds to bring in 

diversity of thought. Interviewee 9 wanted better proficiency assessments based 

on key work tasks such as being able to find artifacts of value and write a good 

report, rather than procedural issues.

 Interviewee 13 was split on the value of certification - there is value when it is 

done right, but there needs to be a recognized standard for training to 

competency because the requirements for certifications are becoming too 

demanding and cumbersome. 
18

⚫ Theme 3: One laboratory also used personnel techniques for non-quality reasons such 

as supporting testimony or marketing. 

 Interviewee 10 kept up with their International Association of Computer 

Investigative Specialists (IACIS) and vendor-based certifications and training so 

when they testified, they could say they completed the courses. 

 Interviewee 12 encourages vendor training and certifications but was worried that 

certification “does nothing but increase perception that you know something." 

Summary: The large variation seen among interviewees speaks to the need for better programs 

to adequately address the key factors for understanding the professional development of 

laboratory examiners. It also speaks to a need for minimum competency standards across the 

field. Given the breadth of tasks in DE, this may actually need to be multiple standards based on 

the type of work being done. Note that this was a similar theme in the customer interviews.

8. Quality Manager: What is the value of a Quality Manager in

your laboratory and what is this role’s effect on the quality of your 

laboratory’s work? 

Quality managers and quality management systems were viewed as useful in theory by most 

responders, though they generally got scathing (i.e., quite negative) reviews from the digital 

evidence laboratory staff based on specific occurrences in practice. Interviewees spoke highly 

of quality management systems in general, but negatively about quality managers and how 

quality management systems were implemented in their laboratories.

The key message was that quality systems and managers optimized for digital forensic 

laboratories were successful at improving quality but that most quality systems and managers 

were not optimized for digital forensic laboratories. There was a big concern that quality 

managers were asking laboratories to perform tasks that are not relevant to digital forensics 

which took people away from case work. These interviewees often felt that quality management 

systems could be valuable, but not as practiced in their laboratory. There was also a significant 

call for quality managers who knew or at least understood digital forensics. When laboratories 

had quality managers who took the time to learn digital, they felt the quality manager was more 

useful to them. Several interviewees did not have a full-time quality manager nor thought it 

would be useful for their laboratory.

Responses generally fell into four themes:

⚫ Theme 1: Many laboratories did not have a quality manager or formal quality system.

⚫ Theme 2: Quality management (QM) did not increase quality and, in many cases, just 

made additional work for digital forensic examiners.

 Interviewee 10 said that the previous QM did not understand the specific needs 

of digital forensics and tried to enforce rules that did not apply. After the 
19

laboratory stopped having a QM “it was a breath of fresh air to get out from under 

them.”

 Interviewee 1 also had a QM that didn’t understand digital forensics. Interviewee

1 felt that the QM was “just finding stuff to do.” This resulted in a lot of time spent 

explaining digital, such as the difference between validating machines and

software.

 Interviewee 5 stated that it could be good to have a QM if they “knew what they 

were doing.”

⚫ Theme 3: Minimal improvement but with significant overhead.

 Interviewee 9 stated that QM helps with consistency and general efficiency, 

centralizing some tasks and allowing the examiner to focus on case work. The 

QM kept SOPs up to date and performed some spot reviews, but also created 

templates and took charge of various tasks. Interviewee 9 felt this was an 

important function for large laboratories but not an efficient use of personnel for 

medium and small laboratories.

 Interviewee 13 reported that having a QM working with and guiding the technical 

manager helped improve quality, but it made things slower and harder with a lot 

more bureaucracy and a lot more paperwork.

⚫ Theme 4: Laboratories with quality managers who optimized quality systems for DE had 

a more positive experience with quality management systems.

 Interviewee 15 described their quality manager as helping the laboratory “lift its 

head out of the sand.” Improving efficiency and lowering the backlog has been 

the key to better quality work.

Summary: Having a quality manager or quality management system that understands digital 

forensics is essential to success but was rarely found in our survey. The current approach to 

quality management is suboptimal for digital forensics. Many of the frustrations came from 

quality managers not understanding digital forensics as a discipline. Complaints were centered 

around the administrators of the quality management system, not the quality system itself. 

Higher level look at quality processes and accreditation

The last set of questions asked interviewees to list the most and least effective elements for 

quality improvement in digital forensic laboratories. They were also asked to address quality 

improvements outside of discussing the specific quality elements, including accreditation. 

Overall, interviewees felt that having competent personnel, technical reviews, and chain of 

evidence controls were the most important. 

The least valued elements were testimony monitoring, testing/validation, internal audits, and the 

overhead associated with accreditation. While almost every interviewee was aware of 

accreditation, only about a third of the interviewees were in accredited laboratories. In general, 

accreditation was not viewed favorably across both accredited and non-accredited laboratories. 

While people liked that accreditation provided a minimum bar, it was associated with a high 

overhead of resources and time for all types of laboratories. Many stated that laboratory 
20

accreditation standards ISO 17025 and 17020 were not good fits for digital forensics, were too 

resource intensive, and led to delays in getting reports to customers. In accredited laboratories 

or those with more formal systems, quality management was more likely to be seen as punitive. 

The general sense was that quality systems were just used for tracking and not for quality 

improvement. 

⚫ Interviewee 1 stated that early efforts to improve quality were helpful at fixing problems 

such as a lack of documentation. But after fixing the basics, the improvement has 

stopped; quality “improvements” are reducing clarity to customers and "It's been a lot of 

BS." 

⚫ Interviewee 10 focused on the need for a commitment of resources to invest in people 

both financially and institutionally, such as a better career track and mental health 

support. Interviewee 10 summarized their remarks by saying that personnel is the most 

important quality system. They see the formal system as wasteful and stifling of staff. 

⚫ Interviewee 17 recounted their experience in a very large laboratory with a strong Quality 

Manager who is looking at the big picture of the laboratory and what it is trying to do. 

While there was a heavy focus on looking good, it was also about delivering a quality 

product.

⚫ Interviewee 8 emphasized that an ISO standard specific to digital evidence examinations 

is important to this field. Ideally, it would be based on input from many agencies and 

laboratories to determine what practices would scale to labs of all sizes, then implement 

them in customized form for each laboratory. 

Summary: The perception of accreditation and more formal quality systems as punitive is a 

significant problem. Many digital forensics quality systems, as currently practiced, are failing to 

promote a strong quality culture.

Summary of Laboratory Interviews

Laboratory management we interviewed were all committed to quality, but interviewees 

described how the techniques they were using were often making things worse, mostly by 

creating inefficiencies and discouraging people with paperwork and low impact procedures. The 

interviews showed that there can be significant value in quality management systems, but not 

as generally implemented today. Critically, very few problems were discovered in the accuracy 

of laboratory reports. Many problems were discovered with the clarity and timeliness of the 

reports. Some of the key trends that were identified are:

⚫ The first trend that emerged from the entirety of data collected was that interviewees felt 

that the quality practices being performed by accredited laboratories and some nonaccredited laboratories were not contributing to a quality work product; in fact, some 

quality practices were hindering the progress of digital forensic work. In several cases, 

laboratories described an evolution of discarding quality practices, such as testimony 

monitoring, that were consuming resources but not adding significant value. Changing 

quality management practices did not seem to result in changes to the accuracy of 
21

laboratory output. In some instances, the punitive approach of the quality management 

system caused practitioners to bypass the structured system. 

⚫ Another significant trend involved the key element of personnel. The interviewees 

focused on individual examiners as the key to the production of timely and accurate 

work. However, interviewees did not agree on how to hire, train, or assess examiners 

and generally criticized the existing processes as either too expensive, too time 

consuming, or not meeting their needs.

⚫ Peer/technical review and the exchange of ideas for how to address various issues in 

digital forensics was also seen as a key quality element.

⚫ There was significant value in documentation for chain of evidence and related 

processing. This was true for both the laboratories and the customers.

⚫ The other techniques tended to be less valued. In many laboratories, these had been 

implemented because they were part of the laboratory’s existing quality framework for 

other disciplines as opposed to being optimized for improving digital forensics. The 

finding that they had less value in a digital forensic context may be based on how they 

had been implemented, not whether the technique could be used effectively. The task 

group did see the potential for these elements in a few of the interviews. 

Results: Customer Interviews 

The customer interviews did not focus on specific quality techniques but investigated how 

customers interacted with digital forensics laboratories and their perception of the quality.

All but one of the downstream recipients of digital forensic laboratory analysis (“customers”) the 

task group spoke with were lawyers, including prosecutors, defense attorneys and judges. One 

customer was a police chief. Many of the defense attorneys had previously worked as 

prosecutors. Many customers had extensive experience with digital evidence including having 

taken classes and were quite familiar with digital forensic techniques. Most of the laboratories

the customers had worked with were law enforcement laboratories.

The survey included questions covering what customers’ value in a laboratory, the main 

problems they have experienced, and what they want from digital evidence laboratories moving 

forward.

What do customers value in a laboratory?

When customers had a choice of laboratories, they used two criteria to select which laboratory 

they would work with: timeliness and perceived quality. Timeliness was the single biggest 

concern. While lawyers said they preferred a laboratory to be accredited and examiners to be 

certified, this was because they could use it at court, not because they saw an association 

between accreditation or certifications and quality. In some cases, the decision for which 

laboratory to use was based on whether the laboratory could handle a certain type of evidence. 
22

⚫ Personnel/Certifications/training. Lawyers had a very strong sense for which examiners 

did better work, and the lawyers were motivated to get those examiners for difficult 

cases. Only a few of the lawyers – those with more technical experience – could express 

how they knew whether an examiner was highly skilled. This led them to rely on people 

they had worked with before. Lawyers also preferred examiners who could testify 

persuasively.

 Interviewee 31 emphasized the importance of the individual examiner, not the 

laboratory and of the work, not the accreditation. They know great people in 

“shitty labs.”

 Interviewee 22 said selection of examiners depends on the case, what examiners 

are available, and who is trusted. Trust is based on past case experience, quality 

of the work product, and their experience talking to specific examiners.

 Interviewee 18 selected examiners when they needed certain resources or an 

answer to a specific question. Interviewee 18 listed several factors they consider 

when selecting an examiner: 

■ Cost,

■ Whether the person is highly competent or takes liberties when 

researching or speaking,

■ Works within a certified lab,

■ Uses peer-review, and 

■ Can explain digital forensic reports and conclusions clearly to a jury.

⚫ Accreditation: Customers generally only used accreditation to bolster a laboratory’s

credibility in court. Accreditation was generally not regarded as increasing the quality of 

the work product. Accredited laboratories were recognized as having more 

documentation which did not add to the value of the work and caused perceived delays 

in timeliness. Other types of QMSs were recognized as being valuable.

 Interviewee 26 thought that accreditation doesn’t matter and that their models are 

based on things not relevant to digital. Interviewee 26 has worked with both an 

accredited state laboratory and unaccredited local labs.

 Interviewee 18 thought that accreditation was not as necessary as having a QMS 

and following standards in terms of responding to cross-examinations. The 

Interviewee described a case where testing to verify one key aspect of the case 

was considered "quality management" even though there was a lack of formal 

policy or accreditation.

 Interviewee 30 thought that accredited digital forensics labs have more standard 

documentation and paperwork which may or may not be helpful. Interviewee 

wants a system that includes better/more detailed notes and audit logs. 

Interviewee was concerned that QMSs were hit or miss without an accreditation 

and was concerned about the lack of a standard certification for digital.

⚫ QMS: The majority of interviewees are in favor of the laboratory having a quality system 

or elements of a quality system (e.g., SOPs, training program) but not necessarily 

interested in accreditation. Accreditation was not seen as a necessity as long as quality 

elements are in place. 
23

 Interviewee 23 stated that it was important that labs have a QMS, but not 

important if the lab is accredited. The Interviewee was worried about 

accreditation having a negative impact – examiners are more concerned about 

working inside accreditation rules than doing good work.

 Interviewee 24 absolutely cared that labs have a QMS. They felt this was 

important in case evidence is challenged. Interviewee 24 described a QMS as 

having SOPs so they can say they are doing things accurately and credibly. 

 Interviewee 21 does not care whether the lab or the examiner is accredited. They 

want the examiner to have training and if it comes with a certification that’s even 

better. Interviewee 21 wanted the training curriculum to be accredited but was 

concerned about the impact it might have on the practitioners who aren’t 

certified.

What are the main problems they have experienced?

Timeliness and the clarity or understandability were big problems.

⚫ The discussion of timeliness was straightforward: evidence took too long to process due 

to backlogs. The concerns with clarity primarily centered on reports that lacked analysis. 

Reports were often solely the output of a forensic tool or tools, and customers were left 

to figure out what the data meant. Note that clarity was not one of the original attributes 

of quality that the task group considered. However, this is clearly a significant problem 

as it impacts the ability of lawyers, as officers of the court, to communicate clearly and 

truthfully the results of the forensic analysis. 

 Interviewee 26 reported that timeliness is a significant problem and slows down 

investigations considerably. Interviewee 26 also reported that clarity was a huge 

problem because most reports are “completely incomprehensible.” Most of the 

reports are the ones that are automatically generated from the tools. That report 

is sent to the prosecutor, and they are forced to figure them out themselves. This 

has led to significant problems such as prosecutors thinking that there is 

evidence to prosecute a Child Sexual Abuse Material (CSAM) case, but the 

images were actually only in the cache. 

 Interviewee 23 also reported accuracy issues when a search term or search 

strategy ended up glossing over or leaving out important information. Interviewee 

23 was clear that they had never received a report that they thought had false 

information. 

 Interviewee 22 reported major problems with understandability and timeliness. 

Timeliness was a factor in every case with only homicide getting priority. 

Interviewee 22 also stated that most of the end products are unsatisfactory 

consisting of just the evidence (the output from a mobile extraction) without even 

a report.

 Interviewee 27 stated that timeliness is a real problem especially with CSAM 

cases. Interviewee 27 also stated that the labs stopped doing attribution work, so 

the reports were less useful. Interviewee 27 attributed this to time pressures.
24

⚫ There were only a few complaints where customers were able to identify inaccurate or 

misleading reports, and all of them were about cell site analysis (locating mobile phones 

based on which towers they connected to) and laboratories overstating the precision of 

the technique. Other complaints focused on quality management in general. 

 Interviewee 31 said that when courts don’t understand digital evidence, they tend 

to give experts more credibility than is warranted. The interviewee emphasized 

that this leads to testimony given with no basis in fact or science (e.g., the range 

of a cell tower). Interviewee 31 also described being able to give expert 

testimony on stingray technology without ever having seen or used it, suggesting 

that judicial “gatekeeping” of technical experts was lax.

 Interviewee 22 described experiences with problems like lack of search warrant 

procedures, adherence to warrant requirements, unqualified people being 

appointed to positions of supervision level in DE, chain of custody problems, 

logging evidence, etc. Today, DE is valued more than any other forensic science, 

but is not treated with the same rigor as other forensic disciplines in these 

instances.

What do customers want moving forward?

The majority of the requests centered around both consistency between practitioners and work 

products, as well as better baseline for testimony along with identification of the sub-discipline(s) 

the practitioner is an expert in.

⚫ Interviewee 18 thought a centralized resource where different types of qualified forensic 

opportunities/labs/professionals were broken out would be very helpful for prosecutors. It 

would be a resource for prosecutors to use to do a secondary review of evidence to 

determine whether it was done right the first time. 

⚫ Interviewee 30 asked how to standardize who is qualified to testify about what? They 

argued that a person cannot be an expert in all fields of digital evidence, so what is the 

base level for testimony? What qualifies you: degrees, certs, or something else? 

Summary of Customer Interviews

Issues with timeliness and clarity of many laboratory reports are driving customers, including 

police departments, prosecutors and defense attorneys to developing in-house capability or are 

using lawyers and investigators for jobs that should be done by a digital forensics expert. (Note 

that the NIST Foundation Study estimated that there are at least 11,000 digital forensics 

laboratories in the US.2) Many interviewees expressed concern that this could result in 

misinterpreting or missing key artifacts that are pertinent to a case. The customers are eager for 

laboratories to have quality systems that produce more useful, timely, and accurate reports. The 

interviewees discussed many types of QMSs, but rejected that accreditation was beneficial and 

had varying opinions about which elements of a QMS were important to them.

2 Lyle et. al. Digital Investigation Techniques: A NIST Scientific Foundation Review, NISTIR 8354-DRAFT, 

National Institute of Standards & Technology, 2022. https://doi.org/10.6028/NIST.IR.8354-draf
25

⚫ Interviewee 22 stated that they are very concerned that so much DE is done by lawyers 

and detectives rather than DE professionals working with lawyers and detectives. This is 

starting to cause issues and the interviewee suspects that this will increase. The 

interviewee gave several examples of customers doing their own analysis poorly; the 

worst was an analysis (done outside the lab) which missed a living victim who should 

have been located. 

⚫ Interviewee 20 believes that DE needs to be in-house because it is so difficult to use 

external labs based on both timeliness concerns and that labs won’t even take lower 

priority cases. The interviewee stated that “once you have it you don’t want to lose it.”

⚫ Interviewee 25 stated that having a QMS would make DE more impactful and had ideas 

about how to optimize a QMS for the volume and impact that DE can make on a case. 

The focus was on streamlining some of the operations that are overly constrained based 

on a division of labor between the simpler and more complex tasks.

⚫ Interviewee 26 stated that there should be a QMS, but accreditation doesn’t matter, that 

the model is based on things not relevant to DE. Interviewee sees no difference in 

quality between the accredited and nonaccredited labs they work with.

Findings and Recommendations

Based on the responses, the task group recommends that the digital forensics community 

initiate and continue efforts to improve quality management practices. The current emphasis on 

accreditation under ISO 17025, ISO 17020, and formal QMSs is not having the desired 

outcome. One of the problems with the current implementations of QMSs is a lack of 

understanding of the tasks, methods, and sources of errors in digital forensic processes, 

particularly among Laboratory Management and Quality Managers. In digital forensics 

laboratories, a major focus of quality systems has not been on improving quality; it has, instead, 

focused only on meeting 17025 requirements with little regard for enhancing the quality of the 

forensic services in the digital laboratory. The spirit of a quality system is lost in the dogmatic 

adherence to the quality system or the ISO standard rather than examining practices for their 

enhancement of quality management goals. To be successful, a QMS must have accuracy, 

understandability, and timeliness as goals rather than a perception that the laboratory has 

quality. Successful quality systems focused on making things better rather than trying to create 

paperwork. The task group saw this in very few laboratories. Another problem with current QMS 

implementations is that there has been a punitive culture rather than a quality culture. No 

amount of process will improve quality without strong management support. 

Findings

In summary, the major findings are:
26

1. The implementation of quality management systems, as practiced, rarely focuses on 

trying to improve the quality (e.g., products are accurate, complete, timely, and 

understandable) of the product of DE laboratories. Laboratory management tends to see 

quality as demonstrating that the laboratory is “good” rather than a means to improve

reports, testimony, and other products.

2. Many people did not understand what the quality elements were meant to achieve or 

how and why they should be integrated into an effective quality management process.

3. Customers of DE laboratories are unconcerned about quality processes but do have 

beliefs about which laboratories or examiners are better than others.

4. Everyone interviewed was very concerned with producing or receiving accurate, useful,

and timely reports and perceived overall quality (accuracy) as high. Most of the problems 

discussed were not believed to affect the outcome of an investigation/court proceeding. 

There were a few exceptions including overselling the results of cell site analysis, failure 

to provide timely information during a crime in progress, and potential to mix up media 

from different cases.

5. Timeliness was an endemic problem to the extent that many customers started their own 

laboratories or were doing forensic analysis by themselves and only using the laboratory

for extracting data from mobile phones. Very few interviewees thought of quality as 

supporting timeliness.

6. People strongly valued peer review even though no one ever cited it for catching 

anything important. However, interviewees described other benefits in terms of 

information sharing among examiners and satisfaction of having one’s work be favorably 

reviewed.

7. Everyone saw the importance of highly competent examiners. This was viewed by 

customers as the key element of quality. Customers never cited laboratory -wide quality 

management as being important to quality.

8. The value people placed on other elements was superficial; they were “good” only 

because everyone says they are good to have. For example, no one spoke persuasively 

about the benefits of SOPs except for evidence management and customer interactions 

even if they rated them as critical. Tool validation was another example of an element 

everyone performed but few valued. People didn’t seem to understand how the various 

quality elements actually related to quality.

There is value in quality management systems, but not as generally implemented today. 

Peer/technical review and the exchange of ideas for how to address various issues in digital 

forensics was the single most valuable technique according to the laboratory personnel. There 

was significant value in documentation for chain of evidence and related processing. This was 

true for both the laboratories and the customers. The need for highly competent personnel was 

universal but most interviewees felt that the methods for recruiting, training, and assessing them 

were either poor, too expensive, or too time intensive. Most of the other techniques were less 

valued. In many laboratories, these had been implemented because they were in a quality 

framework as opposed to being optimized for improving digital forensics. The finding that they 

were found to have less value may be based on how they had been implemented, not whether 

the technique could be used effectively.
27

Recommendations

Based on these findings, the task group recommends refocusing the quality system for DE on 

improving and supporting the quality of the product rather than forcing “one-size-fits-all” quality 

techniques from generic quality programs onto DE processes. Although the current system is 

suboptimal, the team did interview a few labs where quality was thought about more holistically 

and people actively tailored quality elements to fit the DE process. This should be pursued at 

the community level. A DE-oriented quality management system should be based on an 

assessment of which elements are most effective and how to implement it to achieve greater 

quality. This assessment should be based on this study and include the needs of the many 

stakeholders including labs and customers. The metrics for what is quality must include 

understandability as well as having analysis work be performed by competent examiners rather 

than by the customers. Specific recommendations for designing a DE-oriented quality 

management system are:

1. Develop material to fundamentally redefine a quality management system that is 

optimized for digital forensics. This should be a community-based effort.

a. This could be done incrementally. As a first step, identify and reduce the use of 

less effective techniques. 

b. Develop guidelines to optimize the techniques for digital forensics.

c. Develop a digital forensic specific quality management system.

2. Develop material to educate laboratory management, quality managers, and 

accreditation bodies about quality management for digital forensics.

3. Develop centralized resources to aid labs in addressing the challenges they face, 

including:

a. Personnel recruitment with a focus on critical thinking and technical acumen

b. Personnel training and education

c. Personnel assessment (e.g., competency, proficiency, readiness for independent 

casework.)

d. Tool testing

e. Sharing of resources, tools, tips, and techniques

4. By optimizing the quality system for improving timeliness, accuracy, and 

understandability over meeting quality requirements, this may help with resourcing in 

labs since they will have more time for casework and improve morale. Other issues with 

timeliness are beyond the scope of this document.
28

Appendix A: Survey Questions (Laboratory 

Personnel)

1. Tell us about you and your lab: 

Type: Federal State Regional Local Commercial Private Other (specify):

Role: Director Deputy, Sr. Analyst Other (specify): 

Career Summary: 

a. How big is your lab? Size: small (<5) medium (5-15) large (>15) # of 

examiners: 

b. Who is/are your customers? Customers: LE, judicial/courts, state, federal, 

municipal 

c. What types of work does your lab do? 

d. Approximately, how many cases or items to you handle per week (or 

month)? Turnaround times? Backlog? 

e. How long have you worked there? 

# years @ current position: 

# Years in digital: 

2. Our next questions are about how you manage quality in your lab. We define quality 

in terms of timely and accurate outputs that assist with investigatory leads and court 

proceedings. We have a list of 9 common Quality practices. Can you describe the 

importance of each: waste of money, time or energy, useful, can’t imagine not having 

this (critical), or other. 

a. What documentation do you have and how do you use them? How important are the 

different types of documentation? 

Importance: waste, useful, critical, other (specify): 

How is it used: 

Made life better or worse: 

Result in better products (more accurate, clearer, faster): 

Documentation type: SOPs guidance docs nothing other (specify): 

Level of detail (high level, all steps, some steps, loose framework) 

Deviations: 

b. Equipment and software validation 

Importance: waste, useful, critical, other (specify): 
29

How is it used: 

Made life better or worse: 

Result in better products (more accurate, clearer, faster): 

c. Management reviews 

Importance: waste, useful, critical, other (specify): 

How is it used: 

Made life better or worse: 

Result in better products (more accurate, clearer, faster): 

How often (yearly, monthly, other): 

Focus of reviews: 

d. Peer/technical reviews 

Importance: waste, useful, critical, other (specify): 

How is it used: 

Made life better or worse: 

Result in better products (more accurate, clearer, faster): 

Release results: 

e. Internal and/or external audits 

Importance: waste, useful, critical, other (specify): 

How is it used: 

Made life better or worse: 

Result in better products (more accurate, clearer, faster): 

f. Testimony monitoring 

Importance: waste, useful, critical, other (specify): 

How is it used: 

Made life better or worse: 

Result in better products (more accurate, clearer, faster): 

g. People 

a. Competency testing

Process for determining that someone is ready to do independent case 

work/mentorship. 

How long does the training process take? 

What are its key elements? 

b. Education: 

Degree required: 
30

c. Examiner training: lab-based, vendor, other 

d. Certifications 

e. Follow on if needed: Discuss supervision of examiners and oversight of 

employees and products? What happens when people don’t pass tests? 

Importance: waste, useful, critical, other (specify): 

How is it used: 

Made life better or worse: 

Result in better products (more accurate, clearer, faster): 

h. Quality Manager 

Importance: waste, useful, critical, other (specify): 

How is it used: 

Made life better or worse: 

Result in better products (more accurate, clearer, faster): 

3. Can you tell us any times that your lab’s quality process led to a difference in a 

report for a) clarity b) accuracy or c) completeness? 

a. Can you tell us about any errors or mistakes that you caught and how 

you changed your process to try and avoid them in the future? 

Process changes: training, hiring, accreditation, certification, 

reviews, audits 

b. Can you summarize what you think are your lab’s most important 

quality management processes and which you think are the least 

effective? 

Most important: 

Least effective: 

4. Accreditation or Other Formal Quality Systems 

a. Are you aware of accreditation or other formal quality systems? What do you 

think are the strengths and weaknesses of formal quality systems? What 

do you think is a good idea for formal systems for DE? 

Meta Questions 

5. Are there any questions you want to answer that we haven’t asked you? 

6. Can you recommend other people you think we should interview?
31

Appendix B: Survey Questions (Customers)

Background questions to understand the customer and their experience with DE 

1. Tell us about your role in the judicial/criminal justice/other system and your 

experience. How long have you been doing this? (Do NOT record name or 

name of organization. Categorize by federal/state/local and size) 

2. What kind of cases have you been involved with? 

3. What kind of forensics and forensics labs do you work with? 

4. What is your experience if any with digital evidence? We think of digital forensics as 

a variety of areas such as cell phone analysis (finding photos and text message), 

looking at internet search history and complex analyses such as fraud, cell site 

analysis, etc. 

5. Describe how you work with your Digital Forensics examiners and Digital Forensics 

labs? 

6. Have you had any training/workshops related to DE? 

7. Have you ever had Digital Evidence go to trial or used Digital Forensics examiners 

to testify? 

a. Follow on: have them describe this 

b. Did you use the examiners as fact witnesses or experts? 

8. Do you have a choice what lab to use? Have you made choices about who to 

hire/select to do Digital Forensics work? If so, why do you pick one lab 

over another? 

a. Back pocket question: Do you have a choice what person to use? 

Quality Questions: 

The next set of questions is about Quality. We think of quality as 3 things: accuracy, 

understandability, and timeliness. 

9. What problems have you had with Digital Forensics work products? Keep in mind all 

3 attributes of quality. 

a. If possible, get a description that is rich enough that we understand the 

problem and potential root causes and possible corrective action. 

b. If they go deep enough: Did this result in corrective actions at the lab? 

10.Have you ever had problems getting Digital Evidence accepted? 
32

a. If so, please tell us about that. 

11.Have you ever had concerns about how Digital Evidence-related testimony in 

court? 

a. If so, please tell us about it. 

12.There are lots of types of Digital Forensics Labs and many Police Departments and 

other groups do some Digital Forensics in house. Have you noticed issues 

dealing with these different types of Digital Forensics processes? 

13.Do you care that a lab you use has a “quality management system” or is accredited? 

When we think of a QMS, we are referring to management practices such as 

standard operating procedures, technical reviews, proficiency testing, and 3rd 

party reviews (often as part of a formal accreditation system) 

a. Follow on/skip: Would you use a lab without a documented QMS? Do you 

handle/present their work differently if they have a QMS? 

b. Follow on/skip: Would you use an unaccredited lab for DNA or toxicology? If so, why 

do you think Digital Forensics is different? 

14.If they haven’t talked enough: Describe your best and worst experiences with DE 

related to your cases. 

15.Given a choice, what would be your priorities in determining which lab to send 

evidence? 

Meta Questions 

16. Are there any questions you want to answer that we haven’t asked 

you? 

17. Can you recommend other people you think we should interview? 
33

Appendix C: Quality Management Systems

The task group developed a list of common quality management system elements as part of the 

interview protocol. The list is available in Appendix A. This list was based on multiple documents 

that address quality management systems. The following references can provide additional 

information about each quality element and about quality systems in general. 

1. Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison 

Methods (2016)

https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_fo

rensic_science_report_final.pdf

2. ANAB AR 3125, ISO/IEC 17025:2017 Forensic Testing and Calibration Laboratories 

Accreditation Requirements 

https://anab.ansi.org/2018-iso-iec-17025-forensic-accreditation-documents-0

3. International Organization for Standardization. (2017). General Requirements for the 

Competence of Testing and Calibration Laboratories (ISO/IEC Standard No. 17025)

4. International Organization for Standardization. (2012). General Criteria for the 

Operation of Various Types of Bodies Performing Inspection (ISO/IEC Standard No. 

17020)

5. Qualitative Assessment Review Guidelines for Investigative Operations of Federal 

Offices of Inspector General

https://www.ignet.gov/sites/default/files/files/QAR%20Guide%20Investigations%20July

%202017.pdf

6. Strengthening Forensic Science in the United States: A Path Forward (2009)

https://www.ojp.gov/pdffiles1/nij/grants/228091.pdf

7. SWGDE Myths and Facts about Accreditation for Digital and Multimedia Evidence Labs

https://drive.google.com/file/d/1E0nu8_WWBYpXLykOK3BQX329OoFRqdpc/view

8. SWGDE Establishing a Quality Management System for a Digital and Multimedia 

Organization under ISO-IEC 17025 or 17020

https://drive.google.com/file/d/14LZ2-uuoQqYzKokrgNj_ntW5aeFjKd3z/view

9. SWGDE Minimum Requirements for Testing Tools used in Digital and Multimedia 

Forensics

https://drive.google.com/file/d/1QePvaHM2yz4rwL0iiDHm-XVBqW5vAJJk/view
34

10. SWGDE Requirements for Report Writing in Digital and Multimedia Forensics

https://drive.google.com/file/d/1_O1WxWa1FTkOqDPJ3n1aul5VXkGlTAZh/view