Framework for Registration, 

Classification and Evaluation of 

errors in the Forensic DNA 

Typing Process

Ate Kloosterman

NFI Departments HBS & WISK 

a.kloosterman@nfi.minvenj.nl 

NIST WEBINAR May 28 2014 
Transparency and accountability in forensics 

• Transparency is a key characteristic of 

forensic institutes and their professionals: 

they are open in the clear disclosure of 

processes, guidelines but also failures. 

• Forensic institutes like the NFI are 

answerable for their actions and there is 

redress when the investigation does not 

meet the quality standards.

2 
Daubert validity factors for scientific evidence 

• Whether the technique upon which the evidence 

rests has been scientifically tested. 

• Whether the technique has been subjected to 

peer review or publication. 

• General acceptance in the scientific community. 

• Known or potential rate of error. 

The Daubert standard does not provide a specific 

definition for rate of error. 

3 
The Quality Management System 

• Achieving a 99,9% level of quality 

means accepting a 0,1% error rate. 

• Some failures can have great consequences. 

4 
Definition of the error rate in forensic science 

• The National Research Council has defined error 

rates as misidentifications "proportions of 

cases in which the analysis led to a false 

conclusion (as the percent of incorrectly identified 

cases among all those analysed)”.

5 
Error rate in forensic science 

The error rate includes both type 1 (false positive) errors and 

type 2 (false negative) errors. 

• A type 1 error in forensic DNA analysis is the 

event where the DNA profile of the reference 

sample from a suspect is incorrectly concluded to 

match with the crime sample; 

• A type 2 error is the event of wrongly reporting a 

non DNA match between two samples when in 

truth there is one. 

6 
Error rate in forensic science 

• To prevent false convictions facts on the rate of type I errors is 

of particular relevance to the criminal justice system. 

• Lack of knowledge of the true rate of error creates an important 

element of uncertainty about the value of DNA evidence1. 

1. William C. Thompson, Franco Taroni and Colin G. G. Aitken (2003); How the Probability of a False Positive Affects the Value of DNA 

Evidence J Forensic Sci 48 1-8

7 
Type 1 and type 2 errors at the NFI 2008-2012 

Definition: 

• Wrongful DNA match (or non match) reported to the authorities 

(public prosecutor) 

• Misidentification notified by internal control or external notification 

by police or prosecution service 

• Characteristic: all defenses have failed

2008 2009 2010 2011 2012 

Type 1 error 2 1 0 0 0 

Type 2 error 4 3 1 2 4 

Type 1 and type 2 error 0 2 0 0 2 

Total (2008-12 = 21) 6 6 1 2 6 

8 
Type 1 and Type 2 errors: breakdown by cause 

Errors can occur at various stages in the DNA typing process: 

• Pre-analytical (sample collection, labeling and storage). 

• Analytical (occurring within the laboratory and profile 

analysis). 

• Post-analytical, whereby a correct analysis result is 

generated but is incorrectly evaluated or reported.

9 
Type 1 and Type 2 errors: breakdown by cause

Conclusion: the source of most of the misidentification errors 

are in the post-analytical phase. 

Errors 2008-2012 

(total= 21) 

Pre-analytical 2 

Analytical 6 

Post-analytical 13 

10 
Error rate in forensic DNA typing 

• The approach to focus exclusively on research 

into the rate of type 1 and type 2 errors will only 

partly provide insight in the actual quality status 

of forensic DNA laboratories. 

• The actual absolute number of such errors is 

expected (and shown) to be low and does not 

allow for a comprehensive statistical evaluation or 

recognizing trends in the error rate of the forensic 

DNA typing community. 

11 
Error rate in forensic DNA typing 

• It can be anticipated that a significant number of wrongful 

conclusions on the source of the DNA will stay undetected. 

• Most cases with an incorrect DNA identification result will 

be noticed by the laboratory before the forensic report is 

released. 

• Difficult to compare misidentification rates with error rates 

in other fields of laboratory science (i.e. clinical laboratories 

and genetic testing centers). 

12 
Broadening the scope of error rate 

• Robust quality management is a requirement for 

the accreditation status of the Netherlands 

Forensic Institute (NFI). 

• For that reason the NFI management has 

incorporated a registration system that allows for 

the registration of quality issues. 

• An "internal quality issue notification" in relation 

to the laboratory analysis of biological samples is 

made after any event that could lead to a failure 

or diminished quality of the result.

13 
Aim of this study 

• To develop a structured procedure for the 

identification, classification and scoring of 

quality issues within a forensic DNA 

laboratory which can be incorporated into 

routine practice. 

• It must allow for the prioritization of 

corrective actions and prevent the 

possibility to happen again. 

14 
Quality Issue Notifications 

• The NFI has created a culture in which the 

existence of risk is recognized and error 

prevention is recognized as everybody’s 

responsibility. 

• Notification of quality issues is part of the 

standard process of quality improvement. 

• Notifications take place in a blame free 

environment. 

• All staff members are authorized to report quality 

issues. 

• Dictum: don't accept error, learn from it. 

15 
QOL Quality Issue Notification

The NFI Quality On Line (QOL) system which is available on all 

workstations in the laboratory contains an automated, webbased system designed to make notifications. 

The QOL notification holds the following headings: 

Description of the quality issue 

1. Cause 

2. Scale (number of samples effected) 

3. Action and measures taken to correct the failure and 

prevent future incidents 

4. Operational nature of the improvement 

CSAO “De vier O system” CCKL (clinical lab): Oorzaak, Omvang, Oplossing en Operationaliteit

16 
Assessment of Quality Issue Notifications

• All notifications are scrutinized by the QC 

manager of the biology department. 

• Notifications are categorized and graded by 

potential and actual impact. 

17 
Referring to similar areas of laboratory science 

i.e. Laboratory Medicine 

A ‘quality failure’ in relation to a laboratory test is 

defined as any failure to meet the required output 

quality necessary for optimal patient care. 

Maurice J O’Kane, P L Mark Lynch and Noel McGowan Ann Clin Biochem 2008; 45: 129–134

18 
Quality failures in Laboratory Medicine 

Review of the literature

Laboratory Medical 

laboratory1 

Medical 

laboratory2

Molecular genetic 

testing center3

Medical 

laboratory4

Year of Publication 1997 1998 1999 2007 

Data collection period 3 months 3 years 1 year 3 months 

No. of tests 40 490 676 564 88 394 51 746 

No. of errors 189 4135 293 160 

Frequency 

(% of test results) 0.47% 0.61% 0.33% 0.31% 

1. Plebani M, Carraro P. Clin Chem 1997;43:1348–51 

2. Stahl M, Lund ED, Brandslund I. Clin Chem 1998;44:2195–7. 

3. Hofgartner WT, Tait JF. Am J Clin Pathol 1999;112:14–21. 

4. Carraro P, Plebani M. Clin Chem 2007; 53:1338-1342

19 
Quality failures in genetic testing 

“Mutation analysis of the other 1001 samples resulted in 10 discrepant 

results between the counter-expertise and reference laboratories. 

The overall accuracy of the reference laboratory was 99.8%, with two 

false positive results identified by the counter-expertise laboratory. 

Conclusion. The currently used mutation analysis is associated with 

a very low error rate. Therefore, we do not recommend routine use of 

duplicate testing”. 

5 Cholesterol 2013, Article ID 531658

20 
Interactive question 

• Forensic DNA typing laboratories make a certain 

inescapable minimum number of errors, no 

matter how diligent they may be. 

• No official body or forensic laboratory has 

published yet standards on the acceptable rate of 

error or desirable goals for the accuracy of the 

forensic investigation and report. 

• What is considered an ‘acceptable’ 

error rate in forensic DNA testing? 
Forensic Biology and DNA analysis at the NFI: 

number of quality issue notifications 2008-2012

2008 2009 2010 2011 2012 

No. of DNA-analyses 66.391 82.896 89.977 100.407 132.456 

No. of notifications 328 329 435 526 572 

Frequency (%) 0,5% 0,4% 0,5% 0,5% 0,4% 

Laboratory Med 

lab1 

Med 

lab4

Med 

lab2

Genetic 

test 

center3

Genetic 

test 

center5

Frequency 

(% of test 

results) 

0.47% 0.31% 0.61% 0.33% 0,1 % 

22 
Different categories of notifications

2008 2009 2010 2011 2012 

a. External origin 23 10 23 54 100 

b. External contamination 3 0 5 24 22 

c. Opportunity for improvement 11 6 3 (2) (10) 

d. Positive response 19 9 11 6 17 

e. Clerical (no adverse 

outcome) 

29 25 92 77 82 

f. Not related to case work 13 9 20 10 5 

g. Other (NFI related) 230 270 281 355 346 

Total 328 329 435 526 572 

23 
Categories of quality issues of type g 

(NFI related) by cause

24 

 2008 2009 2010 2011 2012 

g. Other (NFI related) 230 270 281 355 346 

Contamination 49 56 57 130 135 

Human Error 105 124 135 139 114 

Technical Problem 17 28 37 21 19 

Deviation quality 

Document 0 0 3 5 2 

Capacity / Planning 1 1 0 1 0 

Deviation from 

Competence Matrix 0 1 0 0 0 

Sample Mix Up 24 32 25 30 34 

Other 34 28 24 29 40 

Ongoing 0 0 0 0 2 
Total number of contamination notifications

2008 2009 2010 2011 2012 

Contamination NFI related (g) 49 56 57 130 135 

External (b) 3 0 5 24 22 

Not related to casework (f) 1 2 10 4 3 

Total contaminations 53 58 72 158 160 

25 
Who contaminated?

2008 2009 2010 2011 2012 

Total contaminations 53 58 72 158 160 

Contamination with DNA from a 

staff member 21 18 17 26 53 

Contamination with DNA from 

another sample 29 40 50 108 84 

External DNA contamination 3 0 5 24 23 

26 
DNA contamination in the media 

Number of reference DNA profiles in the 

elimination database from external crime scene 

workers 

Before 2008 158 

2008 174 

2009 241 

2010 410 

2011 567 

2012 620 

2013 820 

Total 09 Sept 2013 3060 

Detectives contaminate DNA 

traces 

27 
What was contaminated?

2008 2009 2010 2011 2012 

Total contaminations 53 58 72 158 160 

Contamination in control (blank, 

negative and positive control) 23 28 39 102 46 

Contamination in a reference sample 9 5 6 8 40 

Contamination in a crime sample 20 23 18 46 72 

Contamination in wipe sample (bench 

monitoring) 1 2 9 2 2 

28 
Intensity of the DNA contamination in the 

control samples

2008 2009 2010 2011 2012 

Contamination in control 

(blank, positive control) 23 28 39 102 46 

Sporadic contamination 9 8 17 74 18 

Gross contamination 

(source identified) 13 12 18 24 21 

Gross contamination 

(source not identified) 1 8 4 4 7 

29 
Potential impact of registered Quality Issue Notifications 

There is an important difference in impact between quality issues that have 

an adverse outcome on the forensic investigation and failures that have 

been recognized and corrected in an early stage of the investigation. 

Grading by seriousness of Quality Issues that have a potential impact 

on the outcome of the investigation 

0 Registered quality issue has no effect on the conclusions of the 

NFI report. 

1. Registered quality issue has a potential adverse outcome but 

can be corrected. 

2. Registered quality issue has a potential adverse outcome and 

can not be corrected. 

30 
Potential impact of registered Quality Issue 

Notifications (NFI related)

2008 2009 2010 2011 2012 

0. No adverse outcome 39 22 78 158 125 

1. Potentially negative outcome; 

repairable 144 197 138 155 137 

2. Potentially negative outcome; 

irreversible 47 51 65 42 81 

Under Investigation 0 0 0 0 3 

Total 230 270 281 355 346 

31 
The actual impact of Quality Issue notifications 

Over the period 2008-2012 we observed a total of 286 notifications where a failure with 

a potential negative outcome could not be corrected. We have investigated the 

actual impact of these notifications for the conclusions in the NFI report. 

0. Failure corrected if necessary; no further consequences are expected 

and no revised report submitted 

 1. Failure with an adverse outcome for the investigation and conclusions in 

the report, failure is corrected after release of the forensic report and a 

revised report had to be submitted. 

 2. Failure with an adverse outcome for the investigation and conclusions in 

the report; failure is irreversible. Failure has been stated in the original 

forensic report. 

 3. Failure with an adverse outcome for the investigation and conclusions in 

the report; failure is irreversible. Failure is detected after release of the 

forensic report and a revised report had to be submitted. 

 4. Actual impact unknown.

32 
The actual impact of Quality Issue notifications (NFI related)

2008 2009 2010 2011 2012 

Negative outcome; potentially 

irreversible 47 51 65 42 81 

0. Failure without adverse 

outcome 

22 16 36 20 17 

1. Failure with adverse outcome; 

failure corrected; revised forensic 

report 

 0 0 3 0 1 

2. Failure with adverse outcome; 

irreversible; stated in the 

forensic report 

21 32 23 22 60 

3. Failure with adverse outcome; 

irreversible; revised forensic 

report 

4 3 3 0 0 

4. Actual impact unknown 0 0 0 0 3 

33 
The actual impact of Quality Issue notifications 

The expected negative outcome for the crime case. 

• In cases, where failures have been repaired early, the NFI can 

reliably assume that there are no further consequences. 

• The actual judicial impact of other notifications is difficult to 

conclude by the NFI on itself; the importance of the forensic 

investigation for the crime case varies and the survey involves 

examination of the criminal file and feedback from the judicial 

authorities. 

34 
Quality Issue Notifications made transparent 

http://www.forensischinstituut.nl/nfi/kwaliteit/kwaliteitsrapportages

35 
NFI errors in the press

“Hundreds of errors in forensic DNA analysis at the NFI” 

36 
DNA errors in the Dutch parliament

“Politicians want clarification on the reported error 

rate in forensic DNA analysis” 

37 
Independent review of NFI notifications

“Quality issue notifications as indicator of quality” 

38 
NFI process optimalization DNA NFIPOD

Bridging the gap between the pre and analytical 

stage of the process 

39 
DNA NFIPOD 

• Automated storing/retrieving of DNA 

extracts from -80 oC storage facility 

• Automated sample preparing, liquid 

handling and analysis. 

• Automating these processes will 

significantly reduce the risk of error in the 

pre-analytical and analytical process.

40 
The post analytical process 

Managing the postlaboratory phase of the 

total testing cycle is of 

equal importance. 

A comprehensive process 

control through integration 

between automation and 

information management in 

the post analytical phase is 

expected to further improve 

the total testing process.

41 
In conclusion 2008-2012 

Number of DNA analyses 472,127 

Number of quality issue notifications 2190 

NFI related 1483 

Potentially negative outcome for the CJS; 

irreversible 286

False inclusion; Wrong exclusion 21

42 
Conclusions 

• Quality Issue notifications have been comprehensively 

incorporated into the routine of the Biology 

department at the NFI. 

• By means of a simple and consistent method for the 

identification, classification and grading of quality 

issues the notifications provide essential information 

on the cause of and on the potential and actual 

outcome. 

• The notification system generates important 

information on the performance of the laboratory and 

provides objective information to prioritize corrective 

actions. 

43 
Publication on error rates accepted in FSI 

Genetics 

44 
45 
To err is human1

It should be in the genes of the forensic scientist to 

create solutions, find better alternatives and 

meet the challenges ahead. 

1 To Err is Human: building a safer health system. 

Committee on Quality of Health Care in America 

46