2 April 10 DOJ Part2

1

JOHN BUTLER: Invited Elham Tabassi, who is from the Information Technology Laboratory of NIST, to 

give a presentation about her work that she’s been doing. Her background is in fingerprint analysis as 

you’ll learn. She’s done a lot of work with other biometrics and is part of a group that the ISO, 

International Standards Organization, is working on a document that will relate to statistical analysis of 

biometrics and also can be applied to forensic evaluation. So Elham is going to kind of review what’s 

being done in this area right now and how it relates to method validation and validation of these tools to 

help biometrics and forensics.

Go ahead, Elham.

ELHAM TABASSI: Thank you, John.

So some of you may remember me from Commission Meeting 11 when we came to NIST and I gave a 

presentation on our evaluations and work we did on forensic. I am grateful for the opportunity to come 

here and talk about our involvement with the newly-established ISO standard on – the title is a mouthful –

methodology and tools for the validation of biometric methods for forensic evaluation.

This is the outline of my talk. I have one slide say a little bit about forensic and biometrics, how they relate 

and how they differ. I’m coming from biometrics.

Then I start talking about standards, which is our first and middle name at NIST, and I get to talk about 

the why, how, and what are involvement and technical approach for standard development.

Then I start talking about this particular standard. I have to give you the wording of the method validation 

as many other terms in forensics means different to different people, so I tell you what we are doing at 

ISO.

And then I try to give you an overview of the development stage of documents in ISO and timeline with 

the hope and intent that you get involved and participate in this.

So biometrics versus forensics. As I told you, I am coming from biometrics. I have been doing biometric 

research since 1999. I get into biometric evaluations in 2001, biometric standardization in 2004, and in 

the past four years, actually the (inaudible) report brought me to forensic area.

They are similar because they both relate to recognizing human from biological or behavioral 

characteristics. You have to trace this and you have to answer the question, are they coming from the 

same source or not.

How they differ is basically they – the part in the middle. There are many different publications and people 

have talked about the relation and differences between biometric and forensic. I’m quoting from a paper 

by Anil Jain and Arun Ross. The main difference, to my opinion, is the quality, clarity, or signal (inaudible) 

ratio of biometric sample versus forensic sample. And that’s because in biometrics, the sample is 

acquired as a result of direct interaction of a person with a sensor device. I put my finger on a fingerprint 

scanner. As opposed to forensic, which is a collection of evidence, of traces, impressions left 

inadvertently, and being collected from the crime scene. The collection is usually manual, again as 

opposed to biometric that’s automated, and that manual and human intervention, as we listened this 

morning, is an aspect of forensic sciences and forensic processes.

In biometrics, more or less everything is fully automated, and we kind of like it about that.

And then because you have a lot of manual human involvement in the forensic, the outcome of forensic 

investigation or determinations also are often communicated verbally as opposed to biometrics that either 

the whole decision or decision coming out of the algorithm is more (inaudible) oriented because you get 

comparison scores.
2 April 10 DOJ Part2

2

So in my talk I’m going to go between biometric and forensic. I will tell you a little bit of where I’m coming 

from and what we have done. To help you follow, if I’m talking about biometric or forensic, those icons on 

the top, biometric is that nice, clear smiley face fingerprint. The signal is clear and is happy domain for us 

working with computers. And forensic is that messy, monstrous-looking – that’s supposed to be a latent. If 

I’m talking about biometric, I put the cross on forensic. If I’m talking about biometric I push it back to 

biometric. And if there is no cross it can be applied to both of them.

So let’s talk standards. As I said, standards is our bread and butter at NIST. It’s like in our, as I said, first 

and middle name. Standards are vital for technologies to become ubiquitous. This is not me saying it, this 

is Economist saying it. And what are standards? They are often specifications for interoperable and 

uniformly interpretable specifications for exchange of data, for performance testing. And the key 

emphasis is in uniformly interpretable. Two people can read it and get the same thing, and interpret it, 

and if its standard is developed correctly and clearly, they come up to the same thing.

What it does is give it a common basis. It makes people talk about the same thing in the same language 

and prevents talking past each other, which is my experience working in the forensic world.

It prevents vendor lock-in or protocol lock-in. It allows for marketplace of off-the-shelf product and allows 

for modular design and technology refresh. Being part of the Department of Commerce we are sensitive 

to that.

Standards for uniform testing and reporting and method validation also facilitate and promote repeatable 

and reproducible research because if we are all measuring the same thing, now we can talk about the 

same stuff.

And it also allows for performance improvement by preventing garbage in-garbage out. And that is, you 

know, you say standard, people roll their eyes. They say that it’s boring, it’s about syntax, it’s about 

(inaudible), about (inaudible), it’s not. Well, it’s all about that. But it’s more. It’s a lot about technical stuff, 

too. You can get quality design into the standards so that compliance with the standard ensures some 

level of not only uniformity, but performance, too.

So let me talk about that in the context of one of the work we did. Last time I talked about fingerprint. This 

time I talk about our effort in iris standardization. I’m talking about this just to show you and illustrate to 

you our approach and how do that and how we’d like to repeat this for (inaudible) standard that’s just 

being started in ISO for forensic method validation.

So back in 2000, iris came up as a viable – was introduced as a viable – biometrics. By 2005, the first 

international image (inaudible) format in iris was developed in ISO. In 2009 we were already revising the 

standard, improving the technical integrity and (inaudible). And one thing became apparent. That while 

the community had consensus that if you get a good-quality iris, recognition error is pretty low, there 

wasn’t a consensus of what constitutes a good image quality.

Algorithm developed for (inaudible) iris manufacturer, camera manufacturer, saying that the camera 

doesn’t produce good images, the manufacturer blamed the developers that your algorithm doesn’t know 

how to deal with this. The bottom line was that the failure to (inaudible) or failure to process was really 

high. It was something about in the thirty-something percent. So there was a lot of interest in studying a 

standard to define what is iris quality and how to measure this thing.

So we got lucky and NDHS was interested in that they supported us and gave us funding. And what we 

did, we started an evaluation-based program to answer this question, along working with community, and 

give quantity support the development of the standard. You might ask why evaluation, why you want to do 

all that work? The answer is to get the discussions founded in data so that you are not going in all 

different directions.
2 April 10 DOJ Part2

3

So what we did is we get the community together. The first thing we did, we came up with a long master 

list of anything that anybody thought had something to do with recognition performance, any image 

property that could constitute image quality. We gave many rounds of public comment working with the 

community. We came up with a list that everybody agreed that a subset of that should be altogether 

constitute iris image quality.

We asked community to implement that and give it to us. We ran broad-scale evaluations and we relate 

each of those image properties to the error rates getting out of the algorithm.

It was a lot of work. It was, you know, our joke was that our place was full of eyeballs for quite a year-anda-half. And out of that came – and what it did, it enabled scientific progress in iris image quality.

Remember we started by not even having a consensus of what is image quality, and we ended up with a 

standard that defined iris quality. And the standard had got all the empirical validation that it needs 

through the evaluations. That standard was published in 2015. We started it, I believe, in 2005, so it is 

quite common that standard takes three to five years in ISO development. And that standard right now is 

part of U.S. Government procurement. So anybody that wants to go and buy iris camera, they had to 

show that they are compliant their requirement in the standard.

By engaging the community from early on, not only we got the buy in from the community, but by the end 

of the development of the standard, the vendors had already semi-standard compliant product.

What is inside the standard establishes methods to quantify, you know, (inaudible) it says how to 

measure it, and it also says what is the acceptable range. An example of that, for some of the image 

properties, like anything else, you know you have some areas that consensus is easy, some areas that 

consensus is moderate to difficult, and some areas that you cannot get consensus. Example here is 

usable iris area. Everybody – I should have said at the beginning, iris is the colored part of our eyes.

So everybody agreed that for making iris recognition work, you have to have enough of iris visible, right?

You’re not doing iris recognition with closed eyes (inaudible) facial recognition with your face covered with 

a paper bag.

There was also consensus on how to measure the usable iris area, but there wasn’t any consensus on 

what the threshold should be. Camera manufacturer wanted 50% of iris visible. Algorithm developer 

wanted something like 80-85%. And our empirical work showed that the threshold is 70%.

Example of area that we had a medium consensus is the contrast between sclera and iris. And what 

everybody agreed that this has a role, an effect on recognition performance, and there was semiconsensus on how to measure that. There wasn’t any consensus what the threshold should be.

So what standard did, among the wish that, you know the wish list, that we started at the beginning, find 

out which one of them there was consensus to for definition, for computation methods so the (inaudible) 

what to measure, computation method, how to measure, units of measurement, what the threshold 

should be, and units of measure, and they become requirements. The “shall” statement in the standard.

Of course there were some statements that we couldn’t get consensus, either because we didn’t know 

how to compute it or the community cannot agree on a quantifiable, verifying, reliable, interoperable either 

metrics or value arranged. Example of that is looking directly to the camera. So they become 

recommended, not required but recommended. And standard, when it gets revised, maybe that gets 

tightened up.

So what we did in the process, and I just explained it to you for iris. In September I told you for the 

fingerprint, and this is basically our blueprint for doing standardization, it’s a cycle. It can start anywhere, 

but, for the example that I told you, it starts with identifying a gap. There was a gap in what constitutes iris 

image quality, right? Then we develop a research or large-scale evaluation to bring the community 
2 April 10 DOJ Part2

4

together to provide quantitative data to give empirical backing, at the same time granting the discussions 

all in the data so we can’t move forward.

We do that evaluation or research, and then we get the technical contribution, go to the standard meeting, 

advocate for that. All along keep the community informed. And out of this comes standard.

We hope to apply all this lesson and this approach during the development of this new standard. This is 

Part 8 of a already multi-part standard. The standard is Biometric Performance Testing and Reporting.

The values part of that talk about performance metric for biometric recognition algorithms. False match, 

false non-match, specificity and sensitivity that we heard this morning.

The standard took about a year-and-a-half socialization before the new work item was passed. The 

multiple presentations. There was a lot of pushback because the first presentation that they came, it 

came from NFI. You probably know Didier (sp), Muller (sp), and Daniel Ramos. They had a publication, at 

that time it wasn’t yet published, that talks about (inaudible) ratio methods for forensic evaluations, and 

they insisted on standardizing that content. And that got a little bit of pushback because we didn’t just 

want to go with one particular method.

At any rate, the new work item was submitted. About a year ago in January the result of the ballot was 

discussed, and it was passed as a new Part 2, 19795. But U.S. position was that we’d like to see it as a 

separate project, more focused on the forensic approaches rather than being part of the biometric.

There was discussion about is it within the scope of (inaudible), that’s biometric, or should it go to the 

different subcommittee. Out of the meeting of ISO subcommittee biometrics in January, it is established 

as a new part of biometric performance testing and reporting, but at the same time, the resolution asked 

for establishment of a new project for this. The reason for that is we see that it can be a multipart, and 

there was some consensus in the room that a good portion of this should be technical report rather than 

(inaudible) requirement.

On the right, you cannot see it, but I have the Table of Contents on the division of this. Basically what it 

does, I just had the discussion before lunch, you thing about methods of validation and you say that, oh, 

it’s going to go and tell me how to validate my product, or tools, or, you know, how I do the determination.

It’s actually – the way it’s being structured right now, everything is on table right now thinking about the 

iris quality, that we start by having everything on table. It talks about approaches for forensic evaluations, 

but it includes both strength of evidence and strength of hypothesis. This morning there was talk about, 

well, you know, the penultimate document that they were talking about, I hear that you were discarding 

the strength of hypothesis, rightly so I think. But the standard already have a clause. The clause is empty, 

but it’s waiting for contribution.

Let’s go through all the causes and see what the standard is about.

The standard as it is right now is, as I said, empty clauses. So we have a direction that we think it’s going 

to go, a general path forward, but like any standard it’s going to be shaped by the contributions and 

comments it’s going to get. It really depends on the people sitting around the table and how they 

advocate for where they want to go.

Any standard, any ISO standard, has three terms and definitions. And I put here that right now the 

standard has definitions for empirical probability and subject probability. And you guys know that depends 

on if you are Bayesian camp or non-Bayesian, one of the two doesn’t exist. So it’s going to be interesting 

discussions in the room trying to define both of them when half of the room is going to say that subjective 

probability has no meaning.

Another interesting thing is that it defines Bayes’ Factor separate than Likelihood ratio. It’s advancing 

Likelihood ratio just purely as the ratio of the two probabilities of the probability event under two 
2 April 10 DOJ Part2

5

hypotheses where the hypotheses are mutually exhaustive and exclusive, emphasis on mutual 

exhaustiveness of that.

Bayes’ Factor wants to get that ratio multiplied by the prior to go to the posterior. And at this point the 

standard is separating the two.

It’s worth to mention that the editorial team of that, we have people that are purely Bayesian, we have 

people that are purely non-Bayesian, and there are people like me who don’t know what Bayesian or nonBayesian is, where should I be. So it’s going to be interesting discussions.

The conformance clauses and other standards clause in every standard.

Then we have Clause 5, which we think that is going to develop as an informative, maybe separate as a 

technical report, that’s going to talk about approaches for forensic evaluations. And we’re going to talk 

about strength of evidence that basically says that the evidence shows more support for hypotheses of 

HD versus HP. And then strength of hypothesis that, I believe, that the data or the evidence shows 

association for or support for same-source conclusions.

There are some texts in the section for strength of evidence. There is no text for the strength of 

hypothesis, but there are people coming from biometrics who are pushing for what in the field is being 

called as score-based likelihood ratio instead of being the ratio of the probabilities being the ratio of the 

cumulative distribution functions that makes them distribution of errors, so we will all see where it goes.

Then on the Methods for Forensic Evaluations, for the strength of evidence, it goes and talk about both 

Bayesian and non-Bayesian. Under the Bayesian, the part that I put in quotations is all that is in the 

standard for now. So you think that Bayesian is all there, everybody agreed to, but that’s all we get for 

Bayesian. Posterior equals Likelihood Ratio times priors. I’m sure that’s going to develop as it goes 

forward, but the non-Bayesian section is actually about one-and-a-half or two pages that talks about 

Fisher’s Likelihood Ratio, which is the ratio of the probabilities and other methods.

Then we come to Section 7, Validation of Methods. That’s a normative part of the document, and that is 

the contents in Didier’s paper. What is does is basically from the beginning establishes that this section 

should be agnostic to the approach that you get. So your approach is Bayesian or non-Bayesian, is 

strength of evidence or strength of hypothesis, these validation methods should be applied. And it goes 

on to define what to measure, how to measure. Didier’s paper talk about validation criteria or basically 

threshold for accepting that it’s validated, but it’s not in the standard yet. Instead there is just the graphical 

representation.

So when it talks about what to measure, it defines whether the – I hear the word figure of (inaudible) this 

morning. So you want to measure accuracy. You want to measure discriminative power, calibration. And 

there are secondary characteristics, that robustness, generalization, and monotonicity. I think your people 

call it coherence. The people at the room said coherence and statistical analysis has a particular 

meaning.

The problem with (inaudible) is that as the (inaudible) information content evidence gets larger, you want 

to see a larger likelihood ratio. But because the way implementation of the (inaudible) method works, it’s 

not always like that. Sometimes you clearly have evidence with more information in it, but the value of 

likelihood ratio is smaller. So that’s that secondary metric there.

For each of the performance characteristics or what to measure, there should be a metric that explains 

how to measure that. That’s the purpose of the middle section of this slide. And then graphical 

representations. Advancing the (inaudible) and so many other (inaudible) that has been used in detection 

estimation, sometimes computer vision and medical disciplines for a long time, and (inaudible) them in a 

more formal standardized format.
2 April 10 DOJ Part2

6

So what is the timeline where we are? I stole that slide from the presentation that Gordon, Warren, and 

Karen gave to you in meeting number five. The slides show the progression of an ISO standard. It starts 

with new work item proposal, goes to working draft, then committee draft, then draft international, then 

final draft, and then publication. Each of the cycle, it can cycle through, it can iterate through. You can 

have Working Draft 1, 2, 3. Committee Draft 1, 2, 3.

At the end of each – at each of the meetings for each of the stages, there is a comment period. The 

comment period for the base draft of the standard is up to May 27. Anybody can – when it is in working 

draft, I was just checking (inaudible), you can go, you look at it, you can comment on the document. The 

comments all get reviewed and discussed at the next meeting. Each meeting is six months away. So you 

can see that going through that, even if it only stays one or two times in each of the cycles, it will give you 

a good three-to-five years.

I should have said that, as you can see, 19795, the standard has not even yet made it to the Working 

Draft stage yet. The next meeting of ISO is going to be in beginning of July.

So how to send your comment, how to get involved. This is the subcommittees for ISO/IEC. And you 

have heard ISO/IEC JTC1. JTC1 is the Joint Technical Committee of ISO and IEC under each of them.

Under the JTC1 is different subcommittees dealing with different scope and program of work. Thirtyseven is for biometrics, but we also work with 27 and 17 that deal with the security and communications.

All these subcommittees, they are on the boundary of each other. They harmonize their work. First at the 

time of the new work item proposal, you know they have to make a clear case that why it has to be within 

the scope of this subcommittee and if there is any standardization in any other subcommittees. And then 

later on through liaison relationship, they can get documents from each other and look at it and comment 

on it.

There is also the Technical Committee 272, which was established probably two years ago, I think. I have 

two slides on that and they can tell you. But the first one (inaudible) in the sense that it is more on the 

process on product whereas 37 is more on the data on performance testing and interfaces, stuff like that.

One thing that 272 did, and again as part of (inaudible) this document we’re going to do is development of 

the vocabulary, which is tedious but very important. 272 has borrowed the first draft of the vocabulary 

from OSAC, but they may go and take it and do whatever they want to do with it so the result may or not 

be similar to what is in OSAC.

So participation in these subcommittees is through your national bodies. Each country has its own 

technical advisory tag, technical advisory group tag. NC is for US, BSI for UK, (inaudible) for Germany, so 

on, so forth. For you guys, if you have – SC37 10795 is at SC37, but it’s (inaudible) liaison 272.

If you want to participate, if you have good ideas or empirical data or questions and you want to be part of 

this, I was just checking with Fran. I believe that up to Working Draft you can – up to Committee Draft you 

can submit comments as expert through your national body, but I’m not sure if you have to be part of NC 

or not, but I’ll talk about this later.

So Issues and Challenges. So this is my pitch to you guys that why it’s a fun thing to come, ignore all 

those non-smiley faces. You know, at the end of the five years you will have a smile on your face.

Standardization process doesn’t always go as planned or intended. You start by something in your mind, 

and you almost always don’t get what you want. And it always takes longer than what you want. But it’s a 

consensus-building process. It means that majority wins, but majority may or may not be correct all time.

That’s why it’s important to participate. That’s why it’s important to keep the discussions data driven and 

grounded in data so we can get standard that’s as clear, as implementable, as tightly-defined as possible.

Doing empirical work and giving quantitative support to standard obviously needs data, and you want 

data that is right for the study that you want to do. And that is always another challenge to come. That’s 
2 April 10 DOJ Part2

7

again why you should be involved because each of us has a little bit data but all of us together don’t have 

a lot of data to answer all of the questions.

So I talk about a lot of these things, so, again, standards, don’t roll your eyes. It’s all about commas, it’s

all about syntax, but it’s not just about them. There is a lot of good, interesting technical discussions going 

on. I suspect that we’re going to have a lot of fun and frustration discussing the definition of probability 

and empirical probability versus subjective probability, Bayesian versus non-Bayesian. And while it seems 

quite easy to – oh, accuracy is easy, you get specificity and sensitivity. But unless you know what is the 

method that you are going to use, you are not going to be able to write a metric for evaluating that. So 

there’s going to be a lot of – you know, many of the discussions that I hear this morning and discussions 

that we had at our May workshop last year is going to be repeated there.

To ensure interoperability, requirements shall be stated in a clear, implementable, sufficient, and testable 

manner. And that’s were, again, all the quantitative backing of the data comes. That’s why you need 

empirical validations. You want to make sure that two persons can get the standard, and read it,

understand the same thing, and when they implement it, they come to the same result.

It has to be sufficient that the guidance in the standard can take you to the implementation that you want.

And I talk about majority is not always right, and that’s, again, why it is important to have many different 

point of view participated and come to the meeting.

So you can be part of that. You can bring all the good discussions you have here, all the studies and 

research you are doing in your agencies, in your laboratories to the standard development process. How 

to do that, so part of it is – I think I have another slide on that, or maybe I have to go back.

So part of that is by commenting on the drafts. As I said, again, the standard right now is a lot of clause 

with titles, heading, and no content inside each of them. And when the standard gets developed, again it 

takes a lot of commenting and reviewing because it’s the committee that takes the document where it 

needs to go.

It’s still at the very early stage. There’s a lot of opportunity to comment. The base draft is out to comment 

until May 27th. For you to comment on that, I believe you can go to the ISO. I didn’t put the link here 

because I couldn’t disable the password on my laptop for that.

But I checked that, and I put it in the presentation that I put in the folder of this meeting. I believe you can 

go get it and then send a contribution.

You can just review and comment on that, but even better you can contribute technical content that 

supports it by data and show where the empirical backing and quantity backing is coming from.

And the best is be a member of the INCITS M1, which is the technical advisory group for the ISO. Then 

you can come and advocate for your organization and your organization and your research.

That’s my presentation.

JOHN BUTLER: Thank you, Elham. Are there one or two questions before we move to our next group?

Yeah, there are some questions. Yes, Greg.

GREGORY CHAMPAGNE: A lot of law enforcement agencies, and particularly those that run jails, are 

beginning to collect iris scans and have been for a few years, and I’m one of those. The technology was 

fairly cheap, and we use it – most correctional institutes that use it, it’s part of the booking process. You 

take the fingerprints, take a mug shot, put their face in the camera, we’ve got the iris scan. So I’ve got all 

these iris scans and we’ve been collecting them, and I’m just wondering, you know, no criminal leaves an 

iris scan at the scene of a crime, and so I’m just wondering if it’s good to be analyzing it. I’m still trying to 

figure out if it’s not a science in search of a practical use. We use it to make sure we don’t release the 
2 April 10 DOJ Part2

8

wrong inmate. Okay, we took your iris scan a few days ago, you’re bailing out, and so we can be sure 

you’re the same person that we booked. So are there any other practical forensic uses in criminal trials 

that you’ve heard of?

ELHAM TABASSI: No, actually that was – that was exactly that. So (inaudible) started trying to get into 

forensic iris. And they were talking about – it’s this iris expert working group – and they were talking about 

it, it was something about seven months ago, and I asked the same question, that what’s the forensic –

because people don’t go around leaving their irises in the crime scene. They are – for identifying people 

after death from the irises, so apparently they are doing some studies about how long irises stay viable 

because fingerprint post mortem has more problem, I’m told, than iris. But, right, you know, iris is part of 

(inaudible) of NGI, and then the question was, well, we don’t have a database so what are we going to do 

if iris is this good, and the first application that come to their mind was for booking, and right, controlling 

the people going inside and outside of the (inaudible).

GREGORY CHAMPAGNE: Do you know how many vendors are collecting iris scans, or what’s the 

volume of them being collected?

ELHAM TABASSI: Vendors?

GREGORY CHAMPAGNE: Vendors. Companies. How many. I mean, I know of one company. It’s called 

B2 Technologies.

ELHAM TABASSI: So what is the – actually the number of iris companies is growing.

GREGORY CHAMPAGNE: Right.

ELHAM TABASSI: So when we did the first test back in 2005, we have five or seven participants. When I 

did this test in 2009, I had I think 12, 13 participants. Right now we are doing a evaluation of iris, and we 

have a lot more, I want to say close to 20 participants. The other thing that we allow, we allow each 

vendor to participate more than (inaudible) algorithm. But the point is back in 2005, there were only big 

names, like (Inaudible). Right now you see a lot of smaller companies coming.

The utility patent that was expired in 200 – I don’t know – 2, by Professor (Inaudible) that allowed a 

flourishing of the iris market.

GREGORY CHAMPAGNE: Thank you.

JOHN BUTLER: Any other questions? Well thank you, Elham, (inaudible) for speaking.

Okay, Nelson.

NELSON SANTOS: Okay, so we’re going to move into the (inaudible) panel, and I’ll turn it over to Peter 

who is going to open with some comments and then he’ll moderate the session. So Peter?

PETER NEUFELD: Okay. So, can I get a minute to assemble? All right.

So for the afternoon’s panel I’ve been asked to moderate. You’re going to be hearing first from Keith 

Harward, who is a DNA exonoree. We will be joined by video by David Angel from the District Attorney’s 

Office in Santa Clara who runs a conviction integrity unit. We will have two speakers from the National 

Institute of Justice – of the Department of Justice, Ms. Sarathy-Jones (sp), and our own fellow

Commissioner Gerry LaPorte. Gregory Dutton, who’s up there, is there to help with questions if need be 

later on. And finally we will end with the Executive Director of the Innocence Project, Maddy deLone.

I’ve been asked to just give a few minutes of background for Mr. Harward’s presentation on the lawyering 

and legal context for Mr. Harward’s story, which I’ll do in a few minutes.
2 April 10 DOJ Part2

9

His story begins on September 14, 1982 in Newport News, Virginia. On that night a couple lived in a 

house with their three tiny children a couple of blocks from the U.S. Naval base. In the middle of the night, 

a single white male intruder entered the house. The wife was awakened, hearing her husband call out 

while he was being struck repeatedly with a crowbar and killed by this intruder. After the intruder had 

killed the husband, he proceeded to rape the wife. He raped her several times, and during the course of 

that rape he also bit her on her leg repeatedly.

When the perpetrator left, she immediately called the police. She was taken to the hospital where a rape 

kit was prepared. She gave a full description, as best she could, which namely was a white male in a U.S. 

Navy sailor’s uniform, clean shaven. And she said that the perpetrator had not only raped her, but after he 

raped her he cleaned himself off on a baby’s diaper in their house and a blue towel in their house.

The attention of the detectives in Newport News focused on the USS Vinson, a huge aircraft carrier that 

had just come into dock a couple of days before for repairs, so there were a couple of thousand people 

on leave in the Newport News area.

A naval dentist, helped by a local forensic dentist, compared the photographs of the bite marks taken 

from the woman’s leg to literally hundreds of sailors on that ship, and the two dentists were able to 

exclude those hundreds of sailors including Mr. Harward as having a positive association to the 

impressions recorded and the photographs taken by the police.

Approximately six months later, Mr. Harward was involved in an argument with his girlfriend that 

ascended into a physical matter, and he was charged initially with assaulting her as a misdemeanor, and 

she had claimed that during the course of the assault that he bit her. The charges were ultimately 

dismissed, but meanwhile the detective had Keith Harward brought to the courtroom for an arraignment 

just so he could bring the wife who had been the victim six months earlier of the rape and murder to see if 

she could identify Mr. Harward as the perp. She came to the court, looked at the show up, and said, no, I 

can’t identify him.

Nevertheless, despite the non-identification, the detective took new dental molds of Mr. Harward and sent 

them off to one of the leading forensic dentists in America, Dr. Levine, who was one of the founders of the 

American Board of Forensic Odontology located in New York and working for the state police there. And 

Lowell Levine said that it was a match to Mr. Harward. Later on, of course, he would testify that it couldn’t 

have come from anyone else but Mr. Harward.

The police secured a second forensic dentist’s opinion locally. This raises the whole cognitive bias issue 

that this Commission has taken up. He agreed with Dr. Levine.

They went back to the two dentists who had said initially there was no match, namely the Naval dentist 

and local forensic dentist. They were told of Dr. Levine’s conclusions. They then recanted their initial nonmatches and said that they were now convinced that indeed it was a perfect match to Mr. Harward. The 

defense tried to get their own experts, other board certified forensic dentists, but they all agreed with Dr. 

Levine.

This was a capital case, he faced execution, he was convicted, he’ll explain why he wasn’t executed. The 

case went up on appeal. There was a minor technical error. It went back for a new trial. The two forensic 

dentists gave the same exact testimony they had given in the first trial that there was no probability that it 

came from anybody else or that there was a very, very, very, very small chance that anybody but Mr. 

Harward could have produced those bite marks. He was convicted, got life. Spent a total of 34 years in 

prison.

And he eventually wrote to the Innocence Project. We took his case. We got cooperation from the 

Commonwealth’s attorney in Virginia to seek DNA testing, not only on the rape kit but also on that diaper 

and that blue towel that the victim said were handled by the perpetrator when he was cleaning up.
2 April 10 DOJ Part2

10

All the evidence without exception came back to a single male semen donor which excluded Keith 

Harward. It was run through CODIS. CODIS got a hit to another man. That man was also a sailor on the 

USS Vinson in 1982. People tried to find him, and he had been sentenced to prison quite recently in Ohio 

for the abduction of another woman, but he had died in prison.

The lawyers for Keith went to the Virginia Supreme Court (inaudible) what’s called a Certificate of Actual 

Innocence. That certificate was not opposed by the prosecutors but indeed joined in, and it was granted.

And after 34 years in prison, Mr. Harward, about a year and two days ago, walked out completely 

exonerated.

The two final points I just want to make from a legal perspective is that each scientific review that has 

occurred of forensic bite marks over the last several years, all of it have found it wanting. None of it have 

found it valid. And in fact the most recent review by Peak has said that given the data it was not only not 

valid, but the likelihood that it would ever be demonstrably valid is so small that it wasn’t worth the 

resources.

Despite the fact that every scientific review reached that conclusion, this type of evidence has been 

admitted in just about every state of the Union. It’s never been kept out of any state. It’s never been found 

to be invalid despite all the scientific findings. So that shows you the incongruity between what the 

scientists say and what the courts say.

And finally, because it’s about this Commission, this Commission passed a document recommending the 

adoption of a code of personal responsibility which said that when errors are discovered that there be a 

notification to the affected parties. And in this case, despite the fact that it’s now been 12 months since 

Mr. Harward has been proven absolutely innocent, none of the responsible forensic dentists have

acknowledged any error at all, and no single individual who has been convicted on the basis of their 

testimony has been formally notified. And the reason there has been no formal notification, contrary to the 

wishes of this Commission, is that the only way those people can even be identified is with the 

cooperation of the forensic dental community. And so far none of the dentists who committed error in Mr. 

Harward’s case or the other dentists are willing to provide lists of their clients so that those people who 

are in prison, or may have been executed, can have their families notified.

Mr. Harward.

KEITH HARWARD: (Inaudible) am the first exonoree to be at this Commission, and apparently the last.

And I’m sorry to hear that because what I’ve witnessed so far, this is some good stuff, way beyond my 

intelligence level but you all seem to know what you all talk about in contrast to the two odontologists who 

testified against me.

A year and two days ago I stepped out of Nottoway prison in Burkeville, Virginia being unincarcerated and

unconvicted. Some would say, well, you’re a free man. I will never be free of this. There’s no possibility.

Excuse me if I get emotional. That I spent more than half my life in prison behind the opinions and the 

expert egos of two odontologists that at my trial one even made the statement to my lawyer when 

questioned about this, he says, well, sir, you’re just a lay person. That’s a mistake that a lay person would 

make. Just believe what I say.

In a court of law in the United States of America, contrary to popular belief, you are guilty until you are 

proven innocent. When you’re in that courtroom, the jury is going to believe that you’re guilty because the 

police, the prosecutor, and the judge is not going to have an innocent person be tried for a crime he did 

not commit. But ladies and gentlemen, here I am. All three of those people that were involved knew what 

they were doing. And they allowed this bite mark evidence to be used because they had no other 

evidence. None. And what evidence we found out later, thanks to the Innocence Project, my lawyers did 

not get.
2 April 10 DOJ Part2

11

The odontologists, Levine and Kagey, and I won’t call them misters or doctors or anything else because 

those yahoos don’t get respect from me because what they’re doing, they’re ruining peoples’ lives. And 

they’re still doing it I found out. There’s a death penalty case in Pennsylvania that’s going on now, and the 

judge is going to allow bite mark evidence. How many people have to be wrongly convicted before they 

realize that this stuff is all bogus, it’s all made up?

They were willing, the two odontologists, were willing to have me murdered by the state of Virginia behind 

what they said, which in all actuality has no basis in truth. They were willing to have me put to death 

because their egos said, okay, we’re experts, and what we say goes. And when you present an expert to 

a jury of people, or course, whatever they say is true because the judge allows it. The prosecutor 

presents it. It has to be true. So, of course, human nature, whatever they say is going to be true.

Even before they even start giving evidence, when they start talking about their credits, commendations, I 

was a past president of this, I was a chairman of that, the jury was done and so was I. And in my case, 

the shipyard guard, which if you read my story, was not the only person that was hypnotized at my trial.

The jury was eating out of their hands just because of the things that these odontologists had been 

involved with. The people, and the trials, and the cases. Chasing Nazis, Ted Bundy, things like that. And 

that gave them credit. That gave them credibility. And it’s just not right. Why is it still allowed to happen?

Why is a death penalty case in Philadelphia, I think it’s Philadelphia, or Pennsylvania, the state of 

Pennsylvania, still – they’re going to allow it. Why? Why can’t you err on the fact that, okay, well let’s find 

some other evidence to convict this person. Why use bite mark evidence when just this year alone two 

other people besides myself have gotten out (inaudible) bite mark evidence was used in their trial. I 

mean, explain to me what does it take to realize this stuff is all crap.

There should be some kind of regulation, there should be some way that experts have to meet a bar. A 

very high bar. Because you end up ruining people’s lives, and in my case, I’m not the only one that was 

ruined. The witness – I mean the victims, they were victimized all over again. The woman that was raped, 

a horrible, horrible situation, finds out 34 years later that the person that raped her originally, supposedly 

me, was on the streets. A woman can never get over a rape, I don’t think. I don’t this it’s possible. They 

can move on. If you lock the guy up where he’s not going to come back, the bogeyman is not going to

come back.

Well, those odontologists and those criminals in Newport News, the judge, the prosecutor and detectives, 

went out of their way to convict the wrong person just because they couldn’t find a way to do it any other 

way. They took shortcuts and railroaded me to fill that conviction. And for her to find out all these years 

later that a guy was still out there, and in this case he was still doing bad things, and that’s true with 

wrongful convictions. When you use bad science to convict somebody, the perpetrator continues on to 

commit crimes. So what good are you doing? You’re locking up the wrong person. You’re allowing the 

person that did it to go on. And the state of Virginia, the good people in Virginia, they were sold a bill of 

goods. Or as they say in prison, there was a perpetration of fraud by those people in Newport News and 

those odontologists.

And the most important victims of this were my parents. The only time I ever saw my father cry was when 

he was on that stand begging for my life. And nobody should have to go through that, ever. And I was the 

youngest, I was the baby. And I was spared the death penalty. My parents were not. Every day they had 

to deal with it. And it killed them. Just because of these odontologists making up stuff and testifying just to 

stroke their egos and to get a conviction. The prosecutor, the police, and the judge, so they could get a 

conviction. So they could satisfy their needs. And it’s just not right.

And to hear that you’re still doing that, why can’t you all stop it? Why can’t you all find out a way? I mean 

it’s just common sense. Why? Why when there’s a possibility of something not being true, or a possibility 

that something is wrong, why speed forward and go with it just because you can? Why not stop and say, 

okay, let’s pause for a moment and study this. Like you’re doing today with the documents trying to get 

the proper words going. Oh, you could mow right through it, and people aren’t going to be satisfied. But 

you all are going to take to stop and say, okay, we’ll just get this right because it’s important.
2 April 10 DOJ Part2

12

And to allow forensic science to give credit to where there’s no credit, none whatsoever, common sense, 

it shouldn’t be allowed. It should be stopped. There should be some kind of regulation or some way, far 

beyond me to figure it out. You all are the smart ones here, not me. To be able to have something set up 

to where people have to pass some type of validity test with their testimony.

I just saw something on the TV the other night that shocked me. I mean it just stopped me in my tracks.

There is a guy that is testifying to the headlights on an automobile at night to identify the perpetrator’s 

vehicle, and he’s standing up saying, yeah, well, see the light pattern, if you look at this it goes that way 

and we measured and – come on. I mean, really? Use DNA. That’s what got me out. And if it hadn’t been 

for that I wouldn’t be here today. I’d still have my butt sitting in prison waiting for these great people to try 

to get me out. But the DNA, it’s, you know, it’s, you know, here you go, here’s the proof. And six 

odontologists altogether agreed, and you’re wondering maybe how is that possible? Well, the reason why 

is it’s an old boys’ school. It’s a club. And they all rely on each other to back each other up. If one 

odontologist stepped out and said, well, I’m not sure, that’s career suicide. He would be done. And, you 

know, that would be it. And the old dog, the people that are still involved, they’re still in charge. They’re 

not willing to fess up to the fact that they made mistakes. They’re not men enough to say, okay, I was 

wrong. Now how can I correct this? Because there’s people still in prison, behind (inaudible) evidence.

Regardless of any other evidence that’s involved. But that stuff is so powerful that it would taint anything 

else that comes on. My case that’s all there was. But if there was other evidence, it could be – so give 

those people an opportunity, again, to have a trial and take that evidence of bite marks out and use 

what’s truly there. Get a right conviction, you know. People need to be punished for what they did. I didn’t.

And I spent more than half my life. I went in when I was 26. I got out a year and two days ago and I was 

60. I still have six more years before I break even behind what somebody said was true and was not true 

and the courts allowed it.

What do judges do – what are they supposed to do? Mr. Judge – I don’t see him. He was sitting down 

there. Aren’t they supposed to stop this type thing or do something to study it a little farther instead of 

letting it go?

But it’s just not right. And I’m sorry to hear that this Commission is coming to a halt because it sounds like 

you all would be on my side and moreover people that are on the inside’s side. And they are the ones that 

need the help. I’m out now, and I’m doing everything I can to help. But I’m just one person. And I 

appreciate you all for what you all have done in the past. So.

DAVID ANGEL: Hello? Can you hear me? This is David Angel.

UNIDENTIFIED FEMALE SPEAKER: Yeah, we can hear you.

DAVID ANGEL: Excellent. I can hear you now. I lost you for a minute.

UNIDENTIFIED FEMALE SPEAKER: David, I think when we’re not speaking it’s very – you don’t hear 

anything, the mic has to be on. So Peter, do a little momentary introduction and then David will start?

PETER NEUFELD: Okay.

DAVID ANGEL: I’m sorry, do you want me to start now because, again, I can’t –

PETER NEUFELD: Yes, please.

UNIDENTIFIED FEMALE SPEAKER: Yes, David, please start.

DAVID ANGEL: Here we go. All right, folks. Well, let me see if I can get onto my PowerPoint. Here you 

go. So unfortunately I cannot hear you, so it’s a little unnerving. I’m hoping that you can see my 

PowerPoint now.
2 April 10 DOJ Part2

13

My name is David Angel, an Assistant District Attorney in Santa Clara County. We have run a conviction 

integrity unit here since about 2004, so we were one of the first in the nation to start one. And what I 

thought I’d do is just spend a little bit of time talking about what we do in some of the issues surrounding 

it. I think from the prior two speakers you can certainly understand why it’s so important, you know, to 

have a conviction integrity unit.

But let me just start off, I think the first question one needs to ask, and this is the first challenge, is what is 

guilt? What do we mean by guilt? And at some intuitive level I think we all know what we mean by this.

We mean, just like our last speaker, someone who actually didn’t commit the crime. But if you begin to 

probe that a little deeper, you realize well do you mean actual innocence, like our last speaker, someone 

who just didn’t do it at all. Or do we mean how do we measure that. So, for example, do we mean actual 

innocence in that you have DNA evidence that just dispositively proves they could not possibly have 

committed this crime, or do we mean that the evidence in the case has eroded somehow so that maybe 

they’re – do you mean that they are innocent beyond a reasonable doubt?

The legal standard tends to be new evidence that’s been presented that raises a reasonable probability 

that the jury would have come up with a different verdict. I think that in my experience, and I’ve personally 

run, I think, four or five exonerations, and when I’m in the – you know, usually at the end you are going to 

be with the elected DA trying to make a decision on what to do on this particular case. Usually it’s a less 

formal standard. The DA – you ask yourself, would we have tried this case? Would we have charged this 

case, even, had we known everything we know now.

I do want to add, though, that there is another standard that sometimes people want to use which is has 

the integrity of the trial so collapsed that even though you can’t really say this person is innocent, you no 

longer have faith in the outcome of the verdict. And these things are judgment calls. I mean, the easy 

case is, at least intellectually, is if you have DNA evidence that just proves that this person couldn’t 

possibly have done it but this other person did. But in, I think, four of the five cases that I’ve worked on, 

we didn’t have any DNA evidence. It was just kind of old-fashioned police work and prosecutorial work, 

which means at the end of the day you’re just weighing a lot of different pieces of evidence and trying to 

figure out what they mean.

The next question that you have to consider is well how are we supposed to detect if someone is 

innocent. And in thinking about this question, or what I was going to talk about with this particular group, 

you know, you are forensic scientists. And so the challenge here is we have a system for figuring out 

whether people are innocent or guilty, and it’s called our criminal justice system, our trials. So what do 

you do when that system fails? And I think that an analogy could be to what a lot of you do. You know, 

there’s certain tests that if you run them, you’re going to get a certain number of false negatives and false 

positives. Sometimes you can just re-run the test, right? You can use some degree of redundancy to 

reduce that likelihood of error. But there’s other times that you’re going to get the same bad result each 

time because of certain systemic blindnesses within the system.

All of those are challenges that face us when we’re looking at these cases. So bear in mind, you know, for 

someone to be arrested, they’re not supposed to be arrested unless there’s probably cause, and they’re 

not even supposed to be searched or stopped unless there’s some sort of reasonable suspicion. And 

certainly in California, I think every state, there are – if somebody is arrested and they believe there 

wasn’t probably cause, there’s actually a court hearing that you have that determines just that question.

Then there’s a preliminary hearing or a grand jury where they also are supposed to determine whether 

probably cause exists to believe that this person probably committed the crime that they’ve been accused 

of.

Of course then there’s the trial where you have a reasonable doubt standard. And then if people don’t 

think that was right, there is an appeal process, and then there is a habeas process. And then since 

you’re a federal commission, you should be aware, or you are probably aware, that you then do it all over 

again, right? So you did it all at the state level, and then you do it all over again federally.
2 April 10 DOJ Part2

14

And so one might wonder, you know, how is it that after you’ve had all those tests for innocence or guilt, 

how do you have anyone left over? But the fact is now we know that we do. And in many ways, thanks to 

the work, I think, of the Innocence Project, both in New York and across the country, any of us – I 

certainly believe and I became a prosecutor over 20 years ago, that the likelihood of an innocent person 

being convicted was like the chance of someone being struck by lightning. You know, vanishingly small.

But, in fact, we now know that it does happen, and it happens with some regularity. And that’s why I put 

CIU, Conviction Integrity Unit, last with a question mark. So what is going to be the role of a conviction 

integrity unit?

Well I think offices should have them. We’ve had it for a long time and I think they’ve done a lot of good.

But you have to decide what your role is going to be. I think one role should be looking at new evidence.

And an example of that was raised by our last speaker, and I think Peter earlier. One category of new 

evidence is if you have advances in forensics disciplines. And in this case the advance to some degree is 

pointing out how evidence that was previously deemed reliable is not viewed as reliable at all anymore.

So I think that’s one category.

I’m sure you are well aware of one issue that we’re looking at in conjunction, or at least in collaboration, 

with our local Innocence Project is hair microscopy. So here is a forensic discipline in which we now 

realize that claims that have been made in the past are not really scientifically valid. I’m sure you all know 

this better than I do, but as I understand it, it’s not that you can’t look at two hair samples and determine 

whether they could not have come from the same source, or whether they could have come from the 

same source, but we used to believe, and we had experts testify, very specific numbers, likelihoods, 

probabilities, and those are not based on valid science. Which means that there is the possibility that 

people have been convicted based on jurors hearing evidence, again, not unlike that alluded to by the 

prior speaker where very specific probabilities were given and they based a guilty plea on that 

assumption. And we now know that to be false.

So I think you, you know, that’s another role of the conviction integrity unit is to look not just at new 

evidence as to a particular case as it might come up, but also to look at whole categories of evidence, 

particularly in the forensic field.

I do think there is something to be said for special training, so whoever you have running your conviction 

integrity unit, it probably should be somebody who, either through training or experience, or ideally both, 

is aware that the role that, for example, implicit bias can play in our system. Or confirmation bias. Or 

tunnel vision. Or the risk factors. We now know, because there’s been a statistically significant number of 

known wrongful convictions, and in fact it was the federal Department of Justice that really took the lead 

over a decade ago in trying to look at what were the common factors in these wrongful convictions. And 

by looking at those you could see some known risk factors.

So, for example, eyewitness identification. We now know that people can in good faith believe they are 

identifying the person who assaulted them, but be wrong. That even leaves aside the people who are 

lying, which, of course, happens, too. But we now know that eyewitness identification cases are ones 

where you want to pay special attention for a source of error.

Similarly, as I said, a lot of the forensic evidence cases you have new types of evidence that you need to 

look at. And as I said, other things like implicit bias, or actual bias, so I think the person who is looking at 

these cases, or the committee looking at these cases, should have some education into those particular 

risk factors.

The other challenge that I want to point out is if you recall that list I started with where you start with the 

arrest, which is supposed to be supported by probable cause, going all the way through an appeal and a 

habeas, look, by definition, everyone who has been convicted is going to look really guilty. Right? I mean 

it’s unlikely that someone is going to be convicted and there was no good reason to think that they were 

guilty. And this is a fact that just has to be fronted and acknowledged because if you don’t front and 

acknowledge that, what ends up happening, and I’ve seen this happen with other prosecutors’ offices that 

look at it, is that if you only are going to look at those innocence cases that kind of look like a made-for-TV 
2 April 10 DOJ Part2

15

movie where you can just see from the beginning that this is obviously an innocent man who has been 

railroaded, you’re probably never going to find any. Because for it to have gone through so many stages 

in the system, there is probably going to be at least a surface appearance, and often a very deep 

appearance, that the person is actually guilty.

And in fact if you go to the Innocence Project’s website, which is an excellent website and has really a lot 

of extremely moving and heartbreaking case studies of people who have been wrongfully convicted, but 

one thing they have in common is usually it’s multiple errors. In other words the system broke down in 

multiple ways. My observation is the system is pretty good at correcting itself if a single thing goes wrong.

Right. So if the one thing that goes wrong is you have a bad prosecutor. Well, if you have a good defense 

attorney, and you have solid forensic evidence and good police work, that’s probably going to be 

revealed. What you often find, or I have often found, and I think the data suggests this, is where you end 

up with wrongful convictions is where you have multiple things go wrong in the course of the trial.

But what that means is when you’re starting to look at this after the fact, you have to bear in mind that you 

are looking for the outlier case, not the typical case. You’re looking for the case where despite the person 

looking guilty, they are still innocent. You really have to approach this with a very open mind and willing to 

acknowledge that things really might not be how they appear.

And then I do want to point out another challenge is that everyone has, at least a motive to lie. You know, 

if you’re, I don’t know, you had a speaker from one of the Innocence Projects before, but I think most of 

them will tell you they get a lot of people asking for their services and many of them are not, in fact, 

innocent. In other words, if you are serving a long sentence, you don’t have a real disincentive for telling 

people that you are innocent.

Now, similarly, I don’t want to point one set of fingers. I mean witnesses can have motives to lie.

Everyone can have motives to lie. So you’re dealing with a problem in which everyone kind of looks guilty, 

there’s probably multiple things that went wrong, a lot of people have motives to lie if not everybody, and 

you still have to somehow figure out the truth about what’s going on, often about a case that’s very old.

So let me very quickly give you a story of two cases to illustrate this.

The person was actually the very first case that I worked on. And the crime itself was horrible. It was the 

rape of a young mother. It happened in her home. She was asleep in her room. Her child was asleep next 

door when there was an intruder late at night. She survived, her child survived but she was sexually 

assaulted. And she did what she was supposed to do which was to protect herself and her child as best 

she could, and she was successful at that.

The perpetrator, after raping her, stole some knickknacks from a bedside table. Small objects without a lot 

of value but they were important to her.

Well, eventually someone is arrested and he is in possession of those small stolen items, and then he is 

picked out of a lineup as the perpetrator. And he claims his innocence, and we, you know, I get the case, 

as did the Innocence Project. And you think about it, this was a case in which the defendant, he was 

convicted and sentenced to life. But it was a hard case, right? There were a lot of red flags here. First of 

all it was an eyewitness identification by a victim. So as I told you, that’s a – that’s an area in which we 

know historically there can be errors.

It was in a bedroom which just had light from a small bedside table, so the lighting was poor.

There’s something known as weapon focus which essentially means, it makes a lot of sense when you 

think about it, if somebody has a weapon, you tend to focus your attention on that weapon because that’s 

your threat. You don’t focus your attention on details that might help you identify the face later because 

the face is not really relevant to your survival at that moment.
2 April 10 DOJ Part2

16

It was also cross-racial. Honestly it’s a long time ago so I can’t remember the races of the different 

people, but they weren’t of the same race. And we know there’s just a higher error rate when people are 

trying to make identifications across race.

Fortunately in this case, though, it’s years later, and we found DNA from the perpetrator, some male 

semen on bed sheet. And so we were able to test it, and what we found, a decade later, that the guy was 

both a rapist and a liar. In other words he was making these innocence claims but the DNA proved that it 

was him.

Now I sometimes kind of give a rhetorical multiple-choice question as to what happened next. Do you 

think there was a media firestorm on how the DA’s office got it right? Or there was an award for the 

original trial deputy? Or that I got to go on Oprah and talk about this incredible work we did in proving the 

right guy was guilty? And of course the answer is the last one, nothing. So, you know, that’s just what 

happens when the system works the way it’s supposed to happen.

But now, years later, I had another case, and this involved this woman here, Michelle Bullington and 

another guy named Kenneth Foley. And they were arrested and charged with an armed robbery. The 

victim in this case, he was sleeping at his office, and he heard a disturbance out back and he saw two 

people trying to break into his vehicle to steal the radio. And he goes out, he has a handgun, he confronts 

them with the gun. Then he told us this woman, Michelle Bullington, pulled out a handgun and threatened 

him. And they basically had a standoff. And he was able to get them to go away. They flee in their 

vehicle. They – I’m going to put it back there for the picture. He called the police, and then essentially he 

got a license plate.

They found that vehicle was stopped 24 hours later with the same license plate with a moving violation.

They do a photo lineup of the driver of the vehicle with the victim. The victim picks out the driver as the 

same one who had robbed him. They interviewed that driver, who said he didn’t know what they were 

talking about but he had borrowed the car from this woman here, Michelle Bullington, who was in the car 

as well. They then do a photo lineup with her. She’s also picked out of a lineup. It goes to jury trial, and he 

is convicted. And she, this woman, Michelle Bullington, she gets, I think, about a five or six-year 

sentence, and he ends up with a life sentence. The reason it was a life sentence is it was an armed 

robbery and he was a three striker.

Years go by and we eventually get the case, and I got the case from a local defense attorney who 

essentially just said, look, you know I’ve tried a lot of cases, and I’ve got to tell you this case just sticks in 

my craw. I just feel like something went wrong and I think you got the wrong people. And we think it’s this 

other person. And he gave the name of another person, a guy named Luke. And he said, and we told 

everyone about Luke at the time.

And so I go back, and I pull the trial transcript, and they do tell everyone about Luke at the time. In fact, in 

the very first interview this woman, Michelle Bullington, said hey this other guy – first of all she said there 

was no gun, okay. And secondly, you’ve definitely got the wrong guy. It wasn’t Kenneth Foley, it was this 

other guy, Luke, who was in the car.

Well, you know what? They did a lineup – not a lineup. They did a live show up in the courtroom with this 

guy Luke along with Kenneth Foley, and the victim picks out Kenneth Foley again and says it wasn’t this 

guy Luke.

Well it turns out we think the victim just got it wrong. Well maybe not just got it wrong. We think that the 

victim might have just gotten it wrong, but was also, perhaps because of pride or other reasons, was 

pretending to a certainty he just didn’t have.

There were a lot a steps in investigating this case, but the one that sticks in my mind the most is that we 

ultimately – we thought about it. We looked for security cam footage. We looked for DNA or fingerprints or 

any forensic evidence that might help us, and we came up with a blank on everything.
2 April 10 DOJ Part2

17

So then I thought to myself, well why don’t we interview this guy’s coworkers and friends and see what he 

was saying at the time of the trial. Maybe he told them something he didn’t tell us. And we found the 

accountant, the bookkeeper for his business. And so we go to interview her. Remember, this is many 

years later. And we tell her we’re from the DA’s office. I have an investigator who is a peace officer. I 

come there as well. And she says, you know I always thought I would speak with you guys, and then she 

pulled out this huge file. And we asked her, what’s this file, what do you have here? And she said, oh this 

is a second set of books that I always kept. These are the true books, these are the true files, not the 

fraudulent accounting that I always filed on behalf of this victim and his business for tax purposes.

So then we ardently told her we weren’t investigating any fraud but we were talking about this case. And 

essentially we got a very different picture, and we essentially had her telling us that at the time of the trial 

that this victim, who had presented himself as very reluctantly testifying, was, in fact, very excited to be 

testifying. And thought that he was, you know, this was the most important thing that he had done. He 

was very proud about it and felt like he was breaking up a major crime operation. And told her all sorts of 

stories that weren’t true about how he got awards, and had been, you know, kind of been made an 

honorary member of the police.

So we were able to discover that he had told a whole lot of things that weren’t true at the time. We 

eventually interviewed him, and at that interview he admitted that he had been lying about seeing a gun.

That he had actually told them that he had a gun because he was worried that we would arrest him or 

charge him for his possession of a gun if we found out that they didn’t have a gun to justify his pulling it.

Well, this is that category of case where the case just collapsed. I mean really what happened is we just 

didn’t believe him anymore. Can I prove that’s absolutely true? No, there’s no DNA evidence to exonerate 

these people or to prove they’re guilty. But the bottom line is it’s a one-witness case and this witness we 

now believe was lying.

So we wrote our own motion and tried to get these people out right away. Ms. Bullington has really done 

well. You know, she’d been an addict and really living a lifestyle that was, you know, she was going to 

end up in prison or dead probably. She has completely turned around and has had jobs with security 

clearance. Got her college degree.

And Mr. Foley, he was actually in isolation serving a life sentence and then learned he was going to be 

not just released, but exonerated and released all in one day. So I think that was, you know, obviously 

very important to him.

So what is the conviction integrity unit? You look at these two sorts of cases. I think essentially it’s a risk 

management – or risk assessment model. I think it’s not as helpful to look at these things as right or 

wrong or purely as moral issues, but to recognize we have a system. And like any other system for 

asserting the truth, it’s subject to error. And part of what we try to do is reduce those numbers, reduce 

those sources of error.

So, for example, if you have forensic science that you know is questionable, you want to err on the side of 

not using it. If you have a forensic science that you know your testimony about it can be misleading, you 

want to try to make sure that you use testimony and train people so that they can testify about it 

accurately.

And then, of course, you want to do things like have body cameras, record subject statements, do 

eyewitness identification in a double-blind manner and record it so that you can do the best you can to 

enhance its accuracy.

And as you can see from that, there’s going to be two components. You’re going to have a reactive 

model. That’s dealing with crises that come up. For example, we have a crime lab. I think they do a 

phenomenal job, but you’re always going to have problems. And you want to have a conviction integrity 
2 April 10 DOJ Part2

18

unit that can figure out what to do when problems arise. That you’re not just reacting in a negative or 

panicked way.

And similarly, you want to look at cases that come to your attention like Michelle Bullington’s case.

You also want a proactive model so that you are designing policies of how you are going to pursue justice 

in a way that reduces the risk of wrongful conviction. And you want to train to try to implement those 

policies and to educate your prosecutors and your community and your forensic scientists and others on 

what risk factors are most likely to lead to a wrongful conviction.

The reactive element I think I just told you about, so I think I’ll just go through here. Those are postconviction assertions of innocence. Actual cases. And your crises will be things like forensic issues that I 

alluded to earlier, hair microscopy. If you have allegations of prosecutorial misconduct, and so on.

And your proactive element, as I alluded to before. You want to try to record your suspect statements, for 

example. You want to have a Brady policy, which I don’t know how much you’ve touched on, but that’s 

making sure that the policy and the methodology to make sure that all appropriate evidence, which should 

really be just about all evidence that we have, is turned over in a timely fashion to the defense.

You want a collateral consequence policy, and that’s essentially so that you’re making sure that your trials 

and your truth finding is not being irrationally distorted by consequences that have nothing to do with 

justice in that case.

We looked at eyewitness identification.

Prosecutorial misconduct.

Training.

So I’m just giving you an example, these are all things we’ve looked at.

Body cameras.

And then finally I’m just going to touch on what are some of the rough numbers. I believe right now we’re 

up to 349 known DNA exonerations, but this, of course, is going to be a smaller number of the total 

exonerations. As I pointed out, in my county alone we’ve had five exonerations, only one of which has 

been a DNA exoneration.

There’s Professor Gross has come up with a study finding more than this using only the official definition 

of exoneration. And in 2012, he came up with a larger number looking through newspaper articles and so 

on.

His definition of exoneration was anyone who was eventually relieved of the legal consequences of a 

conviction based on new evidence. That’s essentially it.

Well, you can see from this that this is going to be over-inclusive and under-inclusive. There are going to 

be people who are actually innocent who aren’t captured by this data. And on occasion you’re going to 

meet people who are actually guilty but because there’s new evidence that arises, and, you know, you 

can’t retry the case years later, perhaps the victim has passed away, they are going to be deemed 

exonerated. That’s why I think it’s important not to get too caught up in the exact numbers but just to use 

these numbers to guide your decision making an policies.

There was a debate that went out a little while ago, and I’ll kind of finish up with this, that was looking at 

these numbers, and essentially the head of the NDAA at the time, which is the National District Attorneys 

Association, he made an argument that I don’t really agree with even though there is some real truth to it.
2 April 10 DOJ Part2

19

He said, look, let’s take these known number of exonerations, and assume that we’ve only found one out 

of ten. So, in other words, if there’s 340 known exonerations, let’s assume that means there’s 4,000 total 

innocent people.

And then what he said is he divided the total number of exonerations from the total number of convictions 

over the same period of time, which is about 15 million, to get an error rate. And the error rate, if you look 

at it that way, is really small, right? It’s this very small number, 4,000 over 15 million, so you can see 

whatever that comes out to, .0267%. Which means you’re talking about a 0.27% error rate. Which means 

you’ve got a very, very high accuracy rate.

The problem with that theory is twofold. First of all, we do a lot of trials. So let’s even assume for a 

moment that that’s correct, and I think there’s reasons to be concerned about that number. But let’s even 

assume that that number is correct. That means you have three innocent people for every 10,000 

prosecutions. Well we do close to 40,000 prosecutions in my office alone in a year. So these very, very 

low error rates, and I do think that ultimately the error rate is low, but no matter how low you go, when you 

have this number of prosecutions, you can see you’re going to have a significant number of wrongful 

convictions. And, of course, let’s remember this number is probably – the problem with that (inaudible) 

estimate. First of all your multiplier could be wrong. Secondly, this low estimate looks at the error rate for 

all prosecutions instead of your ever rate for all cases that go to trial. And you’ll notice people who have 

been wrongfully convicted disproportionately have gone to trial which makes sense. They’re not so likely 

to plead guilty if they’re actually innocent.

Conversely, your total number of prosecutions, in, you know, upwards of 90% of them, guilt is never 

seriously contested by the defendant. It’s really more about sentencing.

Now some innocent people plead guilty to things they didn’t do. In fact one of my exonerations comes out 

of just that pattern. But you really can’t look at your total number of wrongful convictions and look at all of 

your cases as opposed to looking at the cases that go to trial. And I give the example of the Pinto –

remember the car? You know, when you had a car that had a disproportionate chance of blowing up 

when it was rear ended, you don’t want to look at the total number of car acci - you know, you don’t want 

to look at all cars and determine how likely they are to explode when rear ended, you have to look at the 

specific risk factors. And that would be true in this case, too.

If you look at error rate in felony trials, it’s a much smaller number – or higher number. I’ll just get to the 

end. It brings you more like one false conviction for every 200 trials. And to me that means you can 

simultaneously believe that you have a very high accuracy rate in your system overall, a much lower 

accuracy rate for cases that actually go to trial. But in any case, you’re talking about a lot of potential 

people who are all, as we saw from our prior speaker, human beings who essentially have been 

victimized by our criminal justice system. And so justice alone calls for doing something about it.

So my final suggestions are when I’m talking to prosecutors is really focus on discovery, making sure 

you’ve turned all the documents over that you have, and that would apply to post-conviction 

circumstances. To really do DNA when you can. To try to get to the merits of your cases. In other words 

not to try to pass these cases off by some sort of procedural bar but to really try to reach the merits of 

your case. And to collaborate whenever you can. And looking at a risk management model. Knowing that 

you’re never going to know for sure what happened but you can take steps to try to reduce the risk of 

these happening.

So, okay, like I said, a little unnerving. Because of technical reasons I can neither see anyone nor hear 

anything, so I recognize there’s a chance I’ve been talking to myself in my office for the last 20 minutes.

But I’ll close this off. I will leave my email and my phone number, so I encourage anyone, I’m always 

interested in talking about these issues. Feel free to contact me, either online or later. And I am hopeful 

that after I stop talking I can hear again, so I can listen to the other speakers and then be part of the 

discussion afterwards. But thank you very much.

UNIDENTIFIED FEMALE SPEAKER: Thanks, David. We could hear you.
2 April 10 DOJ Part2

20

DAVID ANGEL: I’m glad to hear that.

PETER NEUFELD: So what I’m asking everyone to do who is remaining is to do a little self-censorship 

and try and abbreviate your remarks if you can.

Next up is Ms. Sarathy-Jones from the National Institute of Justice, the Bureau of Justice Assistance, so 

maybe that’s (inaudible) OLP instead – not OLP but programs.

UNIDENTIFIED SPEAKER: (Inaudible).

PETER NEUFELD: Thank you. Thank you, Gerry. Acronyms go early.

PRIVA SARATHY-JONES: Hi, everyone. My name is Priva Sarathy-Jones, and hopefully – I’m pretty sure 

I’ll be brief.

So I’m here today to talk about BJA, the Bureau of Justice Assistance’s wrongful conviction review 

program. This is a program we’ve had since 2009 with the primary focuses of, one, providing high-quality 

and efficient representation for defendants in post-conviction claims of innocence, and then secondly, 

whenever possible to actually get – to identify the actual perpetrator of a crime.

And so as I say this is a program that we have had since 2009, and we’ve made awards directly to 

organizations such various Innocence Projects or law schools with clinics that do post-conviction relief 

claims as well as public defenders’ offices that may have post-conviction relief units as well within the 

organizations. We’ve made those awards directly to those organizations as well as had separate awards 

for training and technical assistance in this area to provide support to the field as a whole on issues of 

post-conviction relief.

So within the program itself, the type of things that we’ve funded in the past have been, you know, staff 

salaries, intake coordinators, investigators, a lot of resources that the organizations that do postconviction relief are really lacking and need this type of assistance with.

We do evaluation and litigation claims of innocence and cases of potentially flawed witness identification.

They can look into cases that that may be the claim of innocence.

We allow them to look at cases that they are looking at as a result of forensic evidence as well.

Help in defraying costs.

Case management systems as well. A lot of times, as David sort of spoke to you, the issue is identifying 

the right cases to pursue, and that I just recently went to the Innocence Network convention and really 

sort of realized that that task in and of itself is a very daunting task when you’re getting these pleas and 

you’re really trying to sort out both innocence claims but also ones that you could actually try to adjudicate 

through the post-conviction process.

And also recently, and most recent here, we’ve added a provision for counseling and mental health 

services for individuals that have been exonerated as a result of the awards that we have put out.

So that sort of covers what the program itself does and what we’ve looked at in the past. In 2010 we 

awarded the National Criminal Defense Lawyers Association, NACDL, to do an assessment of our 

wrongful conviction review program. And they have actually been funded in recent years, about two years 

ago, to do a new assessment. So this last assessment they did was 2010 so it was quite a while ago, but 

now they’re looking at sort of wrongful conviction review grantees since 2010 and seeing what the results 

of these awards that we’ve been making, what does it actually serve or result in a tangible way when it 

comes to wrongful conviction.
2 April 10 DOJ Part2

21

So in the 2010 assessment, which we have information on, as a result of the funding from – I believe they 

looked at solely the 2010 and possibly the 2009 awardees as well – there were 26 exonerations that 

resulted directly from that funding, or directly or indirectly as a result of that funding, and there were over 

20 actually perpetrators identified as a result of that funding as well.

Almost 13,000 new cases were reviewed by grantees as a result of that funding, so they were able to 

really look into some of the cases that they were getting.

The other thing that I want to mention is that we, through the training and technical assistance award that 

we have with NACDL, one of the biggest things in addition to the assessments that they’ve done and they 

are planning to do in the future is they do a lot of trainings and make a lot of resources available to the 

field for various organizations or individuals or private offices who take on these cases and need sort of 

training in how you actually adjudicate post-conviction cases and knowing that each case is going to be 

based on a different reason for why you may be pursuing these claims.

So just to go over really quickly, we have some trainings available that can be accessed online on bite 

marks, examining the microscopic hair comparison reviews, the challenges of flawed forensics, things 

that we’ve spoken to earlier, and so how defense counsel can really be trained. This is another aspect of 

that. We’ve talked about judges, we’ve talked about prosecutors, but there’s also a piece of this that are 

the defenders themselves being trained and have the proper resources and tools to actually properly 

adjudicate and defend their clients as well.

There’s a long list of past topics as well that I won’t go through, but that is something that we’re making 

available online. We actually have not had a wrongful conviction review website, but we are now building 

one out, so all these resources will be available directly through the DJA website hopefully in the next few 

weeks so that we can direct people to – and defense attorneys and stakeholders in the field to some of 

these trainings and resources and publications that have been put out through these grant awards.

And the other thing that I will mention now is that we are also now looking at wrongful conviction review 

and where it’s heading in the years ahead in terms of what we may be focusing on, the roles of conviction 

integrity units in the wrongful review process. I think David spoke really well to this, but, you know, there 

are multiple players all which have significance in this process and we all play different roles in this 

process. And while some of the clinics and public defenders’ offices and NIST and organizers (inaudible) 

have been great in pursuing this work, there’s another piece of this which is the prosecutors are doing to 

look at this work to identify cases as they come to them to be opened, as David has said, to either 

identifying themselves or could the same scenarios he gave with a different prosecutor involved turned 

into someone saying we’re not going to look at that case, we’re not going to review this just because your 

witness seems a little sketchy now. And if a defense counsel was the one who presented that, it may not 

have been as strong an argument as when the prosecution themselves identifies that as the issue.

So those are sort of the areas that we’re looking to moving forward and seeing what the role of those 

prosecutors and defense attorneys are and looking at collaborations between prosecutors, defense 

attorneys, and law enforcement as we look at both adjudicating wrongful convictions after they happen 

but also preventing them from ever happening in the first place. And as Mr. Harward has said, this is a 

wrong that can’t really be undone, so part of the objective here is to not have them happen in the first 

place.

So, thanks.

PETER NEUFELD: Ms. Sarathy-Jones, thank you. That was great.

Our next speaker needs no introduction, is our own Commissioner from the National (inaudible) of 

Justice, Gerry LaPorte.

GERALD LAPORTE: Thanks, Peter.
2 April 10 DOJ Part2

22

Can we have the presentation?

So I’m going to talk about NIJ’s post-conviction DNA testing program. This program I have a lot of love for 

because I managed it for almost four years. And started back in 2008, so I got it in its infancy. I have 

watched the program grow significantly.

My co-presenter here, Dr. Greg Dutton, knows the program very well. I handed it off to him a few years 

back, and he’s continued to evolve it as well, too. He’s here to handle any questions if we get into the 

minutiae of the program. I can certainly talk about the overall aspects of this program.

So, of course, the usual disclaimer. Slides. I’m not allowed to give any opinions, so if you hear them, 

they’re my opinions not those of the Department of Justice.

The program, this is just – the current solicitation is out right now. It was posted on March 23, 2017. It 

closes on May 9, 2017. But this – basically the program is to provide funding to assist jurisdictions in 

defraying the costs associated with any kind of post-conviction case review, evidence location, testing of 

DNA evidence in violent felony cases where that testing may show that the results – where the results 

may show that someone was actually innocent.

Really this kind of – the simple idea of this program and how it differs from BJA’s wrongful conviction 

program, is our program is for very four specific purposes. One of them is to identify any potential postconviction DNA testing cases where you could potentially show actual innocence.

Funding can also be used to review appropriate cases to identify those in which DNA testing could prove 

the actual innocence of the individual, so it’s very focused on finding evidence.

And then locating evidence. And you’ll see I have some numbers here. Locating evidence, as you might –

I don’t know how you would perceive this – but is not an easy task. We spend a lot of hours and we 

spend a lot of money on just trying to find evidence to begin with.

And then the final purpose of this grant is then to perform the DNA analysis of any biological evidence 

that’s found.

So at that point in time the program stops, so we don’t cover all of the other things that happen, which is a 

lot of stuff, which could be the investigative part. The prosecu – or not the prosecution but all of the legal 

things that take place afterwards.

The program itself, so that gives you an idea of the number of awards and the amount of money that 

we’ve funded each year. So since 2012 we’ve received a $4 million appropriation. We generally give all 

that money out. So you’ll see that those numbers don’t add up, at least from 12 to 16, don’t add up to an 

even $4 million, but that’s because we don’t actually get $4 million total. It’s appropriated, and then 

there’s administrative expenses and so forth that are taken off the top.

But we fully get – we get lots of applicants for this, so there’s quite a bit of demand. But since 2008, we’ve 

awarded a little over $40 million.

So here’s kind of – here’s the numbers that are I guess somewhat interesting. Since 2008, we’ve 

provided funding to review 73,522 cases. And then of those there’s been evidence searches in 29,000 

cases. There’s biological evidence in 25,000 of those cases. DNA analysis was conducted in a little over 

1,700 of those to analyze almost 3,000 pieces of evidence. They had 233 CODIS uploads, and then 52 

CODIS hits resulting in 39 exonerations.

So one thing I do want to be very clear with in this data is that do not interpret 73,522 minus 39 as rightful 

convictions. Okay, so there’s a lot of stuff that goes on in these cases. And we try and do as much as 
2 April 10 DOJ Part2

23

possible to fund whatever we can to review these cases. But a lot of times evidence is just not found, so 

there’s no evidence to be found at all.

We’re starting to keep track of some metrics of when we do get a CODIS hit, and then that CODIS hit 

actually matches the person that has been convicted. So we’re still working on some of those numbers.

And then as you’ll see I’ve got an asterisk for FY14 and 15 and that’s because those are still open 

awards. So that’s still all ongoing activity. So those numbers will grow over time.

This is kind of an interesting slide. If you look at the total hours in reviewing cases that we’ve funded, it’s 

been 325,000 hours, which is approximately 37 years of reviewing cases. And hours locating evidence, 

101,000, so that’s 11.5 years spent just trying to locate evidence from these cases. So you can see there 

is a huge effort that goes into all of this.

So we’ve had 39 exonerations in 12 states. Those are the list of the exonorees that we’ve at least helped 

fund in some way or another. And we generally – so what happens in some of our cases is although the 

focus is on DNA, every once in a while, a case will – there won’t be any DNA located but there will still be 

an exoneration once we start funding the case. So you’ll see some asterisks next to those individuals.

They were exonerated but it wasn’t because of DNA in our case, but it started with our case. Or with our 

funding.

There’s a map of all the states that have received or currently receive post-conviction DNA testing money.

So we had Idaho last year. A couple of new states. Puerto Rico was one of them as well, too. We’re 

constantly trying to expand this program so that every state could potentially, at least at some point in 

time, receive money to take part in this program.

And then generally in terms of application demographics, so we get a wide variety of things. You can see 

just the geography up in the top right-hand corner is pretty well evenly dispersed amongst the U.S.

I will talk about our peer review process. So our peer reviewers, generally we put together a panel of a 

prosecutor, defense attorney, a judge, and a DNA expert. There’s a lot of people in this room that have 

peer reviewed for this program as well. I’m not going to name names, obviously. So we try to keep it 

balanced that way as well, too.

And then finally there’s just some – we have a lot of information on the program. We have all of the 

stories of each of the exonorees on our website as well, too. And then we have some other information on 

post-conviction DNA testing.

And that’s it.

UNIDENTIFIED SPEAKER: (Inaudible).

PETER NEUFELD: Gerry, thank you very, very much.

Our next speaker is Executive Director of the Innocence Project, Maddy DeLone.

MADELINE DELONE: Okay, like magic it came up. That’s good because I’m doing nothing down here.

UNIDENTIFIED SPEAKER: (Inaudible).

MADELINE DELONE: Hi. So I’m Maddy DeLone. As Peter said, the Executive Director of the Innocence 

Project, and thank you, first of all, for inviting me here, and it’s really great to be on the panel with the 

other few people.
2 April 10 DOJ Part2

24

And from the Innocence Project, just also to thank all of you for the extraordinary work you’ve done over 

these last three years. A number of people from the Project have watched the work very, very carefully.

We’ve contributed comments when we can. We think it’s an unbelievably important institution that helped 

really for the first time, as has been said a number of times today, bringing a lot of key stakeholders 

together and adding the very critical addition of independent scientists who have deep expertise in their 

fields, and I think it’s really through a conversation like this in whatever form it gets to take, and I guess 

this exact one is now off the table or appears to be. We hope that there will be some continuation like this 

because otherwise I think we work more back to where we were before and we didn’t get as far in that 

prior period. So I just thank you for showing us all, showing the country, that there is a conversation worth 

having and progress that can be made.

Our mission is up there. It’s to – we have two parts of our program. One is to free innocent people who 

remain incarcerated. And to bring reform to the system responsible for their unjust imprisonment. And our 

reforms, all the reform work we do, are focused on preventing the wrongful convictions from happening in 

the first place. I think you heard David Angel say that’s where the conviction review units go. And, in fact, 

I think that’s where this Commission starts, which is what do you do on the front end to make sure that 

later on you have not, in fact, incorrectly contributed to the conviction of an innocent person.

I’m going to do three things, and I’m going to try to do them very quickly. First talk a little bit about the 

number of people we know who have been exonerated in the United States since 89, so I have an update 

on David Angel’s numbers. A little bit about the contributing factors that lead to the wrongfully convicted 

whose convictions were later overturned by DNA, so that’s the Innocence Project’s DNA list. And finally to 

talk about what the Innocence Project sees as the advances in forensic science over these last three 

years as the Commission was first instituted and to suggest some additional areas of work, which some 

body, like this one, or perhaps the individuals who have dedicated so much time – this will work? Okay –

dedicated so much time to this task will do in different capacities. It would be a shame to have all this 

good work and energy and information gathering lost.

Now? No. Nope. Okay. Great.

So this is the National Registry of Exonerations which was started at the University of Michigan has now 

recently relocated to UC Irvine. But the National Registry keeps track of every post-conviction exoneration 

that is after the stage of the appeal has been exhausted that has been found in the United States since 

1989. 1989 is the first recorded year of a post-conviction DNA exoneration, and so the Registry decided 

to start their count in that same 1989 year.

So the bigger number, the 2,008, which is actually 2,011 today, it was 2,008 on Thursday when this was 

submitted, is the number of people in the country whose cases have been revealed who are innocent and 

convicted, and the smaller number, this 349, is the number of post-conviction DNA exonerations that the 

Innocence Project counts on our list, the list that we keep. And our definition of a DNA exoneration is one 

where the new DNA evidence is critical and central to establishing the innocence of the previouslyconvicted person. And they are also exonerated by the same definition that David raised earlier.

We know that, and there have been some discussions in past presentations, that there are some 

differences between the way that the Innocence Project categorizes cases and contributing factors that 

cause wrongful conviction and those that are counted by the National Registry. We continue to work with 

the Registry to try to harmonize that data so that it won’t be confusing to people, although we certainly 

encourage people to call us when they are confused because there might be different data on each site 

and we can – the easiest thing for us to do is to go down to the source documents and explain to you why 

we designate something as one sort of problem or another.

The DNA exonerations, as has been said several times, there are 349 of them to date. They come from 

37 states. They represent 20 people who were originally sentenced to death. They include 37 people or 

about 11% of people who pled guilty. During the exoneration of the 349, there were 149 people who 

actually committed the crime that were identified, the people who committed the crime then our client 

served time. And those 149 people went on to commit an additional violent crime, we have documented 
2 April 10 DOJ Part2

25

147 convictions, so there are probably many other crimes that were committed, but these are ones for 

which there were convictions. So I think the point that David made and Keith made earlier is just this is 

also of public safety importance. Getting it wrong, not being as good as we can at getting it right, has two 

profound implications every time. One is the innocent person goes away, spends time in prison, is lost 

from their family. And the other is the person who committed the crime goes undetected and often goes 

on to commit additional crimes. Two reasons to work very, very hard at the front end to get it right.

DNA. The exonorees spend an average of 14 years in prison. On average they go in at 26-1/2 and get 

out at 42 years of age. This is a distribution of exonorees by race. You can’t see it. It’s 30% are white, 

seven percent Hispanic, 62% black, and one percent Asian among the 349. Clearly men and women of 

color are overwhelmingly represented – disproportionately represented there. And they spent the good 

portion of their young adult lives in prison.

The contributors to wrongful conviction. So these are the categories we count. These are the patterns that 

we see time and time again. It’s important to remember that these are DNA exonerations, so we are often 

talking about sexual assaults or sexual assault murder. There are some other kinds of crimes in there, but 

that’s the predominant core. And it’s important just when you compare it to the contributing causes in the 

broader sample that’s registered by the Registry that this is in part because of the kinds of crimes that 

were here. It’s also the kinds of crimes where biological material that can be identified with the perpetrator 

to the crime are likely to be found, sexual assaults and sexual assault murders.

You see the 71% of the cases, of the wrongful convictions, include misidentification by at least one 

eyewitness. Forty-six percent of them misapplication of forensic science. Thirty-two percent false 

confessions, admissions or guilty pleas. Most of those are false confessions. And 17% incentivized 

informants.

In the area of misapplication of forensic science, which is really the material that you have been working 

on, we divide the kinds of contributing factors into five sorts of data, and sometimes any individual case 

will give you two examples. One is just the issue of the unreliable or invalid forensic discipline. I think the 

bite mark in Mr. Harward’s case is such an example.

Insufficient validation of a method is another area.

Misleading testimony. We heard a bit about that today. Overstating the evidence to enhance probative 

value. Downplaying exculpatory evidence. Or failing to include any statement about the limited use of the 

technique.

Sometimes there are multiple kinds of science in a case, and we count the case as a problematic science 

case even if an analyst got one technique right and testified appropriately and another one and testified.

So somebody could testify appropriately to serology and inappropriately to hair, and we would identify 

that as a misapplication of science for the purpose of a contributing factor to the wrongful conviction.

There are some mistakes, and those are also included there. I think an example is a DNA sample that 

was switched by the police department and eventually it was unraveled and the police helped but the 

person who had the switched sample was identified with the actual crime scene sample went to prison 

while the other person didn’t. Eventually it was reversed. 

Or misconduct. And there are some examples of forensic misconduct throughout the cases.

Just to also point out that people look at our website and see these numbers, and then when they actually 

try to identify what’s the harm – what is the actual forensic problem in the case, the narrative descriptions 

that we put up on the website which are abbreviated for purpose of keeping the attention span of the 

average reader sometimes leave out those details. We will try to go back in because I know it causes 

some people concern to read the profile, see it’s marked as a misapplication of forensic science and not 

be able to see what the misapplication was. So we will work through the cases to try to identify them. But, 
2 April 10 DOJ Part2

26

again, we ask that you please call us and ask your questions and we can provide detailed information, 

transcripts, anything else we have on these cases.

And just look at the time. Sorry.

I think – wrong way. So this group has been dedicating a lot of its time and important work to improving 

forensic science. We think about the work that’s being done and needs to be done in sort of three areas. I 

think you divide them differently but bear with me. For us it’s the issues having to do with sciences and 

standards, ideas of validity and reliability of the discipline and the setting of standards. Testimony which 

includes both report writing, although you might include that up above but also testimony in courts. And 

the basic idea of testimony is making sure that you’re taking the scientific principles and the work that 

you’ve done and translating it to the science to language that the police, the prosecutors, the defense 

attorneys, judges, and the juries can understand and that capture the important elements of the scientific 

test or assay that you have that you have done.

And finally, there is work to be done and work that you’ve addressed in addressing errors when mistakes 

occur. Things like the conviction integrity unit that David discussed, that’s another way of addressing 

error. Finding those mistakes. And also the federal grant programs have been incredibly useful in finding 

the errors and then at least bringing justice back to the people whose lives were taken away from them.

There’s been much progress on the national level. Just to remind us, in 2005 Congress first 

commissioned the NAS study which was released in 2009. In the end of 2013, the National Commission 

Forensic Science, this group began to meet. NIST began to take on a more robust role. The OSACs were 

developed. And finally in 2017, while all of your work was going on, the PCAST issued its report on 

forensic science.

Your document – the documents that you’ve prepared for the Commission, going through them there are 

enormous numbers of important things that you have added to the field and to help improve forensic 

science. From our humble place where we sit and look for reform that protects the innocent.

First of all, just as I’ve said before, having this multidisciplinary group here together is extremely 

important, really first, in particular having independent scientists in such strong numbers that there is 

really a voice and an opportunity for them to bring in from their fields of study the important principles that 

will help make the scientific work in this arena better for all.

In Science and Standards, NIST has implemented some technical merit evaluation and is committed to 

doing more foundational validity. There may be slowing down on that from your initial plans, just based on 

some of the uncertainty of funding, but we really hope that that work will continue. We know that NIJ is 

looking at the post-doctoral translational research programs. Hopefully they will be able to continue.

There have been documents addressing scientific certainty saying that it is – and calling for the end of it, 

and the Department of Justice labs, it offers too much, it’s more robust than what can possibly be true.

And that earlier work – in earlier Commissions, you voted on documentations, case records, and reports, 

something you didn’t pass in a second iteration earlier this morning, but the earlier document is there and 

the Committee’s work, I think, is very useful and will be useful to others.

You’ve also had strong recommendations on root cause analysis, the Code of Professional 

Responsibility, and forensic discovery. And while the Department of Justice did not adopt the 

recommendations fully on root cause analysis and the Code of Responsibility in their full form, the 

recommendations themselves were useful for others and as a strong foundation for the work ahead. As 

we heard this morning, the discovery recommendations from this body were in large part adopted by the 

Department of Justice and should make the discovery, at least in the federal system, more robust and 

more fair, and we hope that as appropriate states will adopt these practices, too.
2 April 10 DOJ Part2

27

Finally, just a few things. I think when I wrote this slide last week we were still hopeful that this body could 

be renewed. As the new Administration seeks feedback on the next steps in forensic science, we would 

encourage anybody setting policy to include the same strong showing of independent scientists that this 

body has held.

We think it’s important that the foundational research continue. It is needed in so many disciplines, and as 

I know this body is well aware, that research is needed for technical merit evaluation that also must 

continue.

Similarly the standard setting based on strong research will be critical in this area.

There is much work to be done on the role of science in the cause of death and manner of death 

determinations. If not done by this group, we hope some of you will continue to pay attention to those 

issues.

We also believe that crime labs need the operational support to incorporate new research standards and 

practices into their workflow, and he Innocence Project stands ready to stand with them and work with 

them to advocate for appropriations for programs like Coverdell Laboratory and improvement grants to 

help make the science stronger and stronger.

The Department of Justice, I guess, hopefully is still drawing on all of you who remain committed and 

interested will continue to facilitate the development of the Ultras and for the continued – and seek your 

input on how to continue and to continue the forensic science discipline reviews. They are an important 

quality assurance mechanism, and the results of those kinds of reports are very important to people like 

Keith, no longer like Keith, but new people who are coming into the system who are facing courtrooms 

where testimony will be given and if it unfairly and unscientifically casts certainty over opinions that are 

given or made will result in more and more people being wrongly convicted in part with the contribution of 

forensic science, so I hope you will think of new ways within the Department of Justice, or new ways that 

all of you can help ensure that we get better and better at more accurate testimony and at more accurate 

report writing to ensure that these sorts of things don’t continue to happen.

We believe that the federal grant programs that Gerry and Pria describe should continue to allow innocent 

people who have been convicted to be identified and freed. That there should be a model defendant 

victim notification process when errors do occur, and we’re in the addressing error section, both as a 

moral duty, although some people don’t think it’s necessarily moral, but also then, let’s say, for a due 

process duty and a right.

There should be model policies for retrospective reviews by some portion of this body or one of these 

new groups. Labs all over the country need guidance on how to do these reviews well. There are 

requests that come to us asking for help to do them. We are not the experts. We would feel much better 

about a set of reviews that were sort of – and procedures that were developed by a body like this, but 

they are sorely needed if we’re actually going to ferret out error and make sure we understand the 

implications of those errors.

And finally, I think there is a need, and you haven’t gotten to it, and I think, you know, Keith’s case and 

the data Peter gave at the beginning about the fact that bite mark evidence, despite every scientific body 

that’s looked at it, and many forensic science bodies that have looked at it, you know, continues to be 

admitted in courts. There’s yet to be a court in the United States that has kept it out. The Texas Forensic 

Science Commission agreed that it was problematic and shouldn’t be used until it was more valid, but 

people are going into courtrooms every day in this country – maybe not every day, but often – and being 

convicted in part on bite mark evidence. And we really need a body like this to help the legal system 

understand that science changes. And you can’t rely on the precedent of what you did with science A 20 

years ago to admit science A, when you understand its frailties, into the courts today. The courts, the 

prosecutors, the defense attorneys, are not as good as they could be at figuring out what to do when the 

kind of evidence they always admitted changes, when we understand something different about it. And it 

would be very helpful for a group like this to really grapple with that question. And then there should be 
2 April 10 DOJ Part2

28

ways that innocent people inside, whose convictions rested in part on that bad science, on that frail 

science, on that error-ridden, error-prone science, how they can get back into courts and try to vindicate 

those claims. So there’s something very important left to be done in that whole area of addressing error.

I think my time is probably more than up. I’m just going to do here what I do to myself. Let’s see if I can do 

it. They locked me out. It’s the very last slide, is just to leave you with the people who have spent time 

inside for things they didn’t do. each of the men is a person who was wrongfully convicted. It doesn’t like 

it. Sorry, I can’t get to it. Well, you just have to look at Keith. He’s like thousands of people. And –

UNIDENTIFIED SPEAKER: (Inaudible.)

MADELINE DELONE: They are a reminder of the fact when the system gets it wrong, real people are 

harmed. Exonerated men and women inspire us to make the system better. I hope they will be a reminder 

to you and will inspire each of you to continue this important work and to make the system more fair and 

more just and to continue to decrease the number of preventable wrongful convictions.

Thank you very much for your work.

PETER NEUFELD: Thank you very much, Maddy. With Nelson’s permission I’ll take two comments, and I 

think we’re out of time then. So does anybody have a question or a comment? No? One. Yes.

JOHN HOLLWAY: Mr. Harward?

KEITH HARWARD: Yes?

JOHN HOLLWAY: I want to apologize to you on behalf of myself and other people who have been 

involved in the criminal justice system. But I want to apologize to you not just in terms of the scientists 

and the odontologists, because it’s a legal problem as well. And we as lawyers have an obligation to 

make sure that the scientific testimony that comes through is authentic and is validated, and that is the 

purpose that I think has gathered everybody here. And the tragedy of what happened to you, I really 

appreciate your empathy with the ultimate – the other victim in your case, the woman who was assaulted.

I think that’s essential. And making sure we get the right guy in the right way is the motivating factor. And 

what I think is your story really brings out the tragedy of putting this Commission on hold. Because what 

we have done here, and I’ve only been a – I’m only here by proxy today, but I’ve been part of two 

subcommittees, and what I’ve seen is that everybody here has been motivated, not necessarily with the 

same perspective, but we’ve come to recognize each other, and respect each other’s views, and that 

becomes the foundation upon which real progress can be made. And I worry that by putting this 

Commission on hold, the next group that comes together will lack that foundation, will have to build that 

foundation, and we will lose time to help the other people like you who are incarcerated improperly, or 

worse, the people who are still to be incarcerated improperly because we have not solved these problems 

yet.

So I really want to thank you for bringing that to our attention.

With bite marks in particular, it all started with one case where there was no Fry (sp) hearing. There was 

no hearing. The judge simply said, I’ll allow it, without the hearing. Lawyers in 16 cases in 12 states then 

went with precedent on that case until the West Virginia Supreme Court said we can just take notice on 

this it’s universally accepted. And so that’s actually not just a failure of science, that’s a failure of law. And 

the organization of law, the Department of Justice, that is putting this Commission on hold, needs to 

recognize that we as lawyers tend to think that peer review means other lawyers and not go outside. And 

so the interdisciplinary nature – I want to thank all of you, it’s been a pleasure working with you. The 

interdisciplinary nature of what we have here has been really special. The ability to build the relationships 

that we have has been really special. And I hope that we can encourage the Department of Justice to be 

quick in their assessment of this and bring this group back together.
2 April 10 DOJ Part2

29

Thank you.

PETER NEUFELD: And as in life, I think the final word will come from the bench. Judge Hervey.

HON. BARBARA HERVEY: I, likewise, want to thank you, apologize to you, and thank you for being part 

of trying to fix the system. But I did want to correct Peter on one thing. In Texas six dentists have come 

forward with their lists, and notifications will be sent out on these cases at the next Texas Forensic 

Science Commission meeting, and notification has already been sent out on hair and DNA mixtures. So –

PETER NEUFELD: I think that the moral is that, particularly with the word that we got this morning, Judge 

Hervey, that this Commission will cease to exist, that all of us around the country will be turning to Texas 

to be the leader in almost every aspect of reforming forensic science. Thank you.

HON. BARBARA HERVEY: Thank you, and we will continue to share whatever we have, and you’ve got 

all kinds of brilliant people in here that are going to keep marching forward, not just the state of Texas.

JOHN BUTLER: Okay. Thank you Peter, thank you panelists. That was very informative.

We’ll take a 15-minute break and reconvene.