4 April 10 DOJ Part 4

1

JONATHAN McGRATH: Welcome back to the National Commission on Forensic Science Meeting 13, 

Day 2. Thank you for those that are dialing in through the webcast. We will get started in a few minutes.

I'm going to hand the microphone over to Nelson before we get started with our first speaker.

NELSON SANTOS: Good morning, everyone. Before we start with Dr. Speaker, I'd like to give Pam the 

floor to go over some of the amendments that were made to the summary document as a result of the 

votes we got yesterday. We had indicated that we would go back and fix those so that we can see what 

that looks like. I think John has it available for viewing. Again, unless there are issues with this, we're just 

going to move forward without a vote; I think it's pretty benign.

PAM KING: When we voted on this document yesterday, I had highlighted for the group that there was 

this language, which is now up on the board, that references the two documents that we voted on 

yesterday. As a result of voting both of those documents down, I thought it was appropriate to change the 

documents, which was what we voted on – to approve the document, with an additional change, once 

these were voted on one way or another.

The language that I've added to the document is what is below in blue. The highlighted portion, which 

says, "These two documents are currently in draft form and may be finalized by the subcommittee for 

discussion and vote by the full Commission at the upcoming Meeting #13, in April of 2017," that would be 

deleted. Then the new language, which you can see below, would be, "At the 13th meeting, neither draft 

received the required two-thirds majority vote; however, the issues each report addresses have been of 

concern to the Commission since its inception and remain crucial to the forensic science community."

UNIDENTIFIED MALE SPEAKER: So moved.

UNIDENTIFIED MALE SPEAKER: Second.

GREGORY MOTTA: I just want to ask, do we document other documents throughout the process that 

were voted on but turned down?

PAM KING: All of the documents appear in the Appendix, which has not yet been amended to reflect this

change as well; but within the document itself, yes, almost all of the—

NELSON SANTOS: Pam, I don't know that we've had one that has been voted down. It usually goes 

back to subcommittee and then readdress itself; but because it's the last meeting, it doesn't have that 

opportunity. I think it's consistent with some of the other topics that we had indicated for future to address.

So I think putting this statement in is consistent with what we've done before on other topics we wanted to 

look at.

PAM KING: And certainly the document itself identifies all other issues.

NELSON SANTOS: Yeah, I think it's consistent.

Any other concerns? All right, so that's what it will look like then finalized. Thank you.

Our first speaker goes by the name of Speaker, so this is very apropos; it is Paul Speaker from West 

Virginia University.

PAUL SPEAKER: Good morning.

First a little bit about who I am and what we're going to go through here this morning. As Nelson 

mentioned, I am from West Virginia University; and my connection to the forensic science community is 

consistent with our Forensic Science Initiative at WVU. Early on in the process, Max Houck, who was 

running a project, brought us in to be able to speak to some of the scientists with ASCLD to try to give

them a little bit of a sense on becoming an organization on their own and running things. So he asked 
4 April 10 DOJ Part 4

2

some of his friends in town to come in and speak to them about strategy, about budgeting, about a variety 

of business issues that we had; and I got invited to that, and that was my start.

Out of that Forensic Science Initiative, one of the things that became apparent very quickly was we had 

all of these people in very prominent positions running organizations but very few who had any formal 

training. Their formal training became as scientists; they were very good as scientists, but put into 

management and decision-making places.

So he, along with Dick Riley, one of our colleagues in forensic accounting, put together a proposal funded 

by NIJ, which is Project FORESIGHT. I'm going to talk a little bit about the history of the project. I'll try to 

do as little as possible, enough to keep you informed; hopefully, those who are very familiar with the 

project, not to bore you in the process; and then spend a little bit of time talking about the work that we 

currently have underway, including some results that have just finished up on a couple of the papers that 

we're working on, but to give you an idea of what this can help do for forensic science services in general.

As I said, Project FORESIGHT is a natural development out of the Forensic Science Initiative. For those 

of you that know Suzanne on the Committee, you probably have a little bit of a sense of what we're doing 

at West Virginia; but it is a byproduct of some of the efforts of Senator Byrd to be able to bring more 

things to the State of West Virginia. With the fingerprint identification facility moving nearby, we have the 

birth of the Forensic Science Initiative here.

Our Project FORESIGHT was something that followed on the heels of a project that had been conducted 

in Europe, the Quadruple Study, where we had four laboratories in Europe that got together and said:

How do we begin to connect some of the various things that we have in place here on casework and the 

details you would have on casework, but to connect that with the financials and the budgetary restrictions 

that you have in the laboratory and to be able to talk about some of the personnel things that we have in 

place?

NIJ funded this project for their fiscal year 2009 and then again for 2010, and we were able to do quite a 

bit with this. Now, the difficulty of course in the laboratories is that the systems that house the information 

on casework were generally separate systems from what would hold anything on financials and may also 

be completely separate from anything on personnel matters. So it was not a simple matter to extract data, 

to be able to get to that; and what we did is we gathered a group from 17 laboratories in North America – I 

say North America because we had two Canadian laboratories that participated from the onset of the 

project – but to come together to, first of all, define a language.

Now, for me, I'm from the dismal science, from economics; so I'm not from the physical sciences, but I do 

read voraciously. So you'll notice that in leading up to just opening this meeting, I had my kindle open; 

and I was here reading, of all things, a novel about a crime. So I do that all the time. So I thought I 

understood the language. When I read crime novels, I seem to get a pretty good sense of what we mean 

by a case, a sample, an item, a test – whatever these things were.

What we discovered when we sat around the room was that 17 laboratories, 17 different definitions for 

everything. So it took us about a year-and-a-half, and it was this dance that we would do – two steps 

forward, one step back. I got very good at it. But we came up with a language; and from the perspective 

of people in the business school, it did not matter to us what people wanted to call things, just that 

everybody agreed to call it the same thing.

So fortunately, we had the Quadruple Study to begin with; and we had, really, quite a bit of work – every 

two months coming together for a day-and-a-half of meetings to be able to kind of hash things out. But we 

came up with something. Is it perfect? No, but it was agreed upon; and so the agreed upon, as you're 

finding, is a really remarkable thing to be able to come with something where everybody says this is what 

we're going to do.

Now, our incentive here was not to tell people what to do but to simply find out what's going on, what's 

working, what's not, where do some of the principles we have from our disciplines help to be able to 
4 April 10 DOJ Part 4

3

explain what's occurring here. Above all, as we have in the last of that, is we want to share what works; 

we want to get the information out there. If somebody is doing something well, describe, figure out why 

that is happening, what's going on; and if something's not, to identify it and find ways to be able to 

improve it.

So amongst other things, we break our data down by areas of investigation. You can see here it had been 

a longer list, but we've compressed some of the data to simply have enough that we would have 

something that we could talk about in some detail. But blood alcohol, crime scene investigation, digital, 

DNA casework, database, and so on, all the way down to trace. Trace is one of the areas where we did 

have to combine a couple of areas. We didn't have enough reporting separately on things like hairs and 

fibers, paints and glass; so we combined some of those within trace.

But this fit very, very nicely; and some of the data I'm going to show you is just from one of these areas. I 

chose DNA casework largely because we have the greatest number of participants who do DNA 

casework, so we get some nice large data properties that go along with that. But anything that I go 

through here and show you today, these are things that can be looked at down to the detail of any of 

these areas of investigation.

Now, along with these, what we do when we talk about budgets and expenditures here, we are talking 

about a full accounting of all of these kinds of things. So we're taking from the specifics of what's going 

on, on personnel; looking at what's going on with, say, consumables in a particular area; capital 

expenditures spread over time -- laboratory equipment, for example, falls into the IRS' five-year class for 

depreciation purposes, so we look at five years and spread that out.

But we also have all of the infrastructure of the laboratory in place. We're asking, "What are the utilities?"

I've gone into laboratories, and I notice the lights are on. I know that there is heating and cooling.

Sometimes it happens to be heat in the summer and cooling in the winter, but there is heating and cooling 

going on. So we look at all of those costs that are going on and allocate these to be able to kind of see 

what is happening and what goes on through there across each area of investigation.

Participation in FORESIGHT is voluntary. Now, I have data; we began this in fiscal year 2009, but we 

have laboratories that were able to go backwards and extract data for us all the way back to a few of the 

laboratories from the fiscal year 2005. We've just collected and are analyzing the fiscal year 2016 data 

with that. In fiscal year 2016, I had 136 laboratories provide data. To provide data, what they are doing is 

they are sending me an Excel workbook that has one page that's extracting a good bit of detail on 

casework. By casework, they'll talk about how many cases do they have for each area of investigation. So 

certainly, you have cases that come in across multiple areas; so we're going to report that in each area.

How many items of evidence have been submitted across those cases? How many items have been 

examined internally? How many have been outsourced? How many samples taken from the items that 

are examined internally? How many tests that are run on all of those samples? How many reports are 

written?

We have a series of examples that show people how to count – you know, how would you count across 

these things; and time has continued to add additional examples with those. Along with this, they report 

some information with respect to personnel -- so how many full-time equivalent employees are both from 

a technical side and from a support side in each area of investigation; how much are generally out there 

that are working across areas – and we have a way to allocate those; hours that are spent in casework 

versus other types of activities – so we look at all of these things on the casework page.

On the financials, we break down to a great bit of details on this. Sometimes it's just general laboratory 

expenditures in a particular area. The biggest thing is personnel expenditures, which we get detailed by 

area of investigation; and we allocate those with the FTE to be able to talk about compensation and 

things that fit in from there. It talks about consumables and, as I said, all of the things in the infrastructure.
4 April 10 DOJ Part 4

4

Then in the report, they are automatically provided some detail on the output. So immediately upon filling 

it in, they get some output results. We go ahead further, analyze the data, and provide reports back to 

each individual laboratory. The reports go anywhere 80 to 120 pages in length, detailing how do they 

compare to other laboratories around the world; how do they appear over time; and each year, we add 

some new items to that.

We had 116 of the laboratories were U.S. laboratories in the most recent year, which is nice because 

we're getting a growing number of international laboratories. We do have representation from six 

continents; we're getting a lot more.

Now, this last piece of this is Project FORESIGHT 2020; that is the big addition, and I want to show you a 

little bit about this. This is a project that is currently underway. But the biggest difficulty for laboratories to 

participate in the study – there's no charge to them for participation, that was thank you to NIJ really for 

taking care of paying for our time historically on this to be able to analyze it. But the cost is in the time to 

prepare it. So we put together, along with ASCLD – worked with the leadership at ASCLD -- to put 

together a proposal to fund a project called FORESIGHT 2020, which will automatically extract the data.

Now, to give you more detail on the metrics, I have listed at the end of this slideshow 20-something 

publications that we've had out of the project already. This is one that appeared in 2015 that gives you a 

really good idea on what kinds of things we extract from the data. If you want to go in and look at more, 

this is Forensic Science Policy & Management, and it really kind of shows you what we produce and the 

kinds of things that come out of there.

Now, FORESIGHT 2020, this is the project that is currently underway. It is a project that is awarded to 

ASCLD; but Max Houck serves as the Project Manager, and I serve as the technical consultant on this 

project. This is what's funded by the Laura and John Arnold Foundation. This is an amazing couple in 

terms of what they've done, in particular for things that improve the justice system. In this, we had spoken 

to them; and what they wanted to be able to do is to make it easier for more laboratories to be able to 

benefit by this analysis.

So we have a cooperative effort with the major LIMS providers to basically provide freeware to be able to 

connect this with existing LIMS systems so that the data that we have is automatically generated. I'm 

going to be able to show you a couple of screenshots of what we have on the beta testing on this to give 

you an idea of what comes out. It's not only to be able to generate our report, which is currently annual, or 

to be able to generate automatically some of the reports, say, from grant work for NIJ that it can extract 

the data and put that report together; but the key is this last one, is that it offers a dashboard, something 

that a manager can look at for any time period to see what is going on and to be able to work their way 

through.

So if you want it for the entire year, you want to compare it to prior years, you want to look 

monthly/quarterly, you want to look by individuals in the laboratory, how they're performing – it has that 

capability. So this is a little bit easier to be able to see, I'm sure, from the slides directly. But just to give 

you an idea on this, this would be the first level of the screens just looking at what is happening. So this 

screen would scroll down. There are all kinds of graphics that go along with this.

You have the ability to dictate the time period at the top. So you can say: I want it for the most recent 

quarter; I want it for the current year; I'd like to look at what happened last year. It automatically updates 

to what is going on in the laboratory – how many cases have you seen over that period of time, what is 

the current activity, what kinds of requests for services out of each of those cases, how many tests are 

going on, and so on. Then you get a graphical depiction on how this is working over time. That's just the 

Home Page.

The next page is strictly the FORESIGHT Page; and in this, these are the items that we're asking for – so, 

again, defining the time period that you want across the top. What it's doing is it's breaking down for each 

area of investigation. You can see there you have a crime scene investigation; you have fingerprint 

identification; you have this working its way down, digital evidence. This is the total number of cases over 
4 April 10 DOJ Part 4

5

that period that have come into the laboratory. It also has these are the number of items that we're 

looking at in here, or how many samples have been examined in here, how many tests have been run. It 

continues to track these down at each level.

If you are interested in additional items that we have from this, looking at the next one breaks down to 

these. Those bottom graphics give information, for example, on turnaround time – so how long are things 

taking, what's going on with that relating to any of these things. And of course, the manager has these at 

their fingertips. They can say what happened last week, what's going on today, last year – really being 

able to make this a very, very prime managerial tool. And it's free – free for all laboratories, again, thanks 

to the Laura and John Arnold Foundation for funding this to make it available.

We've had great cooperation certainly from at least three of the major LIMS vendors; they're excited 

about this, putting this in place. We're going through the testing with some individual laboratories right 

now, and the project is scheduled to wrap up at the end of February next year, in which it will be released 

and available for laboratories, whoever would like to have it, take advantage of that.

So that's the project that's currently underway. I would say certainly to anybody who is attending the 

ASCLD meetings this year, we have our annual FORESIGHT meeting there; and everybody is always 

invited. We like to get questions, new topics. Things are going on, but Max will be there to speak a little bit 

more about where that project is.

Now, that's a little bit on the history; and I'll certainly take any questions you have later about the project. I 

want to give you an idea on some of the work that we're doing currently, including I've got some results 

from projects that are currently underway, not yet published, but hopefully were now results to writing up 

things for submission on a couple of these.

To give you a little bit of an idea, we get our ideas on projects first and foremost from our membership.

When we have our meeting, we find that people ask us questions: Do you know about this? What's going 

on here?

They give us ideas on things to investigate. Other times, we see things in the data itself that raise 

questions that we have. I published a piece a couple of years ago on lease versus purchase. Lease 

versus purchase in corporate finance is a big topic, but it's one that's pretty much been settled; and I was 

surprised to see how little leasing was going on, recognizing that this is a way to expand budgets. So Will 

McAndrew and I did a piece together on that that was published. So we look for scientists to tell us, 

managers to tell us, this is an issue and so we start looking at those.

Four of those that I want to talk about – one of these is probably jumping higher and higher to the top of 

the list, given the current political climate – and that is return on investment. So what is the return on 

investment?

We're working with RTI right now, the Forensics Technical Center of Excellence, on a project, this very 

first one, which is what is the return on investment from the DNA database? I want to talk a little bit about 

that and some of the results that we have.

The second is process improvement. You'll notice that a lot of laboratories are investing in Lean Six 

training and what's this going to do. We want to be able to measure what can you actually get out of this, 

and so we'll talk a little bit about that here this morning.

The other is one now that I have more international labs: Is there a difference between the performance in 

U.S. laboratories versus worldwide? That one is sparked in part by a very talented young man, a master 

student in biostatistics, who has been working with me, who jut noticed something in the data. So I want 

to talk a little bit about that.

The last of these is extremely important, which is the predictive capability of efforts that are currently 

underway. So when you invest in reducing backlog, what's going to happen? We can predict this; we 
4 April 10 DOJ Part 4

6

have very, very good economic law that sits behind this. Now we have measures to be able to talk about 

this. So I'm going to talk about those four here.

The first of these, I want to give some credit to an article you see in there from Jenn Doleac at the 

University of Virginia. If you've not met Jenn, this is a brilliant young woman who just does some really 

cool things that happen to be affiliated with forensic science. She published a paper in one of the leading 

journals on The Effects of DNA Databases on Crime. This is kind of the first really, really good return on 

investment.

Now, what Jenn is able to show in this – you can see there the quote that it opens with: "In 2010, 761,609 

offender profiles were uploaded to CODIS. This cost the State and the Federal Government 

approximately $30.5 million but saved at least $1.2 billion – that's billion with a "b" – dollars by preventing 

new crimes. So you want to look at what the return on investment is there. Now, the difference that we 

have, or the contribution we can make to what Jenn has done here, is she was going from some very 

general data in terms of what the actual costs are.

By our study, and where we can detail what it cost laboratories – and it's a little bit higher than that when 

you take a fully loaded view of the cost of the laboratory and spread those things out – we can also 

individualize what the contribution is by laboratory; and that's what we're taking a look at. So what we're 

working out is to try to take a general look at the benefits and measuring the societal benefits and taking a 

look at this down to the jurisdictional level and what these impacts are going to be, and it certainly has 

some policy implications here.

Now, if you think about this, one additional entry to the DNA database – not one hit, just an entry and 

what the cost of that is -- brings back to society roughly $20,000 in societal benefits. Show me anything 

that has that kind of rate of return, and you want to see what the value is.

So this is what we've been looking at is to say things that you're doing to improve it, what can you expect 

to get?

So there are some questions about right-sizing, where to put your money, how to organize things, how to 

be able to work it to get the greatest returns. Now, we're going to see some increased demand for 

services as more is put towards this, so turnaround time reductions are going to have some effects that 

we'll talk about in a moment. It is going to be a moving target that you're planning for, but there is good 

economic science to help project what that path is going to be. We're talking about more data-driven kind 

of decision making. But just taking results that we have currently, this is suggesting that even a small 

laboratory that is not producing at a high level or at a very low cost, we're talking about returns in the 

nature of 890% for an addition to the DNA database.

For a laboratory that's operating at peak cost effectiveness, you're talking about 3,800% returns for that

laboratory. To put that into perspective, the leading company on the Dow Industrials, Apple Computer, its

return on investment is 28%. So if you want to be able to talk about what the contribution of the forensic 

sciences is going to be, in this case, dramatic – and this is just one area. This is one of 19 areas that 

we're going to be, hopefully, looking at over time to be able to talk about overall returns.

Okay, now process improvement – we wrote a paper back in 2009 when we first looked at the data; and 

in that paper, we did a decomposition of the data. How do you describe what's going on? What should a 

manager be looking for? What can they compare to?

In our early study in 2009, we have a small group of labs; but out of those laboratories, one of our 

participants was from Orange County, California. Another of our participants was the State of West 

Virginia. Well, economic climate in the state of West Virginia, where the laboratory is located in South 

Charleston, where the laboratory was located in Orange County, there are very, very different economic 

conditions. In particular, there is a big difference in terms of salaries, compensation that you have to pay; 

so there are market forces at play.
4 April 10 DOJ Part 4

7

We wanted to be able to take account of that. We wanted to be able to take account of productivity 

differences – a variety of different things in this. So we wrote a paper on decomposition that we have 

there. We've looked at this and were able to come up with a sense of efficiency and what you should 

anticipate based on jurisdictional size – what is the level of activity, and what is efficient for you versus in 

a market economy, where I come from. In corporate finance, I was working in the banking community and 

analyzing the banking community. Well, what happens in the private sector is in for-profit environments, 

you knock everybody out who is not the lowest cost provider. But in a jurisdictionally-based market, such 

as you have in forensic sciences, things operate differently.

So we wanted to try to take a look at that; and, in particular, what happens if you want to instill your own 

improvements. So Lean Six is one we talk about; there are many other process improvement methods.

What can you expect to gain and some of the policy implications you're going to have with that?

Here are some data, and this is kind of interesting. By the way, people have seen me geek out on this 

one before; but what you see theoretically in economics, in the for-profit sector you don't get to observe 

what you know theoretically goes on because you knock everybody out that is not operating at the lowest 

costs. So what I have is data here. This is for DNA. I wanted to ask questions about, well, if you 

undertake a project on something like Lean Six, what can you gain?

So what this data shows is on that vertical axis, we're looking at how many DNA cases are processed --

and this is DNA casework, not the database but DNA casework -- per $1,000 spent. Along the horizontal 

axis, it's the number of cases submitted over a year. So you find the amount that comes out per $1,000 is 

relatively low when you've got a very low volume; but if you can see in here, there is a specific shape that 

is suggested by this. This is what we look at econometrically. That curve that's in there, that isn't a 

freelance curve; that is the best linear unbiased estimator of this relationship. So what it is – it's the result 

of a quadric regression.

Now, there are better, more complicated techniques that can fine tune this a little bit; but you get the 

rough idea. This is what we refer to with respect to productivity is that there is an ideal size. So if you 

were talking the private sector, you're going to find in any business there is a right size to be, where 

productivity is going to be its highest for the level of activity that you have with that.

So for those of you – I see several coffee containers around the room. There was a difference in size 

between the place where you stopped for coffee and perhaps the Walmart that you passed on the way.

There is a right assistance for a big box retailer – although we're not sure what that size is going to be 

anymore, thank you, Amazon; but there is a right size for Starbucks. Everything has its right size.

The same is true in terms of processing work at the laboratory level -- whether it's DNA, fingerprint 

identification, trace, whatever it might be. There is a size in which you're going to get the highest level of 

production. What we're trying to do is to be able to measure that; taking the data that we have in any 

particular area and say, okay, here's what we would see. So I'll give a term to that curve and call it the 

efficient frontier.

Now, what we'd like to be able to do for any laboratory whose market size is determined by jurisdiction, to 

say for your size, how do you fit relative to that efficient frontier? What gains can you get? Not what can 

you be to be the so-called perfect size that we would have in a market economy, but how do you fit in 

this?

This has been one of the bright spots of what we've discovered, is that while the image of the public 

sector is often maligned, certainly by free market economists who would say, oh, the government sector 

can't do things right – because we have this image – and please forgive me anybody that works for the 

Department of Transportation – but we have this image of the Department of Highways and eight workers 

and one shovel and everybody is sitting around.
4 April 10 DOJ Part 4

8

But we're not finding that in the laboratory. What we're finding is that laboratories tend to operate near this 

efficient frontier. Now, occasionally there are issues; and we try to help identify what those things would 

be.

So here is what we'd like to be able to look at. If you are below this curve in productivity, can we measure 

what a program such as Lean Six can do for you? What's that gap? What are the gains? Is it worth the 

expense going through this? Because you may be really close and the exercise itself costs more than 

what the gain can be.

So I was first noticing this in a very, very interesting project that went on in the Netherlands. To go 

through a process with Lean Six, and this is one of the citations with this in science and justice; and it's a 

remarkable project. What happens if you really invest in quality in this sense, in process improvement –

what kind of gains?

When I saw the presentation – in fact, it was an invitation from NIST almost three years ago -- thank you, 

Mark – and it was a great presentation; it was fascinating. I kept looking in the data and saying they made 

a couple of comments. They found that as they improved, their level of activity went way up. And I 

thought, well, you've got several things happening. Can we separate out how much of this is an 

improvement from Lean Six and how much of it is a natural improvement that you get from scale 

economies – economies of scale, rightsizing –- and to try to be able to split these things down. So you'd 

like to be able to combine this work to be able to say, here's where we're headed; this is what we're going 

to see; this is what our policies should funding as we move forward. At the time, they increased their work 

dramatically. So they had to rebudget and recalculate what everything was going to be.

Now, the inverse of this is just looking at the same data with respect to average costs. So people are 

always happy about productivity – what are you getting per dollar spent? Well, maximizing that is the 

same as minimizing the average cost; they're two sides of the same coin. This is the traditional 

economies of scale that an economist would look at. So again, this is what I geeked out on; I was just so 

excited to see the data -- so trying to look at things that improve and minimize that case – so size does 

matter.

Now, what does this mean?

Well, in the case of forensic sciences, we're also working with public health laboratories on this. Public 

health laboratories have found in some areas of work that they do regionalizing some activity has been an 

improvement for everybody. By simply saying, you know, not everybody can be the one that looks at this 

particular problem. But if we can work something within a region that everybody can benefit, everybody 

can lower the cost; and we've seen some success with that in the forensic sciences.

We did a study in Canada in which we took a look at the entire forensic services in Canada. One of the 

things that they discovered is by moving all trace evidence activity to a single laboratory, everybody 

lowered their costs. The difficult part was how do you prioritize; how do we move money? Those turned 

out to be fairly easy to be able to deal with, but scale of economies helps to tell that.

In looking at this, this is what saw with the data. We're beginning to tell people with this, to be able to say, 

you know, you're too close to get a lot of gain from investing in this; you may want to think about some 

other things to be able to do to improve. To be able to hand each laboratory, here's what you measure; 

you're already at efficient. So a new productivity improvement isn't going to change things.

Now, this next one – as I said, John Bassler is this young man whose been working with me; he's 

finishing up his masters in biostatistics. Interesting guy – he's the biggest research assistant I've ever had.

He was in football; he was a lineman at WVU. You don't find too many math and stat undergrads, but they 

exist on the football team. So he noticed in the data that over time, anybody who has the data about their 

laboratory, they just tend to improve – no particular program, knowledge alone seems to be a behavioral 

trait that improves things.
4 April 10 DOJ Part 4

9

So we're looking at how this moves over time. We began asking questions about U.S. labs versus 

international labs and is there really a difference, or is it just that the domestic labs that we've had in our 

study have been there long enough to notice improvements and to get better? So we're working through a 

study on that: Does knowledge alone improve your operations? Early data on that is suggesting some 

things in that direction.

Now, this is the one that I find most fascinating. We were invited in the fall to speak with respect to sexual 

assault kits and policies trying to improve this. While this particular interest was in sexual assault kits and 

backlog reduction, we were just interested in all backlog reduction programs and what happens with this.

So as backlogs go down, so does turnaround time seem to be affected by this.

Now, in economics, there is a metric that we use; it's called "price elasticity of demand." For any industry, 

you can very precisely predict what the reaction is going to be when you change your price. So what 

happens if you lower it? Well, if you lower it, we know from the law of demand, the quantity demanded is 

going to go up; people want more. But they want enough more that you make more money out of it.

Well, forensic science for the most part, there's not a charge. So there is no price. Your price becomes 

wait time. So I gave this a name; and rather than call it the wait time elasticity, I liked the name the 

queueing elasticity of demand – get in line. So what happens when you reduce the line?

What we found first of all just on some simple metrics in looking at this is that it was less than minus one.

What that means is this; if you reduce turnaround time by 1%, the demand for your services goes up by 

more than 1%. So I was teasing Gerry last year and saying backlog reduction program was the wrong 

name because if you're successful, you'll never be successful. Huh? By being successful and actually 

enhancing capacity, and he came to me immediately afterwards and said, "We changed the name 

already, capacity enhancement."

So a very, very important clue about what's going on – but we can measure it. It's a moving target, but we 

can measure it. So we got the results – not yet published, but I'm going to run through some of these 

now.

The first of these is the metric with respect to casework. Now, this is looking from data that we have on 

laboratories from 2005 through 2016, 531 observations. So we measured this – the elasticity with 

turnaround time with respect first of all to casework. Across that dataset what we have found: that a 1% 

reduction in turnaround time increases casework by 1.29%. So part of this that's going on – yes, there are 

other factors that could be at play in here; but largely, it's suggesting this. Cases are being submitted that 

weren't being submitted before. So, okay, you've got to plan for these increases as you get better; but it 

goes much deeper than that.

We looked at this with respect to the number of items presented – minus 3.9 for a 1% reduction in 

turnaround time. Yes, we have an increase in cases; but back up the U-Haul because more items are 

being – almost 4% increase in the items that are asked for consideration. The better the job you do, the 

more the demand for the services. That doesn't mean that every item is going to be examined. So when 

we go down and look at what the laboratories are doing here, what we find is the number of samples that 

are actually examined is increasing at dramatic rate. For a 1% reduction, it's almost a 3% increase in the 

throughput.

How do you fund this?

You've got to plan for it; you've got to be able to look at this. And these are very rough estimates simply 

based on annual submission of data. Once we get data from our FORESIGHT 2020 project, where we 

can then begin to examine for much shorter periods of time, what are these impacts – when does that 

knowledge become generally known that, hey, they're getting better at this; they're getting faster; let's 

give them more.
4 April 10 DOJ Part 4

10

Right now, I can only use annual data. So I look at what was your turnaround time one year; what 

happened to the submissions the next year? Now if I can begin to do a quarterly or monthly or whatever it 

would be to try to find out and predict what this is, to be able to take every individual market and to say

this is what the demand for your market looks like because I've got time series data for the individual 

laboratories. FORESIGHT 2020 allows us to do that.

So what do you need? What do you put into the funding awards? How do you go to your funding body 

and say, "Look, here's what's going to happen here." Not here's what happened historically; I now have a 

predictive model. You can look at program effectiveness, and you can go back. I'm still struck by things in 

2013's GAO Report and saying, well what does it --? You've got it. Does this work? Yes, you can come 

out and you can pull it right out of these things. Policy implications – to be able to now extend to that 

return on investment to the individual laboratory, to be able to talk about what goes on there; and we can 

track it better as we work our way in here.

Moving forward, as I said, topics we look at are occasionally something we find in the data; but generally, 

it's an observation that some lab director has – What's happening with this? What's going on there?

As I said, we have at ASCLD 2017, once again, that's Sunday evening right before the meetings begin.

Anybody that wants to is invited. Please talk. This is how we get our ideas. We welcome anybody to 

participate with that.

The FORESIGHT 2020 support continues through February 2018, but we are about to be flooded with 

data. The ease with which laboratories are able to provide data and get it through there and the big thing 

comes as an industry. One individual laboratory does not necessarily have the incentive to go through 

this. And it's figuring out how, as an industry, you share, you get more people. I've got a small group of us 

that are working with this data. We'd love everybody to have access to the data and have other people 

researching things – but to look at ways to house the data, to collect the data, to make that available.

Our project funding, as I said, for FORESIGHT – that ran through fiscal 2010. We stretched it out as long 

as we could to be able to keep funding the activity that goes in there. This foundation – some grants from 

RTI is keeping this in mind. But as an industry, there is so much that you can benefit from by learning 

these things – keeping that on the table.

Finally, if you look within the current climate, certainly being able to speak a language that is a language 

of business – and whether you like calling forensic science an industry, it is. It is a collection of, yes, 

public sector corporations that are units that are making decisions for different goals, certainly. But 

collecting that information and being able to learn from each other and how to do things better – not just 

in the science, but how to select across these – is just a very important thing to keep in there as 

something as we work forward.

I'd be happy to take any questions that anyone has with this.

NELSON SANTOS: I'll start off with a question because now I'm really concerned. So I have 25,000 

samples in backlog; and if I read what you said right, the more efficient I get, the greater my backlog is 

going to get.

PAUL SPEAKER: Yes.

NELSON SANTOS: When does it plateau?

PAUL SPEAKER: Well, that's the thing. What you see is a demand schedule; it's not one elasticity 

measure. So this is kind of an average across the industry. So this is where we'll begin to get that out of 

FORESIGHT 2020, is to be able to look at that where we can go and we can look at your individual 

demand schedule and say, "How's that going?"
4 April 10 DOJ Part 4

11

So what you find is typically at very high prices, you find very elastic responses. You lower price a little bit, 

and you get big increases percentage wise – big increases; but eventually, it eases off with this. We'll be 

able to see where that point is. We'll be able to measure that for you, for any individual laboratory and 

say, "This is what your demand schedule looks like; as you get across that, you're going to find at lower 

and lower levels it begins to ease up."

NELSON SANTOS: Okay, because I think what's happening with us is exactly what you're saying. The 

demand – so we can't provide it quick enough. So they're going to state and locals – our DEA agents. So 

once they start seeing it come back, you're right; it's going to come back to us. But I'm hoping there's a 

complement.

Cecilia?

CECILIA CROUSE: Thank you, Paul. I've heard your presentation several times; and once I get past the 

word "economics" I'm okay.

I just had a couple of questions. We've conducted Lean Six Sigma in our forensic biology unit, and it took 

us about a year-and-a-half through a grant from NIJ, which we sincerely and deeply appreciate. The most 

important things was buy-in. The second most important thing was that we ended up making sure that 

there were black belts that stayed in our laboratory so that when issues like this happened, they could 

use their Lean Six Sigma training.

What we have found out was – and this is my concept of what we have found out. First of all, the team 

approach – they love it. They are cranking out a lot of work. They are finding out exactly what you're 

saying; and we have a case submission policy, where you're only allowed so many items depending on 

the type of case. That's helped to regulate it, but now we're getting more cases. So that initial ceiling –

this is what we know is coming through the door every month, this is where we need to be – that's moving 

because of this.

The question that I had is I want to know what the effect of critical thinking is. Has anybody thought about 

the fact that what we are finding out is that there are occasions when having this production line, this 

robotic line, that the critical thinking on some of these cases is not happening because other people are 

making the decision. The serologist comes in; they do the screening; they put what should go forward.

And then the DNA teams all get them, and they all rotate; and it's this big dance that kind of happens.

Then at the end, for example, someone would say, "I don't understand why you did the boots; we really 

needed to have the bandana done."

"Well, there was blood on the boots."

"Well, yeah, but they were bleeding; it's not..."

So my concern is, are we creating through this – we're creating efficiency; backlog has come down; they 

love the process. My concern is do we have technicians and what happens to the scientific critical 

thinking that comes with casework?

Well, that's a great question and a very timely one since I just took a call about a month ago from some 

guys working in the area. My background is economics and finance; so my tendency is to be more 

interested in the projects that are really looking at numbers from that standpoint. I got a call from 

somebody working on their master's thesis in management that wanted to measure more of those 

qualities. So they'll be presenting at ASCLD on a proposal for a project looking at those: How do we 

marry both of these metrics – and the critical thinking being the key one on this?

So what they've done is they've taken from studies across all kinds of businesses – any kind of industry 

that you have – and looking at those predictive capabilities on your ability to improve. Do you have those 

things in place as well, combining them. Where we're simply trying to measure what the potential is from 
4 April 10 DOJ Part 4

12

a financial standpoint or a productivity standpoint, and we want to marry to the two concepts to be able to 

say, okay, this is how you're going to gain and this is how you're going to be able to sustain it.

What we're finding, anecdotally at least in a few laboratories, is they've gone into a process; but some 

failures are happening later down the road. The ability to sustain it – and a lot of it comes back to that 

critical thinking aspect. So how do you get the sustainability to get the improved results but also get the 

intellectually stimulating environment so you're getting good kinds of results? So that's one of the topics 

that we'll be discussing at ASCLD.

NELSON SANTOS: All right, Susan and then a couple more, and then we've got to close.

SUSAN HOWLEY: Mr. Speaker, I was wondering how well you've been able to isolate a causal 

relationship between the reduction of the turnaround time and increased demand. It seems that when 

people are addressing backlogs, you also have other things going on – like new State legislation that has 

timelines for how fast samples need to be submitted to the lab. And then there's all this accompanying 

increased public awareness, or especially awareness among law enforcement that, okay, now we really 

do need to count the evidence that's been sitting in the closet and get it turned in.

So how much have you been able to really say it's the fact that we're reducing the time and offering better 

service that's led to the increased demand?

PAUL SPEAKER: Typically, when you're going to look at elasticity, there are a variety of things that you 

assume are held constant. So certainly having a smaller time period in there to be able to look at things 

becomes an essential piece of that. Other things that would affect it – so if I'm looking in here, the results I 

gave you were just for DNA casework. So what's going to affect that? Well, the turnaround time in other 

areas is going to have an impact on that.

So if fingerprint identification is taking longer, then you may shift some of the emphasis in a particular 

case over to that activity, for example. So we know there are a lot of things that are going to affect that;

and that's where the time series work with FORESIGHT 2020 is going to help us, where we can narrow 

that down to a scenario in which we have more of a constant environment.

Right now, what we won't pick up is if there was one thing that occurred from one year to the next. Now, 

will that one activity have affected so many things across time? No, so we're not terribly bothered by the –

in economics, we always refer to "ceteris paribius," all things equal. We're going to hold everything else 

constant with that. But we'll be able to do more of that once we have the shorter time periods to be able to 

measure on that – be able to correct for seasonality, for cyclical behavior, and be able to talk about very 

specific events as you look at an individual laboratory – this law went into place or this thing changed.

NELSON SANTOS: All right, Jim and then Jules, and then we'll go to our next panel.

JIM GATES: Thank you.

Paul, thank you for the wonderful brief. I'm a scientist, and so I love data; and this was a data rich 

presentation.

My question had to do with something that you touched on, but I'd like to hear an expanded response.

Many of my fellow Commissioners know that I have, on a number of occasions, brought up the issue of 

performance evaluation versus proficiency evaluation. One of your comments was looking at the issue of 

performance evaluation across, I think, international borders – I believe it is. Could you expound on that, 

particularly on the axis of do you have data that will sort of tease out whether one of these approaches –

namely, performance evaluation versus proficiency evaluation – needs a more efficacious treatment of 

caseloads?

PAUL SPEAKER: One of the things that we do in decomposing the data is we're able to take a look at a 

lot of the detail as well as the explanation. Our decomposition is a fairly simple one; and it's a process that 
4 April 10 DOJ Part 4

13

now is one hundred years old right now, a thing called a "DuPont expansion," and named after the 

company DuPont, where an employee – I won't go through my normal dynamite salesman jokes here –

but he said, "How do you look at data and performance? It looks too good to be true."

At the time, DuPont was looking at investing in General Motors. They had all this excess money, and they 

wanted to be able to put it in something. So this employee, who was at Carnegie Tech, had a couple 

years of that good quant work, said, "All you have to do is multiply by the number one in a variety of forms 

and create a series of ratios, and you can kind of break down what's going on."

So we've done that here. In looking at the laboratories, we look at what their overall performance is and 

we say, well, let's try to explain, to see what are the pieces that are telling us what's going on with this. So 

I just showed a couple of the curves with this; but we do that. We go ahead and we look at the overall 

performance; we look at productivity; we look at markets; we look at investment in capital because any 

laboratory can improve its performance and particularly by let's put off that investment in the new GC

Mass Spec or whatever it might be. We can get more out of it right now. So we break all of those things 

down.

Now, at the international level, our interest in looking at this was more out of our very early discussions

where particularly our Canadian counterparts would talk about some of the loss that they had to contend 

with and why differences would be in place. We get a lot of those similar comments that are coming out of 

our Australian laboratories, where they talk a little bit about the detail.

So what we've done is we have a look where we can compare to the efficient frontier – and we're trying to 

figure out what the best metric is on that; right now, it's a fairly crude one. It's simply what's the 

percentage gap from that? And we look at that percentage gap over time, and it turns out that everybody 

narrows in on that efficient frontier, regardless of where they're from. So it's really a matter of taking a look 

and saying, okay, how long have you been tracking your own data that you've got some improvement?

A good case in point for why this happens is a 2010 article that's listed at the back there by John 

Newman, myself, and David Dolly, where we looked at how did the Toronto laboratory use the data? And 

I'll say this – very, very smart director. He went and showed the results to every unit manager and said, 

"What would you do differently?" And, boom, they tracked it and began to find things.

So you can track a lot of very, very particular events to the results; and say, yeah, this is a performancerelated thing. We can pull other things out and separate things and say, "This is purely a size 

phenomenon that this is happening."

JULES EPSTEIN: Good morning and thank you. I have one quick question; and if it's off topic, please 

say so. 

One of your slides talked about this remarkable economic gain from having DNA databases. So I went 

and read the article, and I realized it's not your article; and if it's tangential, say so. But are you really 

confident in those numbers? It seems, at best, to be a correlation, not a causation; and it's based on a 

theory of the author that people who commit crimes actually think to themselves when they get out, "Aha, 

they have my DNA; I will be more circumspect."

So again, I've really enjoyed learning a lot about everything else; but it seemed to be a big part of this. So 

I'm just asking, is that good – I was going to say economics or science?

PAUL SPEAKER: Yes, it is very, very good economics that's in there. This is probably – I think it's the 

second most cited journal in economics, where this appears and the rigor that it's gone through. So I have 

a lot faith in the work that appears there.

JULES EPSTEIN: All right.
4 April 10 DOJ Part 4

14

NELSON SANTOS: Okay, thank you, Paul.

[Applause]

All right, we're going to move right into our next panel, which is actually our first feedback about our work 

from the legal perspective. So if I can have the speakers come up, please.

BILL FITZPATRICK: Good morning, everybody.

My name is Bill Fitzpatrick. I've been the elected District Attorney of Onondaga County, located in Central 

New York, a community of about half a million people, for the past 26 years. I've been a homicide 

prosecutor most of my adult life. I've taken over 80 defendants to trial, 4 homicides, most of those cases 

involving forensic evidence; and I did an extensive period of time as a defense lawyer in my community.

I'm a long-time member of the New York State Forensic Science Commission. I've served with Peter 

Neufeld for a number of years. I currently have the honor of representing America's 2,500 elected and 

appointed prosecutors and am Chairman of the Board of the National DAs Association; and I speak to 

you today on their behalf.

Although I will not dwell in my comments on it, there is much about the National Forensic Science 

Commission that my members find troubling. It's the belief of NDAA that state or national forensic science 

commissions, as the title suggests, should actually be composed of practicing forensic scientists with 

input from any advisory panel that a government deems necessary, composed of practicing defense 

attorneys, prosecutors, judges, ethicists, and certainly other scientists. It is an inherent conflict for 

prosecutors and/or defense attorneys to sit on an accreditation or policy board determining whether or not 

crime laboratories are practicing under acceptable standards and then to use that determination in court 

to either validate or attack the evidence that lab has analyzed.

In addition, discovery rules, reporting standards, testimonial parameters, et cetera, should not be 

determined by a majority vote of a Commission. If I were to have taken the PCAST report seriously, which 

I did not, or the 2009 NAS report, which I did not as well, and to embrace some recommendations 

suggested, I could actually picture myself in my office telling a grieving father of a domestic violence 

victim that even though his daughter's ex-boyfriend was ID'd fleeing the scene, even though he left a 

fingerprint that matched his left thumb in the victim's blood at the scene, and even though a forensic 

odontologist matched a bite mark on the victim's chest of the defendant, and the .32 caliber recovered 

from his jacket pocket was ballistically matched to the projectile in his daughter's head, it's just right now 

the case is too thin to proceed, and I won't be able to prosecute. Please.

Finally, I see the word "customer" leaking itself into our vocabulary when talking about the Defense Bar 

and public crime laboratories. Defense lawyers are not customers of crime laboratories. A customer does 

not buy a car, be totally satisfied with every aspect of the car – its performance and reliability – and then 

spend the next year trying to denigrate the salesman.

With that said, let me point to common ground that should unite all practitioners and scientists in getting 

sound, reliable information to the factfinder and assisting them in making one of the most profound 

decisions that our system allows – whether or not someone has been proven guilty beyond a reasonable 

doubt.

My members believe that, number one, all forensic crime laboratories should undergo rigorous 

accreditation by ASCLD Lab, ANAB, or other recognized agencies with some time left over to actually 

analyze evidence.

Secondly, we believe the comparative identification disciplines – such as tool marks, identification, 

ballistics, fingerprints, bite mark evidence, tire and footwear impressions, are reliable science; have 

withstood the test of time; and should be constantly tested, improved, checked for error rates; have 

recognized standards; and be presented in a fair manner in court by certified scientists working in 

accredited labs to help a jury make its determination.
4 April 10 DOJ Part 4

15

Thirdly, that an Office of Forensic Science should be created under the auspices of the Department of 

Justice. Its mission should be the constant improvement of forensic science; how best to present 

conclusions in court; and it should develop rigorous ethics training for forensic scientists working in public 

laboratories with continuous monitoring so that mistakes, such as composite bullet lead analysis or 

exaggerated hair comparison testimony, are avoided and eliminated.

It would also develop training programs for judges who, as the gatekeepers in criminal trial, must make 

the initial determination of what evidence presented and what experts, pseudo or otherwise, are allowed 

to testify. Such a body would also provide advancement of accreditation, certifications, research, grant 

money, interoperability of databases, and guidance for testimony reviews.

The Office – and this, I speak for myself and not on behalf of my members – but it's my belief that the 

Office would ensure that coroner systems in the United States are replaced by medical examiner 

systems. This could be accomplished with a program with tuition relief for forensic pathologists who will 

commit to serve underserved areas. I assure you from personal knowledge that there is a serious scandal 

in this country of homicides going undetected because of lack of competent forensic autopsies.

Fourthly, every DAs office should make conviction integrity a priority. DNA testing, it is our belief, should 

be allowed at any time during the course of a criminal trial, including post-conviction; and it should be 

allowed anytime if possible or germane to the issue of guilt or innocence.

Fifthly, that states that do have forensic science commissions, whether appointed by the governor or state 

legislatures, that those commissions should be populated by forensic scientists. Secondly, we believe that 

each state should have a prosecutorial best practices committee. They currently exist in 30 states in the 

Union. These would provide rigorous training on the latest developments in forensic science, the bounds 

of permissible testimony, the examination of past errors, and constant ethics updates. Thirdly under this 

category, a Defense Bar best practices committee with similar training responsibilities and a recognition 

by each state that in cases involving complicated forensic science issues, a court-appointed lawyer must 

be certified by said committee as competent to practice in that area.

I hope this Commission will consider these criticisms and suggestions as you near completion of your 

mission. Sadly, neither an unethical prosecutor nor an unethical defense attorney is going to be much 

bound or deterred by any recommendations that this or any other body makes. The overwhelming 

number of people that I represent care passionately about justice. A district attorney in a medium-sized 

office will exonerate more defendants in a month than a busy defense lawyer will in his or her lifetime.

But our ultimate mission is to keep our community safe, to speak through the victims of violent crime. We 

all take these oaths to protect and defend the United States Constitution, and we need evidence to 

accomplish our goals to convict the guilty and, just as importantly, to exonerate the innocent -- evidence 

that is fairly obtained, rigorously tested, and presented in a manner that assists – not misleads – a jury.

Tomorrow morning, I will be in New York for a meeting of the New York State Forensic Science 

Commission to discuss and debate an issue that I have been pushing for, for over five years – familial 

DNA searching, a proven, acceptable method of scientific analysis that has demonstrated concrete 

results in the states where it has been used, and yet has been withheld from my citizens by the same 

worn-out clichés about racism and privacy and the sky is falling that are the exact same arguments that 

were made against the use of DNA databases in the '90s. This issue, in a nutshell, represents the very, 

very best and the very, very worst of matters that we discuss. I hope this body opts for the very, very best.

I appreciate your time very much, and I wish you Godspeed in completing your important task.

NELSON SANTOS: Judge?
4 April 10 DOJ Part 4

16

DAVID WAXSE: Well, from a slightly different point of view, my name is Dave Waxse; and I am a Federal 

Magistrate Judge. I know that not many of you are judges, so let me give you a quick background on what 

a Federal Magistrate Judge is.

We are considered the same as district judges, with certain limitations. The great limitation that I 

appreciate is we don't do any felony cases; we can do anything else a district judge can do, but even at 

the party's consent, we don't do felony cases. We do preliminary criminal matters, and that's where my 

interest in forensic science developed because one of the preliminary things we do is something called an 

"identity hearing." I've been on the bench 18 years now, and I've only had one of these.

What an identity hearing is, is when someone in the Federal system is arrested in one district by a 

warrant out of another district, they have a right to challenge the question of whether they are the person 

named in the warrant. So I, a couple years ago, had a case where the defendant said, "That's not me."

He had been living in Kansas for 30 years, had no criminal record, had a job and a house.

According to this warrant, in 1972 in Texas, he failed to show up at a trial. It had a different name; but 

what had happened is this guy was not a citizen. He had a green card for the last 30 years; and every 10 

years you go in and check and redo your green card, and they do a fingerprint scan. The fingerprint scan 

said he was this same guy in Texas that had this warrant from 1972. He said he wasn't. So the statute 

says if there's that dispute, they have a right to a hearing.

So we set it for hearing. Having read some about forensic science, I was interested to see how the 

attorneys would approach this. Neither attorney – neither the prosecutor, the assistant U.S. attorney, or 

the public defender – had any concept about science and hearings. So they called as their witness a 

fingerprint examiner, who was not the FBI's person because she was busy; this was a local woman, who 

started testifying in a manner that seemed a little shaky to me. But neither attorney asked any questions.

She said that was a match, and that was the extent of that. I thought, "If I'm going to send this guy away 

for something he says he didn't do, I'm going to get to the bottom of this."

So I started asking the normal Rule 702 questions about what she had done. It was kind of astounding to 

me because education wise, she said she'd taken a correspondence course in fingerprint examination; 

but she couldn't remember who the course was from. I started asking her how she does her fingerprint 

examination; she said, "Simple, I just look at the fingerprints; and they're either a match or they're not."

I said, "Is there any, to your knowledge, any scientific research backing that method of identifying people."

She said, "No, I don't know of any; but this is what I do, and this is a match."

I said, "Well, I've got to have a little better information." So under Rule 706, I have the right to appoint the 

court's expert to do this. Having read that National Academy of Science report, I called Professor 

(inaudible), who she said she'd be glad to be the expert in this. What happened in the interim, since this 

guy had no criminal record I had released him on bond. Unbeknownst to me, the Government had 

appealed that release to a judge in Texas, who quickly reversed me and said, "Keep him in custody until 

he gets here."

Well, while I'm trying to get the expert set up, the public defender decided they'd had enough waiting in 

Kansas. He came in and said, "We're going to waive this hearing and go to Texas." Because one of the 

problems had been there was no clear law in the United States on who had the burden in these identity 

hearings and what the standard was. There are so few of them that there was no law.

He said, "We don't know what the law is here; but we know in Texas, they've got to prove beyond a 

reasonable doubt this is the defendant."

So they went to Texas; and within a day they were back because the prosecutor looked at all the 

evidence and said, "I'm not wasting my time on this," and dismissed the charges.
4 April 10 DOJ Part 4

17

So with that background, I got interested in how all this works or should work; and just coincidentally, at 

that time I was Chair of the Judicial Division of the American Bar Association. Are there any ABA 

members in this group?

A few -- for those of you who aren't members, we are an association of approximately 400,000 members 

that's been in existence since 1878; and our purpose is to serve equally our members, the profession, 

and the public by defending liberty and delivering justice. The ABA is a membership organization that's 

governed by an elected House of Delegates and the Board of Governors. One of the groups making up 

the ABA is the Judicial Division, which consists of approximately 4,500 both Federal and State judges in 

the United States. As the Chair of the Judicial Division, I was able to select a program for the Division to 

focus on during my year. I selected forensic science and criminal justice as the program and appointed a 

committee to work on an issue, which coincidentally included Judge Hervey and Professor Henderson 

and Barry Scheck, who have been here and part of this meeting.

The committee then drafted a resolution for the House of Delegates to consider; and it's kind of ironic 

because our resolution was that we urged the NCFS to develop a model curriculum for judges in the law 

on forensic science. The ABA passed that resolution; and as far as I know, that's the last thing that 

happened on that issue because the ABA structure is not very good at following up on resolutions. To my 

knowledge, your group has not been approaching to find the solution to that problem with the curriculum.

But the committee has continued to work, and what we've done is worked with other groups. Right now, 

we're working with CSAFE at Iowa State to work on such a curriculum. Part of our effort has been to 

attempt to educate judges through both programs and written materials in the interim. One of my duties 

as Chair was to write a monthly column for something called the JD Record, which I did all those on 

forensic science. We also published, as a Division, a quarterly judges' journal that included an issue on 

forensic science.

One of the things we did is with Northwestern School of Law, we conducted a symposium on forensic 

science; and that symposium ended up in this law review article in the Journal of Criminal Law and 

Criminology containing the papers from the symposium. Some of us have been continually making 

presentations to various groups; and these have included ABA webinars, one of which we got Judge 

Harry Edwards, the Chair of the NAS Commission, to be on our webinar to explain to lawyers and judges 

what they had found. The most recent one I've been involved in, former Judge Gertner and I made 

presentations on forensic science to a conference in Kansas City of about 400 lawyers and judges.

In addition to the efforts of the Judicial Division, the ABA Criminal Justice Section, through its own 

Committee on Science, Technology and Forensics, has been doing similar things; and they've been 

having annual seminars on forensic science, I think, for about 10 years. The last one was chaired by 

Barry Scheck.

The point I want to make though is this problem still exists. There are still cases that just amaze me, 

where I read opinions by judges who clearly have not understood the forensic issues. And a common way 

of avoiding that is to simply say, "Well, we've always done it this way; and I don't see any reason to start 

checking on it now." So that's why we still have bite mark cases, when the science says you don't have 

very much there with some guy saying, "This is a bite mark and it came from this person."

The ABA's goal is to try, through various means, to educate judges and lawyers about this problem and 

try and get to the situation where the number of exonerations goes down because there are fewer bad 

convictions. It's just been amazing when you start studying these exonerations. For those of you that are 

interested in that area, there's a book now, maybe three years old, on convicting the innocent that looks 

at the first 250 exonerations the Innocence Project did and tried to analyze, well, if they weren't guilty, 

how did they get convicted.

The statistics are pretty scary because the largest cause of improper convictions is bad eyewitness 

testimony. The second largest cause is bad forensic science. They sort of tie together because there's a 

lot of scientific research pointing out the problems human beings have of identifying other human beings.
4 April 10 DOJ Part 4

18

It involves both what you see and what you remember, and we're not good at it; and when you start doing

it cross racially, it's unbelievable. It's almost like it never happens correctly when you start asking one 

race to identify someone in another race; and yet that's the largest cause of wrongful convictions.

So many of these things, like eyewitness identification, could be helped by judges allowing expert 

testimony on what jurors should understand about eyewitness testimony and the problems with it.

Because in addition to the problems of human beings having difficulty doing it, there's the problem of 

some law enforcement being very successful at using various techniques to convince witnesses this is 

who they saw. In some of those cases, the testimony is just amazing. When you see these situations 

where someone is saying with 100% certainty, "This is the person," and it turns out, whether it's through 

DNA or other evidence, they were not the person; and some of them have spent long years in prison 

based on improper eyewitness identification.

So the probation, from our perspective, continues; and since our resolution to ask you to do something 

hasn't resulted in it yet, and since the attorney general has apparently decided you're finished, we're 

going to have to redo our resolution, I think, and find somebody else to help us.

Any questions about the ABA's position or role?

NELSON SANTOS: We'll take questions after.

VANESSA ANTOUN: Hi, I'm Vanessa Antoun from the National Association of Criminal Defense 

Lawyers. So already I have demonstrated the problem some lawyers have with technology and science.

I'm here from the National Association of Criminal Defense Lawyers, and thank you for inviting me to give 

a little bit of our input and perspective and our positions on the use of forensic science in the courtroom.

NACDL is a membership organization of criminal defense lawyers; and we even have some members 

that aren't criminal defense lawyers, some associate members who are scientists or practice forensic 

evidence. We have judges. We have over 8,000 members, and their background and practice areas 

range greatly. We have public defenders; private practitioners; we have individuals who have absolutely 

no experience in forensic science issues; and we have individuals who are actually scientists or who have 

great experience doing this in their cases.

A little bit of my background – prior to coming to NACDL, I was a public defender for over seven years 

and eventually ran the Public Defender Office out in Fairfax County, Virginia, and was in private practice 

for a short time before coming to NACDL. I give that background just to illustrate where we're coming 

from.

Some of the things NACDL has done in the forensic area include developing some principles and 

statements. Our position statements, just like any organization, are developed and ratified. We have a 

collaboration with the DOJ, the FBI, and the Innocence Project on the microscopic hair comparison 

analysis review; and some of those experiences have informed our position. One of the important things 

we do regarding forensic science is conduct trainings for attorneys to understand how to use it and to be 

better equipped to address it in court.

I want to say at the outset, NACDL and its recommendations and principles – it is not about any effort of 

the Criminal Defense Bar to prevent the use of reliable scientific evidence or to prevent prosecution in any 

way. What we are striving to do is to help everyone strengthen forensic science and its use in the 

courtroom. To that end, NACDL did develop principles and recommendations for strengthening forensic 

science, particularly its presentation in the courtroom; and I will just briefly go over those. I believe you 

have the full document in your materials, but I think it will serve as an outline to some of the comments we 

have about the impact the Commission has had on what we do.

As far as those recommendations and what we are working towards as an organization, one of the main 

points is the forming of a central science-based agency independent of law enforcement, such as the 

National Commission on Forensic Science, or an equivalent organization; promoting the culture of 
4 April 10 DOJ Part 4

19

science that encourages independence, openness and objectivity; adopting a national code of ethics; 

reinforcing that prerequisite research to ensure the reliability and validity of forensic theories and 

techniques, and their limitations or measures of uncertainty; providing greater education to legal 

professionals, who often lack scientific experience, so they can fully understand and evaluate forensic 

evidence; ensuring transparency of the process; and providing appropriate discovery and disclosure; and, 

of course, having the appropriate accreditation and accrediting bodies.

We would support the continued existence of the National Commission of Forensic Science, or a body 

like it; and there are many perspectives from our membership in NACDL as to why this would be 

important. First of all, from our perspective, it's very important to have scientists working together in an 

open manner, open debates, public comment, and coming up with a consensus product that everyone 

can see. This perspective is very much needed in the criminal justice system, as we have heard from 

others on my panel, because the folks working on these cases are not in a good position to determine the 

reliability of forensic evidence in most instances.

There are few in the criminal justice system working as lawyers on either side or judges that have a 

serious scientific background and are in a position to even necessarily understand what the terminology 

means. Does this word mean what you think it means? In many cases, to a lawyer, it means something 

completely different than what the scientist is trying to convey; and that can become a real problem, as 

we've seen looking at testimony and looking at cases where it is not necessarily that anyone is saying the 

individual providing the testimony is mischaracterizing anything in many instances; but the fact that words 

mean different things to different people and without understanding what those terms mean, it's very

difficult to even present the testimony in a fair way.

So work this Commission does on things like language for reports and what goes into a case record are 

very important in the criminal justice system for people to be able to have that open view of what goes 

into these conclusions.

Another reason why work from this group, or a similar independent body, is important is because there is 

an overburdened system. It simply isn't the case that we can say the players in the system – the judge 

and the prosecutor and the defense lawyer – are in the best position to determine what evidence is 

reliable and judge it because we don't have the resources to do that in terms of training, in terms of time.

And that becomes a problem when that factors into what evidence is used and how it's presented.

The forensic disciplines, many of them – and I do not have a scientific background at all – came about as 

a tool to identify individuals and potentially catch perpetrators and bring them to justice, as opposed to 

trying to find an explanation for a phenomenon that occurs in the natural world.

From the perspective of NACDL, one of the important things about the work of the Commission, or a 

similar body, is that the scientists bring that independent, non-adversarial perspective to what is going on 

– not just can this be used to convict somebody, but what does it mean independent of the other 

implications surrounding that piece of evidence. As lawyers, we need scientists to tell us what good 

science is and what bad science is and what they look like because the fact of the matter is many times 

we can't tell ourselves. Even if we have an idea of a potential problem with a certain discipline or a certain 

individual's conclusions that we want to address, we're not sure how to do it.

This is aided by Views Documents and other reports that this Commission has put forward on ethics, on 

the use of terms such as "reasonable scientific certainty." This openness and explanation really does 

help, from our perspective, defense attorneys to understand what they're looking at; to better challenge it 

if that is appropriate or to better understand that there isn't any challenge there at that point.

NACDL would also like to thank the Commission for being productive during all these years so far with 

the understanding that with any organization, it is not always easy to come to a consensus or produce 

consensus documents. But we have been pleased to see that this has happened many times, and those 

documents are out there for all of us to use.
4 April 10 DOJ Part 4

20

Another reason why bodies such as this are important, from our perspective, is accreditation is not a 

solution to all the problems and all the issues that can arise with the use of forensic science in the 

criminal justice system. Because it's brought transparency and debate to these issues and gives 

scientists and the scientific community an opportunity to openly discuss this and receive comments from 

the public, gives people like me at NACDL and prosecutors an opportunity to get an outside view, we 

believe that brings value and openness and can create an environment that furthers the reliability of 

forensic science and aids everybody in the criminal justice system in presenting it.

The idea that we would move away from the National Commission on Forensic Science and the 

suggestion that all of this work would be placed under law enforcement's exclusive is very problematic to 

NACDL. That is certainly not in line with what our principles are; and it just because of some of the items I 

have mentioned in regard to the transparency, the lack of resources, and putting this type of forensic 

evidence under an adversarial system. That is how some of the issues began, and we all know the impact 

of forensic evidence and how, if it's not used correctly or if it's misinterpreted or mischaracterized by 

anyone on either side – defense attorneys included, it can lead to wrongful convictions and serious 

consequences. We're not equipped to determine these things, but we are aided by some of the work 

product from this body.

Lastly, NACDL would like to recognize that there is obviously still much to be done. I don't need to tell you 

that, but I would like to formally state from our perspective that there are so many items still on the table 

where work needs to be done by an independent scientific body – standards; the uniform language for 

testimony and reports that had been proposed and may be revised and released again; issues related to 

defense; access to experts; providing feedback on the forensic science discipline review or now what may 

be done in place of that or what may replace it; and the importance of having such a review to make sure 

evidence that has been used in the past was reliable and that the testimony was appropriately within the 

limits of the science.

NACDL looks forward to continued work on forensic science through open debate and an independent 

agency that by its work promotes a culture of science, provides necessary education to the criminal 

justice players outside the adversarial system. Yet science is always changing, and it's very hard for a lay 

person to absorb those changes and to understand them and to use them in such a high-stakes setting 

as a criminal trial. A body like this does make it easier, and that understanding is crucial to promoting 

fairness in the system.

I thank you for this opportunity to give some comments and am happy to answer any questions.

NELSON SANTOS: Okay, let's open it up for questions.

Just so the record is straight, we did pass a work product on training curriculum. It wasn't a curriculum.

The AG endorsed it and asked researchers and others to develop a curriculum. That's the extent, and 

actually we're going to have a panel later on to discuss that exact issue.

All right, Greg?

GREGORY MOTTA: My question is for the ABA and the National Criminal Defense Lawyers Association.

The Commission issued a series of recommendations, and the mandate of the Commission was aimed at 

government laboratories and resources because it's a Commission of the Attorney General. But as came 

up frequently during the discussions, it was recognized that some of the failure -- like the use of vague 

terms such as "a reasonable degree of forensic certainty" or "a reasonable degree of scientific certainty" -

- applied and confused jurors whether it was offered by the Government's expert or the defense's expert.

So my question to the American Bar Association and to the NACDL is you have the recommendations of 

the Commission, and you have the press releases of the Department as to those which they've adopted.

You have a current effort underway to examine those and to determine whether or not you're going to 

issue recommendations to your members to voluntarily adopt and impose those with regard to defense 

experts.
4 April 10 DOJ Part 4

21

DAVID WAXSE: From our perspective, those issues are still being looked at. The ABA has numerous 

sections, divisions, committees; and I don't have knowledge of what all of them are doing in this area.

From my limited research, the Criminal Justice section is working on many of these issues. The Science 

and Technology section is also working on some of them. The Judicial Division right now has kept its 

committee in place on forensic science; and they are working, as I said earlier, with CSAFE to try and get 

a curriculum developed so they can be submitted to the proper agencies that would be able to actually 

adopt it.

VANESSA ANTOUN: So if I'm correct, what you're asking is if NACDL has in the works the adoption of a 

formal policy advising defense attorneys to not use the same terminology you discussed.

GREGORY MOTTA: (Inaudible)

VANESSA ANTOUN: To my knowledge, that is not specifically in the works. I will go back and see, but 

that's definitely an idea that I can propose or that we should consider because it seems like something 

that might be important in raising awareness to defense attorneys about what's going on, a formal 

position. So I'll take it back.

DAVID WAXSE: Let me add one little piece to this. When Judge Gertner and I did this presentation a 

couple of weeks ago, it was amazing to me because it was for both judges and lawyers; and large 

numbers of them afterwards said, "This is just amazing stuff. We wish we had understood this when we 

were practicing in these kinds of situations." And it's just constantly amazing to me how many lawyers and 

judges don't even recognize this being an issue.

NELSON SANTOS: Paul?

PAUL GIANNELLI: I'd like to make a comment, just for the record. Actually, some of the presentations 

illustrate what's wrong with the criminal justice system. I was really in favor, very pro forensic science. I 

was a prosecutor for two years, turned out to be more, and then they sent me to the GW forensic science 

cause, and I taught forensic science at the JAG school, and then I've been teaching. So it's 47 years of 

dealing with these issues. Over that time, I went from very pro to very skeptical and, on some of my bad 

days, just so cynical.

Some forensic science disciplines – and that's an important statement because it is a huge field – but the 

pattern disciplines have not been grounded in science. Prosecutors and defense attorneys and judges 

have abdicated their responsibility to challenge this testimony. I have no confidence – and I'd like to 

submit an article to the Commission as a public comment to support my views, which I've studied for a 

long time and written about, that deal with the manipulation of science that is controlled by law 

enforcement.

What I didn't see is any self-correction. I haven't seen the pattern evidence, people correct, unless they 

were forced to do it; and that's very unscientific -- the lack of self-correction. In fact, it's a circle the 

wagons in my research I've found attitude, and fight everything and everybody. It's incredible that I've 

looked at and was trained in bite marks 47 years ago, and there's just no research that supports it; and 

incredibly, it's still being admitted at trial.

That's an embarrassment to the criminal justice system; it's an embarrassment to the legal system and 

the lawyers that that has still happened. And we had these pattern evidence people who were testifying 

that they're infallible, that they can make statements this is the person to the exclusion of all others. And 

they were doing that for 50 years, and nobody questioned it. They got a zero error rate; and that, yes, hair 

evidence is consistent. That doesn't mean it's positive, but there may be some other person in the world 

that has the same hair. I mean, that is incredibly misleading; and no one did anything about it.

I'd disagree. I think that the National Academy of Science report was good precisely because it had 

independent scientists. I don't mind taking the lawyers off the commissions and stuff, but you need 
4 April 10 DOJ Part 4

22

independent scientists. So I'm very concerned about the future. And so I disagree with Bill about the bite 

marks and the other evidence, as if it's scientific. I think it's been demonstrated not to be scientific, and a 

lot of times I'm embarrassed. You don't look at these death penalty cases and the arson investigations – I 

was on the Bullet Lead Committee -- the overstatements in fingerprints and firearms identification, the 

overstatements in hair evidence. So you see this over and over and over again.

Now, I'm not saying they're junk science – well, some of them are – but other ones have just made 

unscientific statements. You say I am a forensic scientist, but don't apply scientific protocols and analysis 

to what I testified to. That is hypocritical to take that position.

So I just wanted to go on the record; and I will submit my research, if you want to read it, on what I have 

documented – I think I've documented – the manipulation of supposed scientific research when it is 

controlled by law enforcement. Thank you.

BILL FITZPATRICK: I don't want to get into a debate about the criminal justice system in America; but let 

me correct one thing. That law enforcement is not self-correcting, that is just absolutely not true.

New York State, we started a Best Practices Committee about seven years ago; and as I mentioned 

earlier during my testimony, it's been copied in 30 states. I expect it to be in all 50 states. And what we do 

is exactly that; we self-correct. We root cause/analyze any wrongful convictions; we try to prevent 

wrongful convictions; we look at developing trends in science. Myself and Peter Neufeld spent a great 

deal of time in New York working collaboratively to correct the over exaggerated hair testimony that 

occurred throughout the United States – issues such as witness interrogation, defendant interrogation, 

eyewitness testimony, improving identification techniques. These are all things that we do in a selfcorrecting manner and discuss on an almost weekly basis.

NELSON SANTOS: Gerry?

GERALD LaPORTE: I'm just going to tail off of Paul's comments; but, Paul, one of the unfortunate things 

about this Commission is we have come together to try and solve a problem. I think that we all agree on 

that we need to get better at. But getting up here and bashing law enforcement, prosecutors and that sort 

of thing, that's not a constructive way to do this. I think unfortunately those kinds of comments have really 

divided the community.

I've enjoyed working with everybody on this Commission; and it's been truly, truly beneficial for me. I've 

learned so much. But we've got 18,000 law enforcement agencies, and I'm guessing that most of them 

don't want to get it wrong either. They're doing their job; they're working hard every day. None of us want 

to get it wrong.

But I want to direct a couple of questions sort of as a going forward to the panel. First of all, thank you 

very much. I don't know if each of you could just give me like sort of a quick thing on there's going to be 

some continuation of this work, I think, going forward; and I'm kind of interested in like if we were to do 

something to improve upon the work that's been done in here, what's kind of one thing that each of you 

would like to see carried on?

I realize training is a big issue, but I think training can go on around this country without this Commission.

That can go on. So I'm kind of wondering if you each could pick one thing of something that – in essence 

if this Commission was to go on -- what's one thing that you would like to see the group kind of do.

BILL FITZPATRICK: Well, if we go in the same order, I'm sure the Judge is going to talk about training. If 

I'm preempting you, Your Honor, I apologize. I think training is something that pops into my mind 

immediately; but because it appears to be the Judge's bailiwick, let me leave that to him. I would say 

constant reexamination and improvement of forensic sciences.

I started as a prosecutor in 1977. In 80 murder trials, I used hair testimony once; and it was not 

comparison testimony. The defendant had a genetic defect that caused his hair to twirl around itself. It 
4 April 10 DOJ Part 4

23

was so unusual, that the microscopist mentioned that it seemed to be very similar to the hairs that were 

found at the scene; and there was a tremendous amount of other evidence that wound up convicting him.

But constant improvement, delineation of scientific standards, improvement about DNA – I suspect that 

we're in an area right now, and we discussed this in New York, where DNA can be analyzed at a crime 

scene. Is that a good thing or a bad thing? I've urged caution on that because I don't want police officers 

making determinations and possibly destroying evidence at a crime scene. But these things are 

developing exponentially; and I think that much good could come, Gerry, from continual exploration into 

improving that product in a way to get it into court. That would be (audio break).

DAVID WAXSE: Well, I think clearly training and education would be the No. 1 thing for us. Part of the 

problem is that I've talked about reasoning lawyers and judges; but there's also this problem out there of 

training or educating the public because these people end up on juries with the idea that every case can 

be solved by some kind of scientific thing, based on what they've seen on television. And it's hard to get 

them to actually listen to what the judge and the lawyer are presenting about science because they've 

already made up their mind that this is the answer, so we don't need to spend too much time thinking 

about this.

As I said earlier at the beginning, I don't try felony cases; but I try jury trials, and it is a very scary thing to 

talk to jurors after a verdict and find out the basis for their decision. It may be the best system we can 

come up with; but it's scary when you start analyzing what goes on in there because it just amazes me 

some of the things that are used to make decisions that didn't even come up in the trial, but the jury has 

somehow gotten hooked on something.

So the system has lots of things that need to be improved; but we've got to start, I think, with the lawyers 

and judges.

VANESSA ANTOUN: Since training has already been addressed, I will say that I believe one of the 

things that would be very important to NACDL is an independent group with independent scientists, such 

as this, developing, in an open and subject to debate and transparent way, some consensus documents 

as you all have been able to produce as opposed to potentially views and policies being developed by an 

agency in a non-transparent manner, where defense, prosecution and judges don't know what went into it 

and it doesn't aid in their understanding like the open process with the comments from all the scientists 

does.

NELSON SANTOS: Jim?

JIM GATES: Thank you, and I'd like to thank the panel for the briefing.

My comments mostly are directed towards Attorney Fitzpatrick.

NELSON SANTOS: Can you move closer to the mic?

JIM GATES: Certainly, I first of all would like to thank the panel for the briefings; they were informative as 

usual.

I have some comments mostly I'd like to address to Attorney Fitzpatrick, who had a very stimulating 

opening set of comments. I was a member of the PCAST Working Group that wrote the forensics report, 

which I suspect you probably know; and I guess a couple of things come to mind from your opening 

remarks.

The first thing is that – well, let me go to sort of the big one. You mentioned the word "race" in your 

presentation at some point, I think; did you not?

BILL FITZPATRICK: I made reference to arguments that are made against familial DNA searching and 

being racist, the same arguments that were made in the '90s about the DNA data.
4 April 10 DOJ Part 4

24

JIM GATES: Sure, and so that sort of rung a bell with me because I'm probably the only person around 

this table whose had a policeman, on more than one occasion, draw a gun on them for no good reason.

I'm also the only person around this table who is of the African diaspora; I suspect those two things are 

connected in some way and so when people bring these sorts of charges, not in an absence of evidence.

Speaking of evidence and the PCAST report, I'm not sure if you are the gentleman who made this 

statement; but there was a statement from the NDAA about the PCAST report actually directed towards a 

version of the report that had not even been released. One might have a little bit of issue with evidence 

there, as you're criticizing something that's not in its final form; but that happened.

Speaking of evidence, there was a time when leeches were used in medical practice. It was a tradition of 

some long standing. We don't do it today because, well, we've learned that we can treat in better ways.

You mentioned pattern matching evidence in your opening statement, bite marks. When we talk about 

evidence from the point of view of science, we're talking about things that have an empirical basis in 

measurement. In fact, Einstein actually says that Galileo is the father of all science because he's the 

person in the Western tradition who pointed out that if you use the word "science," it has got to be based 

on measurement and observation. If you don't have a body of knowledge based on observation and 

measurement, then you cannot call it science, according to Einstein and Galileo.

So while there may have been traditional practices, a body of literature established/used, if it's not based 

on observation and evidence that you can test, then it doesn't qualify as science. That was the main point 

that was attempted to be made in the PCAST report. The PCAST report, unlike some of the ways that it 

has blown up, is very limited; it's on pattern matching, and not all of pattern matching but the pieces of 

pattern matching where we were confident, where you could put metrics on the measurements and then 

say we have a good way to proceed with arguments.

The other part of the PCAST report was indeed on probative testimony in the courts. I suspect that you've 

probably tried cases where you've had forensic experts perhaps use some variation of the term "scientific 

certainty." For those of us who are scientists outside of the discipline, that rings a bell because I don't 

think there's any other part of science where that expression would exist. That suggests that within the 

forensic sciences, there's something that's very different about understanding the definition of science.

That also suggests that it's extraordinarily important that forensic science include in its conversation 

scientists who are not part of the company, who are not part of the community, because we will bring you 

back – this community – we will bring you back to what science is as it has developed in the entire 

Western tradition.

My final comment is I've been the last couple of days talking to a couple of reporters because they've 

noted the sunsetting of this group. And let me use this as an opportunity to thank every single 

Commissioner around this table; I have learned a tremendous amount about this community. I am still a 

student of this community. I think most of you have concluded I'm pretty collaborative; but the part of this 

community I find that is likely the most resistant to change is the part of the community that you represent.

So with this, I'm finished; and any response that you'd like to make, I'd love to hear.

BILL FITZPATRICK: Very, very quickly, I don't know the circumstances of an officer drawing a gun on 

you; but I think it's that type of logic that has something to do with what this group is doing that permeated 

the PCAST report.

Secondly, the fact that NDAA criticized the report before it was actually published is actually not correct; 

what we criticized was a version of the PCAST report in the New York Times leaked to them by a PCAST 

member before the appropriate community had a chance to examine it and comment on it. The National 

DAs Association has written a very, very extensive critique of the PCAST report which, in my judgment, 

has had zero effect on any criminal justice matter in the state of New York. I can't speak for the other 49 

states.
4 April 10 DOJ Part 4

25

NELSON SANTOS: Let's go to you, Suzanne, since I missed you the last time; and then we'll come back 

– a couple more, and then we'll break.

SUZANNE BELL: Bless you, Nelson.

I think my colleagues have done a great job of defending the role of independent science. So I want to 

direct a question to the panel; and this is more about cases where we see, like for example, where drug 

tests in the field are used and end up in plea bargaining convictions, so the juries never get to see this. Is 

the answer to that kind of problem – because one of the things, and I'd talked to Peter about this, it seems 

to me that one of the core issues is lack of access to scientists by all parties. And it's appalling to me to 

read about those cases because drug chemists know what (inaudible) tests are for. They're not for 

identifying drugs; they're for something different. They're for screening presumptive; but there never was, 

except in the 1860s, that's not what they're for.

Similarly when things that scientifically have really been debunked, yet they keep getting into court. And 

the scientists amongst us, all of us, are probably just shaking our heads and saying, "How can this 

continue to happen?"

So my question to the panel is in particular thinking about plea bargain cases, which are the vast majority 

of cases now, how does forensic science get in the game; and how do scientists get in the game, and 

how can they get integrated into the system to help address those kinds of issues? And that's to all of 

you. Thank you.

DAVID WAXSE: It all gets back to the proper education of the lawyers and judges because the lawyers 

involved in those plea negotiations should not let the fact that a field test that's not valid determine 

whether their client pleads guilty.

One of the sad things is the Supreme Court got into this area years and years ago with a Delbert decision 

that said, in essence, scientific approaches need to be taken to evidence; and the simple things of you 

have to have the science in a situation that can be replicated and can be tested. It's now been put in a 

Federal Rule of Evidence; 702 says almost the same thing. And yet, we have all these cases where it's 

not being followed. I hope it's just simply because they've not learned – the lawyers or the judges – that 

it's there and it needs to be followed. The sad thing would be if they recognize it and still ignore it.

BILL FITZPATRICK: Susanne, this may be a little off topic but one of the great problems that we have in 

New York is the dangers now of field testing. Some of these concoctions that are out there can be so 

dangerous to the officer that you almost have to wear a hazmat suit. I'm only speaking for my community.

In other major metropolitan areas, a young man comes in and the field test was positive for X. If he wants 

to get the hell out of there, he pleads guilty, not realizing the ramifications of a long-term criminal 

conviction. That would never happen in my community which, as I said, is about half a million people. We 

would never take a plea until the drug has been properly identified.

But in terms of – and forgive me, I don't know your background; but I would love to discuss this with you 

further if you have some ideas about that – how to improve that system – not only for officer safety, but 

for the accuracy of the results as well.

SUZANNE BELL: It seems like for there to be access to a review of something like that, external to that. I 

don't know if you'd consider making a BS in chemistry a requirement for law school, but that's not going to 

happen. But my concern is how do you say as a defense lawyer or – I mean, somebody is working on a 

plea agreement -- do you have a database where you can go and say, oh, here's somebody who will do 

this for me; who will look at this. Because reviewing a field test result or something along those lines, it 

seems like that's a resource that's not available to you; is that correct – in the plea bargaining stage?

Because your jurisdiction, that's terrific; it may not be that way everywhere else, right? Anyway, that was 

my question, how can we address that?
4 April 10 DOJ Part 4

26

VANESSA ANTOUN: I'll just briefly try to address it – not specifically in the terms of the field tests, but 

generally speaking the use of lab reports in a plea negotiation or what might cause someone to enter a 

plea. Two of the things that have been addressed are the training, but also the communication with the 

forensic analyst who did the results. Letting the prosecutor know this is what it means, letting the defense 

attorney know this is what it means, and having them absorb that information can sometimes be two 

different things.

So I think there needs to, where possible, the lab report reflect as much information as it can to be able to 

explain what that result really means because there are many situations, and will continue to be many 

situations, where it's just the lawyers looking at it. And they don't have the funding or resources to ask an 

expert. So they look at it; they don't understand it; and the consequences can be serious. So hopefully, 

increased use of terminology that is maybe more understandable and part of it at least tailored to an 

understanding from the legal community.

DAVID WAXSE: I think one more little piece is that judges have an obligation to make sure that justice is 

being done. If you get a situation where it looks like both sides have not really done what they should 

have done to make sure this is the correct approach, the judge has an obligation to do that. And the judge 

has tools, as I mentioned in the beginning. Federal judges have the right to appoint their own experts. If 

you have a question as a judge as to what's being done here, you should not just go along with it because 

both sides have agreed; you have an obligation to make sure it's being done correctly.

NELSON SANTOS: Okay, we've got to break for lunch so they can distribute it.

Julia, your tent was up and Matt; and then we'll close it. sorry.

JULIA LEIGHTON: I wanted to address Greg.

I think that there was an important question that you put, where you were essentially asking about 

whether the defense is adopting some of the recommendations. And I wanted to point to that directly.

Actually, I do a lot of training; that's what you do when you retire. You don't do anymore; you teach. And 

we are in the process of planning training where one of the things we're talking about is the various 

recommendations of this Committee, with an eye towards getting the courts to adopt them and therefore 

uniformly apply them.

The reason I don't turn around to you, my client, and say, "Hey, is it okay with you? This jurisdiction let's 

everybody use reasonable degree of scientific certainty. But you and I, we're going to take the high road; 

and we're not going to do it. But by the way, you didn't get to pick me as your lawyer; I was court 

appointed to you." 

So I can't, if you will, unilaterally disarm on essentially the cases of poor people that can't afford lawyers.

What I can do, and what we are doing, is training people about these recommendations with the hopes of 

implementing them through the courts with the understanding that the courts, we hope, will apply them 

uniformly to both sides. And I think that that's the best role the defense can play because we have to 

represent the individual interests of each client, and I can't make policy decisions that might adversely 

impact an individual client. But there is a way to do this through the courts, which is the uniform 

application.

I think the ABA – Matt can address the ABA and what they're doing because I think they also are looking 

for ways to do uniform application.

GREGORY MOTTA: So if I could just say, my question was actually a factual question. Both of the 

witnesses had said they were tracking the documents and the recommendations and so on like that. So 

given the discussions we had here – which is if this language is deceiving to a jury, it's deceiving to a jury 

whether it comes from the mouth of a Government expert or a defense expert. In that sense, it doesn't 

support any end of justice. 
4 April 10 DOJ Part 4

27

So the natural question was these are nationwide organizations that represent lawyers who are outside of 

the Government service. What are they doing to address these issues relative to their old ranks?

With regard to defense experts, there are a number of defense experts who are routinely and regularly 

used. And the question, I think, does have to be asked of the defense expert, even in civil litigation on 

some of the issues we've look at. We haven't addressed civil litigation; but some of the very same issues 

of cause of death occur in wrongful death actions, all kinds of other things that we've discussed here.

So I guess the question really is for this population of lawyers who serve outside of Government service, 

are their associations looking at that and entering into a dialog with lawyers saying, "You need to be 

having these discussions; you're a defense expert. What are your limitations? What do you see? What 

are your reported crimes? How much open discussion are you allowing, and how is it consistent or 

inconsistent with the recommendation coming out of Commission and adopted by the Department?

Because even though we have a limited mandate, unless I heard something wrong, these were issues 

that cut across all plains.

JULIA LEIGHTON: And my point is that we can do that. We can't dictate a policy and impose it on an 

individual defendant; but we can do it, and are doing it, by training and by getting people to make these 

requests of the courts. Then the courts can apply them, and presumably will apply them, to both sides.

GREGORY MOTTA: So I'll make sure I'm arrested in the District of Columbia?

JULIA LEIGHTON: Because...?

GREGORY MOTTA: Because that's where you practice.

MATTHEW REDLE: And, Greg, I'm this year's Chairman of the Criminal Justice Section of the ABA. I 

want you to know that the ABA had already adopted policy by way of a resolution that came through the 

Criminal Justice section on many of the topics that we've weighed in on since and adopted. They have 

been displayed in our publications and available to practitioners; but like, as frequently happens, that 

hasn't been enough; and we're looking at other ways of doing that.

Additionally, it's my understanding that the leak of the PCAST report to the New York Times may very 

well not have come from a PCAST member but rather from some individual in Government.

NELSON SANTOS: You know, I wish I had the standards like they do at the confirmation hearings where 

you get two minutes and then they defer time. It's hard to control. One minute, please, because we've got 

to break.

PETER NEUFELD: Just three things, just to clear them up – one is that – and I appreciate Bill, as an 

honorable person mentioning our work; but I think there are a couple of misunderstandings here.

Professor Gates, there were meetings of the New York Commission; but, frankly, they were dysfunctional.

In contrast to the work of the Texas Commission, that New York Commission not once in 20 years ever 

reviewed a body of cases where wrongful convictions had occurred and attempted to do an audit. In fact, 

whereas 12 states decided to do their own error reviews, New York simply said, "No," and kicked the 

bucket to an Attorney General.

Two, there has been a case recently in New York, contrary to Mr. Fitzpatrick's assertion, that actually did

rely on the thinking of the PCAST report to suppress expert testimony from a handwriting example. It 

happened in the fall of 2016 in New York City.

Three, and most importantly, is I think that Mr. Fitzpatrick's statement on behalf of the NDAA where he 

says that he knows that these disciplines are scientifically reliable, including bite marks, is exactly the 

reason why this Commission, by more than two-thirds vote, decided that those kinds of decisions and 

assessments should not be made by prosecutors, should not be made by defense attorneys, but should 
4 April 10 DOJ Part 4

28

be made by an independent scientific body. And it's why this body, this group, voted to recommend that 

NIST take on that responsibility, just to avoid those kinds of categorical pronouncements by people like 

you and me, who are basically scientifically illiterate and said that in the interim, until NIST can do that, 

that other independent scientific bodies are free to do it as well.

But those decisions about reliability and validity should not be coming from lawyers making those 

decisions, when they don't know better. They should be coming from independent scientists. And I only 

hope that NIST continues to take on that responsibility and other independent scientific groups do it as 

well. Thank you.

NELSON SANTOS: Do you have a comment?

BILL FITZPATRICK: Yes, as you usually do, Peter – as you do in New York – you usually wait till the end 

and think you can get the last word on something. I have never used bite marks in a case. The only times 

I have ever used bite marks is to exonerate an individual. The case that you're very familiar with in my 

jurisdiction, it was actually the withholding of legitimate bite mark evidence that led to a conviction.

I'm not saying that bite marks are the equivalent of fingerprints or DNA. If you listened to what I say, 

exactly what I said, these decisions should not be made by Commissions.

PETER NEUFELD: That's not what you said in the first paragraph, Bill. You said—

BILL FITZPATRICK: I see that nothing about you has changed.

NELSON SANTOS: Excuse me, wait, wait, wait, I wish I had the control that the Chairman at 

confirmation. Okay, that's fine, but it's getting argumentative and it's not productive at this point.

(inaudible)

NELSON SANTOS: I understand, but it's not productive. You said your thing. Let's break. Thank you.

Please come back at 11:40 a.m.