can you help me note take for these sections of this PCAST Report: 

The chapter is organized as follows:
• The first section describes the distinction between two fundamentally different types of feature- comparison methods: objective methods and subjective methods.
• The next five sections discuss the scientific criteria for the two types of scientific validity: foundational validity and validity as applied.
• The final two sections discuss views held in the forensic community.
4.1 Feature-Comparison Methods: Objective and Subjective Methods
A forensic feature-comparison method is a procedure by which an examiner seeks to determine whether an evidentiary sample (e.g., from a crime scene) is or is not associated with a source sample (e.g., from a suspect)102 based on similar features. The evidentiary sample might be DNA, hair, fingerprints, bitemarks, toolmarks, bullets, tire tracks, voiceprints, visual images, and so on. The source sample would be biological material or an item (tool, gun, shoe, or tire) associated with the suspect.
Feature-comparison methods may be classified as either objective or subjective. By objective feature- comparison methods, we mean methods consisting of procedures that are each defined with enough standardized and quantifiable detail that they can be performed by either an automated system or human examiners exercising little or no judgment. By subjective methods, we mean methods including key procedures that involve significant human judgment—for example, about which features to select or how to determine whether the features are sufficiently similar to be called a proposed identification.
Objective methods are, in general, preferable to subjective methods. Analyses that depend on human judgment (rather than a quantitative measure of similarity) are obviously more susceptible to human error, bias, and performance variability across examiners.103 In contrast, objective, quantified methods tend to yield greater accuracy, repeatability and reliability, including reducing variation in results among examiners. Subjective methods can evolve into or be replaced by objective methods.104
4.2 Foundational Validity: Requirement for Empirical Studies
For a metrological method to be scientifically valid and reliable, the procedures that comprise it must be shown, based on empirical studies, to be repeatable, reproducible, and accurate, at levels that have been measured and are appropriate to the intended application.The method need not be perfect, but it is clearly essential that its accuracy has been measured based on appropriate empirical testing and is high enough to be appropriate to the application. Without an appropriate estimate of its accuracy, a metrological method is useless—because one has no idea how to interpret its results. The importance of knowing a method’s accuracy was emphasized by the 2009 NRC report on forensic science and by a 2010 NRC report on biometric technologies.
To meet the scientific criteria of foundational validity, two key elements are required:
(1) areproducibleandconsistentprocedurefor(a)identifyingfeatureswithinevidencesamples;(b) comparing the features in two samples; and (c) determining, based on the similarity between the features in two samples, whether the samples should be declared to be a proposed identification (“matching rule”).
(2) empiricalmeasurements,frommultipleindependentstudies,of(a)themethod’sfalsepositiverate— that is, the probability it declares a proposed identification between samples that actually come from different sources and (b) the method’s sensitivity—that is, probability that it declares a proposed identification between samples that actually come from the same source.
We discuss these elements in turn.
Reproducible and Consistent Procedures
For a method to be objective, each of the three steps (feature identification, feature comparison, and matching rule) should be precisely defined, reproducible and consistent. Forensic examiners should identify relevant features in the same way and obtain the same result. They should compare features in the same quantitative manner. To declare a proposed identification, they should calculate whether the features in an evidentiary sample and the features in a sample from a suspected source lie within a pre-specified measurement tolerance(matching rule).109 For an objective method, one can establish the foundational validity of each of the individual steps by measuring its accuracy, reproducibility, and consistency.
For subjective methods, procedures must still be carefully defined—but they involve substantial human judgment. For example, different examiners may recognize or focus on different features, may attach different importance to the same features, and may have different criteria for declaring proposed identifications. Because the procedures for feature identification, the matching rule, and frequency determinations about features are not objectively specified, the overall procedure must be treated as a kind of “black box” inside the examiner’s head.
Subjective methods require careful scrutiny, more generally, their heavy reliance on human judgment means that they are especially vulnerable to human error, inconsistency across examiners, and cognitive bias. In the forensic feature-comparison disciplines, cognitive bias includes the phenomena that, in certain settings, humans (1) may tend naturally to focus on similarities between samples and discount differences and (2) may also be influenced by extraneous information and external pressures about a case.110 (The latter issues are illustrated by the FBI’s misidentification of a latent fingerprint in the Madrid training bombing, discussed on p.9.)
Since the black box in the examiner’s head cannot be examined directly for its foundational basis in science, the foundational validity of subjective methods can be established only through empirical studies of examiner’s performance to determine whether they can provide accurate answers; such studies are referred to as “black- box” studies (Box 2). In black-box studies, many examiners are presented with many independent comparison problems—typically, involving “questioned” samples and one or more “known” samples—and asked to declare whether the questioned samples came from the same source as one of the known samples.111 The researchers then determine how often examiners reach erroneous conclusions.As an excellent example, the FBI recently conducted a black-box study of latent fingerprint analysis, involving 169 examiners and 744 fingerprint pairs, and published the results of the study in a leading scientific journal.112
(Some forensic scientists have cautioned that too much attention to the subjective aspects of forensic methods—such as studies of cognitive bias and black-box studies—might distract from the goal of improving knowledge about the objective features of the forensic evidence and developing truly objective methods.113 Others have noted that this is not currently a problem, because current efforts and funding to address the challenges associated with subjective forensic methods are very limited.114)
Empirical Measurements of Accuracy
It is necessary to have appropriate empirical measurements of a method’s false positive rate and the method’s sensitivity. As explained in Appendix A, it is necessary to know these two measures to assess the probative value of a method.
The false positive rate is the probability that the method declares a proposed identification between samples that actually come from different sources. For example, a false positive rate of 5 percent means that two samples from different sources will (due to limitations of the method) be incorrectly declared to come from the same source 5 percent of the time. (The quantity equal to one minus the false positive rate—95 percent, in the example—is referred to as the specificity.)
The method’s sensitivity is the probability that the method declares a proposed identification between samples that actually come from the same source. For example, a sensitivity of 90 percent means two samples from the same source will be declared to come from the same source 90 percent of the time, and declared to come from different sources 10 percent of the time. (The latter quantity is referred to as the false negative rate.)
The false positive rate is especially important because false positive results can lead directly to wrongful convictions.115 In some circumstances, it may be possible to estimate a false positive rate related to specific features of the evidence in the case. (For example, the random match probability calculated in DNA analysis depends in part on the specific genotype seen in an evidentiary sample. The false positive rate for latent fingerprint analysis may depend on the quality of the latent print.) For other feature-comparison methods, it may be only possible to make an overall estimate of the average false positive rate across samples.
For objective methods, the false positive rate is composed of two distinguishable sources—coincidental matches (where samples from different sources nonetheless have features that fall within the tolerance of the objective matching rule) and human/technical failures (where samples have features that fall outside the matching rule, but where a proposed identification was nonetheless declared due to a human or technical failure). Forobjective methods where the probability of coincidental match is very low (such as DNA analysis), the false positive rate in application in a given case will be dominated by the rate of human/technical failures—which may well be hundreds of times larger.
For subjective methods, both types of error—coincidental matches and human/technical failures—occur as well, but, without an objective “matching rule,” the two sources cannot be distinguished. In establishing foundational validity, it is thus essential to perform black-box studies that empirically measure the overall error rate across many examiners. (See Box 3 concerning the word “error.”)
BOX 3. The meanings of “error”
The term “error” has differing meanings in science and law, which can lead to confusion. In legal settings, the term “error” often implies fault—e.g., that a person has made a mistake that could have been avoided if he or she had properly followed correct procedures or a machine has given an erroneous result that could have been avoided it if had been properly calibrated. In science, the term “error” also includes the situation in which the procedure itself, when properly applied, does not yield the correct answer owing to chance occurrence.
When one applies a forensic feature-comparison method with the goal of assessing whether two samples did or did not come from the same source, coincidental matches and human/technical failures are both regarded, from a statistical point of view, as “errors” because both can lead to incorrect conclusions.
Studies designed to estimate a method’s false positive rate and sensitivity are necessarily conducted using only a finite number of samples. As a consequence, they cannot provide “exact” values for these quantities (and should not claim to do so), but only “confidence intervals,” whose bounds reflect, respectively, the range of values that are reasonably compatible with the results. When reporting a false positive rate to a jury, it is scientifically important to state the “upper 95 percent one-sided confidence bound” to reflect the fact that the actual false positive rate could reasonably be as high as this value.116 (For more information, see Appendix A.)
Studies often categorize their results as being conclusive (e.g., identification or exclusion) or inconclusive (no determination made).117 When reporting a false positive rate to a jury, it is scientifically important to calculate the rate based on the proportion of conclusive examinations, rather than just the proportion of all examinations. This is appropriate because evidence used against a defendant will typically be based on conclusive, rather than inconclusive, examinations. To illustrate the point, consider an extreme case in which a method had beentested 1000 times and found to yield 990 inconclusive results, 10 false positives, and no correct results. It would be misleading to report that the false positive rate was 1 percent (10/1000 examinations). Rather, one should report that 100 percent of the conclusive results were false positives (10/10 examinations).
Whereas exploratory scientific studies may take many forms, scientific validation studies—intended to assess the validity and reliability of a metrological method for a particular forensic feature-comparison application— must satisfy a number of criteria, which are described in Box 4.
BOX 4. Key criteria for validation studies to establish foundational validity
Scientific validation studies—intended to assess the validity and reliability of a metrological method for a particular forensic feature-comparison application—must satisfy a number of criteria.
(1) The studies must involve a sufficiently large number of examiners and must be based on sufficiently large collections of known and representative samples from relevant populations to reflect the range of features or combinations of features that will occur in the application. In particular, the sample collections should be:
(a) representative of the quality of evidentiary samples seen in real cases. (For example, if a method is to be used on distorted, partial, latent fingerprints, one must determine the random match probability—that is, the probability that the match occurred by chance—for distorted, partial, latent fingerprints; the random match probability for full scanned fingerprints, or even very high quality latent prints would not be relevant.)
(b) chosen from populations relevant to real cases. For example, for features in biological samples, the false positive rate should be determined for the overall US population and for major ethnic groups, as is done with DNA analysis.
(c) large enough to provide appropriate estimates of the error rates.
(2) The empirical studies should be conducted so that neither the examiner nor those with whom the examiner interacts have any information about the correct answer.
(3) The study design and analysis framework should be specified in advance. In validation studies, it is inappropriate to modify the protocol afterwards based on the results.118(4) The empirical studies should be conducted or overseen by individuals or organizations that have no stake in the outcome of the studies.119
(5) Data, software and results from validation studies should be available to allow other scientists to review the conclusions.
(6) To ensure that conclusions are reproducible and robust, there should be multiple studies by separate groups reaching similar conclusions.
An empirical measurement of error rates is not simply a desirable feature; it is essential for determining whether a method is foundationally valid. In science, a testing procedure—such as testing whether a person is pregnant or whether water is contaminated—is not considered valid until its reliability has been empirically measured. For example, we need to know how often the pregnancy test declares a pregnancy when there is none, and vice versa. The same scientific principles apply no less to forensic tests, which may contribute to a defendant losing his life or liberty.
Importantly, error rates cannot be inferred from casework, but rather must be determined based on samples where the correct answer is known. For example, the former head of the FBI’s fingerprint unit testified that the FBI had “an error rate of one per every 11 million cases” based on the fact that the agency was known to have made only one mistake over the past 11 years, during which time it had made 11 million identifications.120 The fallacy is obvious: the expert simply assumed without evidence that every error in casework had come to light.
Why is it essential to know a method’s false positive rate and sensitivity? Because without appropriate empirical measurement of a method’s accuracy, the fact that two samples in a particular case show similar features has no probative value—and, as noted above, it may have considerable prejudicial impact because juries will likely incorrectly attach meaning to the observation.121The absolute need, from a scientific perspective, for empirical data is elegantly expressed in an analogy by U.S. District Judge John Potter in his opinion in U.S. v. Yee (1991), an early case on the use of DNA analysis:
Without the probability assessment, the jury does not know what to make of the fact that the patterns match: the jury does not know whether the patterns are as common as pictures with two eyes, or as unique as the Mona Lisa.122,123