November 29, 2017

REFERENCE:

Adjudication of LRC comments on ASTM E-2927 by the Materials/Trace Subcommittee

The Materials Trace Subcommittee (MTC) would like to note that the test methods E2330, E2926 and

E2927 are highly correlated as all of them address procedures for one of the steps of the forensic

examination of glass: the measurement and comparison of the elemental composition of glass fragments.

As such, these methods have similar scopes, limitations and implications. The main difference between

these test methods is their analytical performance, where E2926 (u-XRF method approved in the OSAC

registry) is less sensitive and less precise than ICP-MS and LA-ICP-MS (E2330 and E2927) and

therefore requires different considerations for the comparison criteria method. Most of the concerns

previously expressed by LRC, STG and/or SAC during the two-plus years of the OSAC revision process

apply to all these three methods, and therefore some of the concerns that resurfaced during the public

comments period of ASTM E2927-16 have been already addressed at different stages during the revision

process of the approved E2926-16 method.

LRC comments on ASTM E 2927

The Chemistry SAC has proposed the addition of the recently revised "Standard Test Method for

Determination of Trace Elements in Soda-Lime Glass Samples Using Laser Ablation Inductively Coupled

Plasma Mass Spectrometry for Forensic Comparisons" (ASTM E2927-16) to the OSAC Registry of

Approved Standards. On the basis of the information that has been provided to us, we believe that, with

two modifications to avoid unintended or potentially undesirable legal impacts, the standard can be

included in the Registry of Approved Standards. Without these modifications, we do not believe the

standard is ready for the Registry.

In urging the modifications, we appreciate the document is not intended as a complete guide for reports

and testimony. Nevertheless, the standard's existence and its wording will affect the way in which

evidence from laser ablation inductively coupled plasma mass spectrometry (LA-ICP-MS) is developed

and presented in courts, plea discussions, and investigations.

The standard also can be cited as evidence of scientific validity and general acceptance of all aspects of

the test method. We have not conducted an independent review of the references to verify that validity has

been established in the peer-reviewed literature. These comments assume that the test method has been

validated by studies showing the extent to which it correctly identifies fragments of glass as coming from

the same source and from different sources. Neither do these comments address the standard from the

perspective of a copy editor seeking to shorten and make other stylistic improvements in sentences.

Recommendation I: Define Crucial Terms that Have Legal Implications

The standard adopts a sharp cutoff based on a "match interval" (§ 11.1.7) for determining whether two

fragments are "distinguishable" and hence "did not originate from the same source of broken glass" or are

"indistinguishable" and hence may have "originated from the same source of glass." (Introduction). It

adds that "laser ablation inductively coupled plasma mass spectrometry yields high discrimination among

sources of glass" (id.) and that "[t]he determination of elemental concentrations in glass provides high

1
discriminating value" (§ 5.2). Since features that are "highly discriminating" enable one to attribute a

fragment to a particular source as opposed to other potential sources, the standard itself strongly

encourages examiners to report (or readers to infer) that a finding that two fragments are

"indistinguishable" essentially proves that they are from the same source. Moreover, the emphatic

statements of high discrimination for the test method and match rule imply that that a questioned sample

falling just outside the match interval is highly unlikely to have come from the same source as the known

sample, whereas a questioned sample lying just inside the interval is highly likely to have come from the

known sample. Yet, the probative value of the evidence is not particularly different in these two situations,

and the standard should be written to prevent counsel or witnesses from using it to dismiss as of no value

a questioned fragment that lies just outside the interval. This clarification could be accomplished by

removing the references to "high discrimination among sources" or by defining the crucial terms

"distinguishable" and "indistinguishable. "For example,

Distinguishable means that the elemental composition of two fragments falls outside the match interval of

Section 11. Such differences occur only approximately W% of the time when comparing fragments from

the same source of broken glass and approximately X% of the time when comparing fragments from

different sources. Using a slightly different match interval will not make a large difference in the two

percentages.

Indistinguishable means that the elemental composition of two fragments falls inside the match interval of

Section 11. Such differences occur only approximately Y% of the time when comparing fragments from

different sources of broken glass and Z% of the time when comparing fragments from the same source.

Using a slightly different match interval will not make a large difference in the two percentages.

Presumably, the values of W, X, Y, and Z can be taken from the validation studies cited in the Standard.

(If not, a question as to the admissibility of evidence produced in accordance with this standard for

declaring matches would arise. See, e.g., Legal Resource Comm., Memorandum in Response to a

Question on Hypothesis Testing in ASTM 2926-13 and the Legal Principle That False Convictions Are

Worse Than False Acquittals, Oct. 7, 2016, reprinted 130 Harv. Rev. Forum 137, 143-44 (2017) ("we note

that a report of a match without more information about the probability of a match to other glass in the

relevant population would not fulfill the expert's role of impartially and adequately educating the trier of

fact about what the scientific measurements establish.")).

RESPONSE (I): Previously considered

The terminologies are out of the scope of this standard test method. Other standard guidelines are more

appropriate documents for defining these terms (e.g., interpretation guideline and the overarching

guideline for examination of glass currently being developed within the OSAC MTC). The glass task

group within the materials (trace) subcommittee considers that the terms “distinguishable’,

“indistinguishable” and “discriminating” are well known and well understood. Moreover, these terms

appear in the relevant peer-reviewed scientific literature that supports the test method, and some are cited

within the test method. Therefore, adding these terms in the document will not add to the overall

substance and utility of the current standard. Moreover, when the test method is followed, as written, the

measurement outcomes are expected to yield predictable and known analytical figures of merit such as

uncertainty boundaries, precision, bias and limits of detection. Finally, the test method cites the expected

error rates when the recommended match criteria are used for the comparison of glass samples and,

importantly, these error rates are based on interlaboratory trials from a substantial number of operational

and academic laboratories with expertise in the field of glass analysis and comparisons.

The MTC consider that the interpretation of the LRC “...the standard itself strongly encourages

examiners to report (or readers to infer) that a finding that two fragments are "indistinguishable"

2
essentially proves that they are from the same source”, is not a reasonable interpretation and indeed it is

an overstatement that contradicts the fundamentals behind the practice of forensic glass examinations.

Importantly, the E2927 test method does not, in any way, suggest that a “match” infers any particular

weight to the comparison and certainly not that a positive match “essentially proves that they are from the

same source”.

The reasoning supporting our opinion has been provided in previous communications within the OSAC

revision process: 1) MTC general comments in response to LRC comments regarding ASTM E2926 and

E2330 submitted October 2015 (see response #1 and #2), 2) MTC specific comments, in response to LRC

comments regarding E2330-12 (October 2015, see highlighted sections on pages 2, 5,9), 3) MTC specific

comments in response to LRC comments regarding E2926-13 (October 2015, see highlighted sections on

pages 4 and 9), 4) MTC response to STG recommendations to FSSB on E2330 and E2926, submitted

August 2016 (see highlighted sections). The basis for these responses applies to ASTM E2927-16. In

addition, the pertinence of adding these terminologies in a test method has been discussed by the MTC in

several occasions with the STG and LRC during the QIC-coordinated virtual meetings.

Recommendation II: Clarify Whether the Method of Interpretation Is Meant to Be Exclusive

Section 11.1 states that "[t]he procedure below shall be followed to conduct a forensic glass comparison

using the recommended match criteria is as follows (8-11) ... ." There seem to be missing or extra words

in the sentence, but the phrase "shall be followed" suggests that only one procedure for comparing glass

fragments is permissible. This procedure only allows two categories of results "distinguishable" or "not

distinguishable." To the extent that an OSAC-approved standard will be more powerful than the existing

ASTM standard, it will have a more preclusive effect on other ways of reporting test results, even though

a substantial number of forensic scientists and statisticians believe that a "weight of evidence" approach is

superior. This may be what the SAC desires and can justify. Or it could be that the SAC only wishes to

endorse the particular match-interval method as the best procedure among those for categorical matching.

For LA-ICP-MS to be applied most effectively in the criminal justice system, the standard should be

clearer on this point. The Introduction or Section 1 could indicate whether the standard is specifying that

whenever LA-ICP-MS is used for forensic comparisons, the measurements must be interpreted as

outlined in the standard or whether it is merely recommending this particular interpretive method as the

best for laboratories that choose to characterize the measurement obtained from the test procedure in

binary terms such as "distinguishable" and "not distinguishable."

RESPONSE (II): Persuasive

A) A typo in sentence 11.1 was identified and corrected: The MTC would like to note the current 2016

version of the E2927 test method has a typo identified by the LRC. In order to fix the typo, an editorial

change was submitted to ASTM and the sentence will be corrected as follows: “The procedure below

shall be followed to conduct a forensic glass comparison when using the recommended match criteria is

as follows” This editorial change should correct the concern expressed by the LRC. The method is a

recommended match criteria and does not prevent the use of alternative method, as far as the user can

demonstrate its performance and validity are similar or better than the proposed approach.

The MTC would like to stress that an editorial change through the SDO does not require a new ballot of

the document, as the editorial change does not affect the substance of the document.

B) The MTC would like to clarify that the fact the comparison method is not meant to be exclusive as has

been previously considered. The reasoning supporting our opinion has been provided in previous

3
communications within the OSAC revision process: 1) MTC response to SAC notification of

non-approval justification of ASTM E2927-13 submitted October 2015 (see response #2). The MTC also

clarified to LRC and STG members during the QIC-coordinated virtual meetings that the method is not

meant to be exclusive, and other methods that provide similar or better performance in terms of error rates

than the recommended method can be used in the practice.

4