and NFPA 1033 as well as several ASTM general forensic standards as the Inspection
Practice/Guide/Procedure definition documents. It cites no particular editions of any of the documents.
NFPA 921 is the standard of care document within the field of fire and explosion investigation. Similarly,
NFPA 1033 defines the qualifications and job performance requirements for fire and explosion
investigators. According to A2LA, the definition of the scope of the accreditation to only fire scene
inspection was set by the organization seeking accreditation. Because 9211's scope is the entirety of the
fire and explosion investigation, the accreditation process utilized only the portions of 921 relevant to
fire scene inspection. The accreditation documentation does not reflect what portions of 921 were used
in the accreditation. Verbally, A2ZLA acknowledged that this is problematic. More fundamental, however,
is that it is professionally inappropriate to reduce the scope of activities to a subset of 921. The fire
scene examination is merely a subset of the data collection process. There are additional sources of data
a fire and explosion investigator needs to pursue that are not scene related. Further, the entirety of the
scientific method process used in 921 goes well outside the fire scene context. Based on 921, a fire
origin and cause investigation that is limited to fire scene inspection is inherently defective.
Nonetheless, this is the scope of the accreditation. The accreditation fails to provide clients and courts
the assurance that the fire investigation unit will provide the services that they require.

Because of the limitations and deficiencies inherent in using 17020 for forensic science, AZLA developed
specific requirements for forensic examinations (A2LA 318/2015) and an associated checklist (A2LA
C318/2015). Neither of these documents references any consensus standards for forensic science
practice. The only referenced document that is not internal to A2LA is LAC G19:08/2014. In essence
A2LA needed to develop accreditation requirements for forensic examinations for themselves without
the benefit of any consensus process. Without judgement with respect to the adequacy of the
developed requirements, | would suggest that it is inappropriate for accreditation bodies to develop
themselves the requirements for accreditation. They should limit their work to defining how the
external consensus document requirements should be implemented in their accreditation processes. To
A2LA’s credit, they are working to remedy these problems that arise out of both their inexperience and
the lack of fit by 17020 to forensic investigation.

There seem to be lessons from our experience thus far.

1) First and foremost is that forensic science is not conformity assessment and 17020 is a bad fit with
forensic science. That is not to say that there may not be elements of 17020 that would serve a forensic
science standard. These can be included specifically by reference to the relevant section of 17020 or
included directly within forensic science standards. ISO 17025 should be examined further to evaluate if
it has similar problems.

2) The forensic science community should define the basis for accreditation through the development of
consensus standards. Accreditation organizations should simply follow the forensic science community.
The scope of an accreditation cannot be left to the client to define. They must be forced to select scopes
that represent complete investigative disciplines that make sense with respect to actual forensic science
practice and which meet the needs of clients and the courts. Forensic science professional organizations
must engage with accreditation organizations. Accreditation organizations inherently lack the
knowledge base to define the accreditation requirements and their implementation. Accreditation
organizations are process experts, professional organizations are content experts.



for inclusion in the OSAC Registry has evaded the processes set by OSAC to assure due consideration,
The QIC should examine the process employed and determine if process changes are indicated.

The subcommittee did not participate in the development of the registry forms. We did not vote on the
forms. We had only a single meeting in the entire history of the subcommittee and that was done at the
very last minute in response to my objections. The response to my concerns for process were pro forma
at best and could be regarded as dismissive. OSAC is its processes and this virtual committee has shown
that the processes at OSAC are broken. There had been NO oversight of the operations of the virtual
committee. The QIC and FSSB have been surprised by my commentary on the lack of process. No one
was watching and no one was checking. As a result, the processes were not followed. ISO 17020 should
be rejected for inclusion in the registry at this time if for no other reason than the process failures.

ISO 17020 should also be rejected on the merits. It will do harm to forensic science practice. Conformity
assessment inspections are examinations of materials, products, installations, plants, processes, work
procedures, or services and the determination of their conformity with requirements and the
subsequent reporting of results of these activities to clients or authorities (ISO 17020/2012). Central to
this process is the definition of the requirements. These requirements are in essence the conditions
expected by the client. There are definite presumptions of what should be present for inspection. This is
antithetical to forensic science work and the scientific method that underlies our work. We approach
our investigations without preconceived expectations. Equally important, conformity assessments
simply report deviations from the requirements or expectations. There is no aspect of determination of
a finding from the data. Central to our work is the application of the scientific method to collect data,
formulate hypotheses, and test hypotheses. These critical forensic science activities have no role in SO
17020 or conformity assessment in general.

The use of a non-forensic, general purpose conformity assessment standard has other unintended and
negative outcomes. While the accreditation is typically identified at a 17020 accreditation, the
substance of the forensic science requirements must arise in reference documents identified for a
particular accreditation. Each and every forensic science body accreditation will be a custom job. This
means that the courts will need to examine the detailed specifics of an accreditation in each and every
instance. This puts an unwanted and unnecessarily burdensome requirement on the courts. The custom
nature of the accreditation can actually result in accreditations with inappropriate scopes that do not
support quality, complete forensic science work.

Subsequent to the virtual subcommittee’s work, | encountered what | believe to be the first
accreditation of a fire investigation unit in the US. The accreditation of a fire investigation unit was
undertaken using 17020 by A2LA. | discussed the process with A2LA, reviewed documentation of their
process provided to me by A2LA, and conducted a site visit at A2LA to further explore the accreditation
process and the particular implementation of that process. This process reinforced my concerns with the
17020 approach to accreditation. | want to thank A2LA for taking the time to educate me concerning
their processes and their openness to hearing my concerns. Again, | have no reason to question the
good intentions of A2LA. They were simply doing their best to implement a model approach to forensic
science unit accreditation defined by their industry group (ILAC).

The accreditation documentation for the subject accreditation states that the accreditation is for
forensic inspection. It limits the inspection to fire scenes (underline included in the A2LA document). The
document further defines the inspection as for fire cause and origin determination. It cites NFPA 921



3) Accreditation of fire and explosion investigation units is at this time premature. Based upon
discussions in our SAC, this is likely the case in most if not all crime scene disciplines. In order to
encourage the development of accreditation processes that are suitable, the OSAC Subcommittee on
Fire and Explosion Investigation has proposed the development of a new standards making committee
at NFPA to develop a standard for Fire and Explosion Investigation Units that will be suitable for use by
accrediting agencies. This consensus standard will be available for use in lieu of 17020 and will
specifically address the needs of the fire and explosion investigation community and its stakeholders.
This is the appropriate path for accreditation documents in the forensic investigation community. We
should avoid the force fit of investigation into a conformity assessment standard.

4) OSAC needs to review its process requirements to assure that candidate documents for registry
inclusion are robustly discussed and true consensus is obtained. The QIC seems to be an appropriate
committee to conduct this review. The FSSB needs to take this issue on directly and immediately. The
reputation of OSAC is at stake and the quality of forensic investigation will suffer if the issue is not
addressed.

The OSAC consensus process for considering ISO 17020 should be restarted and the whole notion of
including {SO 17020 in the registry should be reconsidered.



\ ue

%,
UU
oe aren Go!

PREWSIG 5O°

Scott Oulton.
OSAC Virtual Subcommittee #1 Chair
March 28, 2017

Subject: Final Determination of the Ad hoc Independent Review Panel

The Ad hoc Independent Review Panel has completed its evaluation of comments,
adjudication response, and appeal received for the document listed below in accordance with
RA-1800 Public Appeals.

e ISO/IEC 17020 Comment adjudication completed on September 13, 2016

A summary of the appeal, comments from the Ad hoc Panel, and final determination are as
follows.

1. Summary of Appeal:

An appeal was submitted and received on February 16, 2017 from Dr. Craig Beyler,
Chair of OSAC Fire and Explosion Investigation Subcommittee, The appeal is
attached for reference; however, can be summarized as follows: ISO/IEC 17020 isa
conformity standard and is not applicable to forensic science and the virtual
subcommittee did not allow sufficient opportunity to discuss prior to comment
adjudication.

a. Comments from the Ad hoc Panel:

The Ad hoc Panel participated in a conference call on March 27, 2017 where
this appeal was discussed. The appeal contained opinions on the relevance of
ISO/IEC 17020 which were outside the scope of the appeals process. The
panel agreed that the virtual subcommittee appropriately addressed the content
of the comments during the registry approval process by providing
clarification on the use of the term investigation and inspection in ISO/IEC
17020. In addition, the comments provided addressed the concerns that the
document lacked guidance on findings from an inspection. The Ad hoc Panel



Sincerely,

agreed that the comments were non-persuasive and that rationale was provided
by the committee action. There was also documentation to support that the
RA-700 procedure was followed, and that the virtual subcommittee was
appropriate in their response to the comments provided during the registry
approval process. This documentation has been provided for reference.

b. Final Determination:

This appeal is denied. The Ad hoc Panel recommended that in order to address
the concern about allowing appropriate time for comment, the virtual
subcommittee process should be updated to allocate sufficient time to discuss a
document and comment on a document prior to adjudication.

In regards to the appeal itself, the Ad hoc Panel recommends - No Action.

ISO/IEC 17020 and the initial “negative” comment from Dr. Craig Beyler was
reviewed by five Scientific Area Committees which consists of over 70
stakeholders. The five Scientific Area Committees ultimately decided to approve
ISO/IEC 17020 for listing on the OSAC Registry, despite reviewing Dr. Beyler’s
comments. Dr. Beyler’s concern about adequate time being given for discussion,
comment and comment adjudication at the virtual subcommittee level was noted;
however, when factoring in the votes of the five Scientific Area Committees and
the evaluation by the virtual subcommittee, the Ad hoc Panel recommends no
additional action from the process.

Kris Cano

Ad hoc Panel Chair

Panel Members:

Bruce Houlihan, QIC representative

David Fowler, MD MDI representative

Karl Morgan, Fire and Explosion representative

John Paul Jones, FSSB/OSAC affairs representative



Appeal regarding Public Comment on ISO 17020

Craig Beyler, Chair OSAC Subcommittee on Fire and Explosion Investigation and member of the Virtual
Committee that approved 17020
16 February 2017

Comment Submitted:

17020 was developed for bodies inspecting to determine compliance with expectations. This is not
useful for use in forensic investigations in which we seek to avoid any expectations. There is potential
value in a standard for forensic investigation, but sadly this document does not address this. 17020 does
not address the development of findings from an inspection. The work product is simply a listing of
variations from the expectation.

Subcommittee Chair Response:

Regarding the ISO/IEC-17020 standard for forensic agencies, the term Inspection applies to crime scene
investigations and/or examination of forensic evidence. The term Inspector also applies to an
investigator who uses professional judgment to examine a crime scene with the aim to contribute to
determining what, where, when, how, and why something happened and who was involved. Of those
US-based accrediting bodies that offer accreditation to forensic organizations, two offer accreditation
and have accredited forensic organizations to ISO/IEC 17020. ISO/IEC 17020 is referenced in the
International Laboratory Accreditation Cooperation (ILAC) Guidance document, ILAC G19:08/2014
Modules in a Forensic Science Process., http://ilac.org/?ddownload=805 which is used globally by
Accrediting Bodies around the world. ISO/IEC 17020 addresses the crime scene/examination process
and while it does not further address the analysis of the findings, that does not make the document non-
relevant nor not useful in its application to crime scene investigations. It just speaks for the need for
another standard to be developed that addresses this stage of the investigation. As such, the comment
received is considered non-persuasive and the standard should be posted on the registry.

Rationale for the Appeal:

ISO 17020 is a conformity assessment standard for bodies performing inspections. Forensic science is in
no way conformity assessment. The accreditation community via ILAC has taken it upon themselves to
make decisions for the forensic science community with respect to the appropriate basis for forensic
science body accreditations (ILAC-G19:08/2014). While | can accept that they did this with all good
intentions, they have force fit us into a conformity assessment standard that is not consistent with our
activities. It is time for us to assert ourselves and define standards for ourselves and insist that the
accreditation community follow our lead.

The virtual subcommittee to move ISO 17020 and ISO 17025 through the registry process did not do
appropriate due diligence in their consideration of the standards. As my comments as a member of that
subcommittee reflect, the subcommittee did not take the time to fully explore and question the
standards and their appropriateness for our purposes. A single phone conference was convened on
short notice only days before the vote deadline. This was inadequate and ill-considered. Comments
were dispositioned by the chair alone, rather than by the subcommittee. | do not question the good
intentions of the chair, but { do take issue with the process. My public comment, similarly was voted
upon by the virtual committee with no meeting. The response was formulated by the chair and
summarily put to a vote without any form of discussion or consideration. The entirety of the processing