# Presentation: 721393

## Quality Rep. Workshop  (11/30/2005)

_*Background / Discussion*_
_*Improvements/Requirements*_
_*Demo: SES & ISIS*_
_*Conclusion*_

**Revision to the**
** ****“****Evaluation of Seller’s QA Program”**** **
**Subject Area (SA)**
** ****Now: ****“SUPPLIER EVALUATION”**

## Discussion/Background

**Opportunity for Improvement identified during internal assessment of ERD’s procurement quality practices:**
- Pre-Award Evaluation of Supplier’s “QA Program” not performed for risk-significant (A1/A2) procurements as required by current SA.
- SA does not clearly define how to evaluate suppliers without QA Programs (e.g. service providers)
**Further review of risk-significant (A1/A2) procurements from new suppliers found evaluations either not being performed or not documented (no objective evidence) **

## Team Members: 

- Joe Labas (SME), QMO/PPM
- Steve Stein, QMO
- Dave Passarello, CA-D
- Mike Buckley, NSLS
- Stasia Scocca, QMO
- John Selva, E&WMS
- Kevin Corbett, ERD
_**Reviewed by:**_
- Ann Emrick, Life Sciences
- Diane Cabelli, Chemistry
- Pat Carr, EENS

**Notes:**
333

## Drivers: External Requirements

**DOE Quality Order (DOE O 414.1) and**
**Rule (10CFR830.122)**
***Criterion 7 – ***_***Procurement***_
- “Procure items & services that meet established requirements...”
***“******Evaluate and select prospective suppliers on the basis of specified criteria”***
- “...ensure approved suppliers continue to provide acceptable items and services.”
- Criterion 8 - Inspection & Acceptance Testing
- “Inspect and test...using established acceptance & performance criteria.”

## Current Drivers: “KEY” Assessments FY06

**OA - 40 ISM Program Assessment - FY06**
** ****Similar to 2000 verification Assessment**** **
**Large DOE team on site for a long period **
***PERT Review – *****FY06**
- The review will cover all aspects of procurement.
- The criteria include the following:
**BSA has a system in place to use capable and reliable contractors who either have track records of successful past performance or can demonstrate a current superior ability to perform**
- Technical evaluations are adequate and support source selection decisions.

## Discussion/Background

**The Real Question S/B:**
- Do we need to evaluate the supplier beforehand? i.e. Is the supplier **capable** of providing the items / services that meet all “requirements”                 (e.g. technical, quality, cost and schedule)?
- This activity is primarily designed to identify and reduce the risks of doing business with an “unknown” (uncertain) source!
- BEST PRACTICE!

## Improvement Opportunity

-  Former SA more **restrictive** than necessary:
- The QA Order & Rule does NOT specify that an evaluation of the “QA Program” is required
- Many pre-award “evaluations” are being performed which may not include the supplier’s QA Program (e.g. JNCP, PEP)
- Revised SA allows for these other types of evaluations (e.g. technical, environmental, financial, etc.), while meeting the intent of the DOE Order & Rule requirement.

## Improvement Opportunity

- Implement a web-based system for recording evaluations (**SES: Supplier Evaluation System**)
- allows for sharing of information and potential cost avoidance (evaluated suppliers good for 3 years)
- the individual or group making the final supplier/ contractor selection is responsible for conducting and recording the evaluation results in the SES
- will be utilized to ensure evaluations are conducted PRIOR to placing PO.

## Requirements

**Three requirements added: **
- Pre-Award Evaluations required for **Non-commercial **procurements (not from a catalog; not off-the-shelf) _**over**_** $100K** since a financial risk exists.
- NOTE: Since these types of procurements generally
- require some type of evaluation, the impact should be
- minimal.
- Evaluations must be conducted by a **“qualified evaluator”** - the individual or group conducting the evaluation who possesses knowledge of the evaluation criteria and experience in conducting supplier evaluations using the method selected.
- Evaluation Summary must be recorded in BNL’s **Supplier Evaluation System (SES)**

## Supplier Evaluation Summary

- Evaluation of a supplier’s **capability** is required for ALL** Noncommercial** (e.g. custom; BNL specs.) procurements:
- that are **risk-significant** (either A1–Critical or A2–Major)
**OR**
**greater than $100K**, UNLESS:
- The Supplier has a documented, acceptable evaluation within
- the last 3 years (hence the need to SHARE evaluation
- information in SES!)
- For Commercial (off-the-shelf/catalog) purchases that are risk-significant (A1/A2):
- Documented acceptance criteria and inspection/testing is required (in lieu of an evaluation) – 4-Way Match Rule

## Supplier Evaluation Methods

- Several **methods** can be used to perform an evaluation:
- Third Party (e.g. ISIS Evaluation, ISO 9001 Registration)
- Desk-top Review
- Survey, Technical Evaluation, Supplier History
*JNCP (sole source justification)*
*Proposal Evaluation Panel (PEP) report**	*
- On-site Assessment / Audit
-  **There is NO substitute for an on-site evaluation!**

- most effective

- least effective

## The “New” Web Requisition

## Supplier Evaluation Systems 

- A BNL Supplier evaluation must be documented in:
***Supplier Evaluation System (SES)***
- Web-based, central repository for evaluations.
- _[http://130.199.76.90/ppm/ses/ses.asp](http://130.199.76.90/ppm/ses/ses.asp)_
- Follow-up on any deficiencies is necessary if award is to be made.
- Another Source of Evaluated Suppliers:
**Integrated Supplier Information System - ISIS**
- 			_[https://](https://omniweb.ornl.gov/ISIS/)__[omniweb.ornl.gov](https://omniweb.ornl.gov/ISIS/)__[/ISIS/](https://omniweb.ornl.gov/ISIS/)_

## Conclusion

- SA Title changed to “Supplier Evaluation”
- 	_[https://sbms.bnl.gov/sbmsearch/subjarea/12/12_SA.cfm?parentID=12](https://sbms.bnl.gov/sbmsearch/subjarea/12/12_SA.cfm?parentID=12)_
- Subject Area had wide-spread review & input
- Comments solicited from large & small science, support organizations and PPM.
- To be presented to PPM, Quality Reps. and all Department Administrative Staff
- Implemented in SBMS mid-October 2005

## OTHER

- ISIS Lead Auditor Refresher Training ?
- 1-day, mid-December (12/15th or 16th)
- Web Req. QR Checklist ?
- ?