# Presentation: 138414

## BNL’s  Self Assessment Program
Building On our Past
Securing our Future
 “Reducing vulnerabilities” 
QAWG- Spring 2003

**Brookhaven Science Associates****U.S. Department of Energy**

**BNL’s**** **** Self Assessment Program**
**Building On our Past**
**Securing our Future**
** “****Reducing vulnerabilities”**_** **_
_**QAWG- Spring 2003**_

**Roy H. Lebel**

## Self-Assessment Flow Chart

**Institutional Plan**

**    “****The Best In Class Model”**

## Organization Vertical  Assessment  Re-focused

**Brookhaven Science Associates****U.S. Department of Energy**

**Organization Vertical  Assessment  Re-focused**

**BNL’s****  ****best in class **_**Self Assessment**_** program will provide  Organizations with a **_**Clear**_** Vision of Management System performance expectations **
** ****It includes programmatic (horizontal, Lab-wide) as well as organizational (vertical) self-assessments**
**Annually sets the stage for the next generation Self Assessment Plan **
**It  truly works our infrastructure and make our systems “REAL”**

**Roy H. Lebel**

## Assessment Program

**Organizational Self-Assessment **

**Management System/**
**Programmatic Assessments**

**Independent  Assessment**

**Laboratory Organizations**

## Horizontal and Vertical Self-Assessment and Annual Review

**Horizontal and Vertical Self-Assessment and Annual Review**

**Organizational Self-Assessments**
** ****and **
**Annual Review**

**Management**
** ****System Assessments **
**Annual Review**
**&**
**Roll-UP**

**Laboratory Organizations**

_**Near Term Focus **_**:**
**Improve integration between Management System Assessments and organizational assessment programs**** **

## Organizational  Vertical Self Assessment  Re-focused

**“**_**Say what you do- Do what you say”**_
**It Starts with Our Standards Based Management System**
**Management System Evaluation Process**
**Relay Some History**
**Process Validation**
**Self Assessment Part 2 Re-focused**
**A Proposed Path Forward- Securing our Future**
**Using Management System Evaluations as a key information feeder **
**A Proposed FY-03 Performance Measure**

** **2

## Our SBMS

- The Standards Based Management System (SBMS) is the primary tool for setting institutional standards and conveying information.
- The Management System Maturity concept is an integral part of SBMS system development
- It is also the tool by which we develop, integrate and demonstrate our _**conformance**_ to requirements.
- It underpins all of the Laboratory’s contractual requirements (e.g. ISM, QA, EM...)

**Notes:**
Hierarchy - higher levels (MSD) for management; other areas for staff (subject areas)

SBMS itself is a management system to integrate the other 3 and make it work

COMPARISON SHOPPING
PNNL took 4 years to put into place what we have put together in 2
	- leverage their successes
	- learned from their mistakes

## 			OUR SBMS

_**BNL’s Management Systems **__**provide**_**: **
_***“***__***Consistency of Purpose”***_

_**Generate**_ & _***then sustain***_ an efficient integrated management system responsive to BNL’s diverse  environment, that provides consistent, value added, approaches/methods that address and satisfy appropriate requirements (DOE & Management).

## Management Systems

**33 Management Systems At BNL**
- Highest level operating/business processes.
- MS Steward – ALD
- MS Point of Contact (POC) – Division Manager
- Responsible for implementation, maintaining, assessing, and improving MS operation
- Organizations are responsible for _**Conformance**_

- 5

## Management System Evaluations

- Over the past _**TWO**_ years an approach has been developed with DOE and used to evaluate the eight MS that carry BNL’s  Quality Requirements.
- It is a new, Baldrige–based consensus process
- It evaluates the maturity of BNL’s MS as it progresses through system life cycle
- Areas for Improvement and Strengths are identified for each MS.

## THE PHASES  OF SYSTEMS MATURITY

**THE PHASES  OF SYSTEMS MATURITY**

- 10

**Time is =  to a function of **_**Vulnerability**_** to BNL           **

**                                                     **

** **

**Implement   **

**Verify**

**Behavioral Change ****   **

**Performance Improvement**

**Develop**

## Some Results

- Each evaluation resulted in positive feedback, and learning. The process itself is maturing.
- Evaluation Guide criteria statements have been refined
- Scoring ranks changed
- Standard Question Set has been refined
- Increased emphasis on “Objective Evidence”
- Shortened the Workshop by holding a Pre-Workshop meeting for those unfamiliar with the process.
- It has begun to work our infrastructures
- Some Progression Plans developed

_**December 2000**_ - McCallum & Turner – reviewed BNL’s   Management &  Independent processes
- Some areas for improvement noted:
- “Ensure consistency in the implementation Mgt. Assessment process..”
- “Implement a frame work to proactively identify MS & Op areas where targeted
-       independent assessments are warranted”
_**April 2001**_– McCallum & Turner - reviewed the evaluation process and observed the pilot:
- 	“Evaluation process is sound....”
_**September 2001**_ – The Management System Maturity Evaluation Process was jointly reviewed by:
- The DOE Office of Nuclear & Facility Safety
- Battelle Columbus ESH&Q
- “All SBMS Management Systems should be evaluated using the Management System Evaluation process to ensure quality program full coverage for all operational management elements.”

**Process Validation**

## Process Validation

_**December 2001**_ – Third Corporate Oversight Review of Brookhaven Science Associates recommended:
- “Begin the process of institutionalizing the Management System maturity evaluations.”
- “Consider the value of using the Management System maturity evaluation framework as the annual evaluation template for all Management Systems as an integral part of the Integrated Assessment Program.”
_**April 2002**_ – QAP Verification Report (Breckenridge Consulting Group) stated:
- “...BNL Senior Management should consider using this methodology to verify the implementation of all BNL Management System Documents.”

## Our Path Forward

**Management System Planning and Assessment**** **
**& **
**Its Role in the Laboratory’s Self Assessment Program**
- Improve the definition and implementation of the Laboratory’s Management System Assessment Program
- Facilitate better integration of oversight along management systems               (vs. organizations or regulations)
- Enable performance monitoring and communication of programs implemented through integrated management systems (e.g. Quality, ISM, EMS, ISSM etc.)
- Drive effective and efficient development and deployment of SBMS

## Management Systems Stewards

**Management Systems Stewards**

**Management System Planning & Assessment**

**Annual Roll- Up**

**BNL will:**
- Summarize BNL consensus based Management System Reviews & Progression Plans
- Status SBMS as a process
- Develop & report status to Senior Management (& DOE??)
_**As a Result Annually Plan the next Generation SA’s **_
- Target a **#** Management Systems for a Consensus basis evaluation & Roll-Up
- Annual revisit organizational “Required Assessments” with respective Management System Stewards
- Revise Subject Areas before the next planning cycle

**The Annual Roll- Up**

## Slide 18

## Organization  Vertical Self Assessment

- 3.

**Feedback elements**
** ****used for  Annual Roll-Up**

## Organizational Vertical Self Assessment 

- It is a  program in which BNL builds on its Management System structures as well as organizational commitments & accountability towards those systems and other BNL/DOE contractual requirements.
- It meets the Best in Class Self Assessment Model
-   **Multi-Dimensional** Expectation
- Includes organizational (vertical) self-assessments and programmatic/management system (horizontal, lab-wide) self-assessments

## Evolution of the Management System Assessment Program

- Recognized by both senior laboratory management and DOE as requiring attention.
- Confirmed by BSA corporate reviews
- Critical Outcome process used over the last few years to drive some improvement
- Primarily related to management systems that support the QA program (i.e. maturity determinations)
- Three elements of FY03 performance measure
- Definition and implementation of integrated management system assessment planning and evaluation-focus on 11 management systems
- Maturity determinations (4 management systems)
- Third party evaluation of the management system assessment process

## Evolution of the Management System Assessment Program

- Performance Measure “Project”
- Project being led by Quality Programs & Services, in conjunction with IAP and OMS.
-  Goal is two-fold:
- Meet requirements of the Measure
**Continue to strengthen MS Assessment process **_**for BNL**_
**Improve linkage between MS and the Line**
***WORKING THE INFRASTRUCTURE TO MAKE***
***                ******OUR SYSTEMS REAL ***

## Evolution of the Management System Assessment Program

- FY 03 MS Assessment Planning and Evaluation - focus on 11 management systems.
- Drive focus of management system assessments to align with strategic and/or high impact issues (i.e. priorities)

**Acquisition Mgt**
**Emergency Preparedness**
**Environmental Mgt**
**Facility Operations**
**Facility Safety**
**Hazardous Material Trans**

**Life Cycle Asset Mgt**
**Radiological Control**
**Safeguards & Security**
**Work Planning & Control**
**Worker Safety & Health**

## Evolution of the Management System Assessment Program

- Each system steward provided a planning document containing; system status, where they needed to go/scope of work, and  plans for monitoring performance
- Required Assessments of Line Organizations (Depts./Divs.)
- Focused assessments performed by or on behalf of the system steward/lab management
- Management system stewards will provide an annual roll-up for an integrated report.

## Evolution of the Management System Assessment Program

- Required Assessment Table (IAP SA)
- Scope intended to be reviewed/revised annually
- Focus on specific information needed by MS Steward (MSS) to make programmatic determinations (or to satisfy external requirements)
- MSS receives results
- Focused on specific organizations where appropriate
- Schedule requirements where necessary to meet specific requirements
- Aids to be developed to focus line assessment activities
- We are on a substantial learning curve

## Evolution of the Management System Assessment Program

- _Maturity Determinations_
- Process based on the Maturity Evaluations conducted as part of the Quality Program Verification completed in FY02.
- Peer evaluation of MS against 3 criteria – Definition, Implementation, Planning, Assessment & Improvement
- Perform MS Maturity Evaluations on 4 MS:
- Radiological Control – end March
- Worker Safety & Health – May
- Safeguards & Security – end May
- Facility Safety - June

## Evolution of theManagement System Assessment Program

- _Third Party Evaluation_
- Protocol in final development
- DOE BAO to provide input
- Potential team members are being considered and  contacted
- DOE BAO providing input
- Third Party will:
- observe a MS Maturity Evaluation
- interview cross section of MS Stewards, POCs _***AND***_ depts/divs regarding their self assessment plans and activities.

## Evolution of the Management System Assessment Program

- Summary
- We are on a substantial learning curve.
- Working to meet Performance Measure ***AND*** develop the process that is right for BNL.
- Assessment planning for FY04 will begin in June, so the IAP SA and Required Assessments Table is in place by Oct. 03.

## In Summary

**The Benefits of a Re-focused Organizational **
**Vertical Self Assessment Program:**
- It supports the integrated Management System approach by a utilizing a consistent method for System evaluation & Organization Feedback:
- It works the Infrastructure Establishment & Makes it REAL
- It  aids in streamlining our requirements base
- It supports DOE’s  current “Principles for Office of Science Laboratory Contracts”
- Support DOE’s Vision of “World Class” Science & Technology

-  Requirements at BNL become Transparent

## SUMMARY

** **
** **
** **

**Perceived  Requirements**

**Don’t be ITIMIDATED**
**Understand & Support Institutional systems**

## FY-03 Performance Measure

_**Assessment **_**& **_**Improvement**_
- BNL to evaluate FY-02 Consensus based Management Systems Evaluations & develop & document progression Plans
- This Builds on last years work- 6 Plans
- BNL to Identify Three to Five Management Systems for FY-03 Consensus based Maturity evaluation & document progression plans.
- Four selected

## FY-03 Performance Measure  con’t 

_**Assessment **_**& **_**Improvement**_
- BNL targeted 11 Management Systems using a Self based Planning & Assessment approach
-  We develop & distributed a short (1-2 page) Planning & Assessment template
- Templates were completed, reviewed & required organizational line assessments were outlined in our IAP required line assessments
- Combined based on risk & vulnerability
- Line assessment feed back currently happening
-   This is to be used by Management systems Stewards for Annual Management   Systems  Activity & Performance Plan .
- Third party senior member team being set up
- Year End Review  & Roll –Up  for Senior Management