# Presentation: 692830

- CLOSE THE LOOP

- Tonya Petty
- 5-7-2008

## WHY ASSESS?

- Improve the overall program.
- Assure requirements have been appropriately implemented in procedures.
- Assure work is being performed in accordance with procedures and management expectations.
- Assure corrective actions have been implemented and are effective.
- Required by DOE O 151.1C, DOE O 226.1A, DOE O 414.1C, and 10 CFR 830.122(i).

**Notes:**
DOE O 151.1C states:  7.a.(1)  Self-assessments. The contractor must conduct an annual self-assessment of their emergency management programs. Program and exercise evaluations (including appraisals and assessments) must be based on specific standards and criteria, issued by the Director, Office of Emergency Operations. Self-assessment results must be documented in the ERAP submitted to the Cognizant Field Element.   7.a.(3)  Performance Indicators. Contractor facilities/sites must participate in a program of performance indicators (including performance measures and metrics) to capture and track objective data regarding the performance of emergency management programs in key functional areas.   7.b.(1)  Corrective Actions. These requirements supplement those in the CRD to DOE O 414.1A. Continuous improvement in the emergency management program results from implementation of corrective actions for findings (e.g., deficiencies, weaknesses) in all types of evaluations, including both internal and external evaluations.  7.b.(1)(a)  Corrective action plans must be developed within 30-working days of receipt of the final evaluation report. Corrective actions must be completed as soon as possible. Corrective actions addressing revision of procedures or training of personnel should be completed before the next annual self-assessment of the program.  7.b.(1)(b)  Completion of corrective actions must include a verification and validation process, independent of those who performed the corrective action, that verifies that the corrective action has been put in place, and validates that the corrective action has been effective in resolving the original finding.
DOE O 226.1A states:  2.b.(1)  The contractor assurance system will address the criteria described in Appendix A, or other comparably effective criteria established by responsible DOE line management, for assessments (including self-assessment...)  Appendix A 1.b.(1) Assurance activities must encompass...emergency management and must include assessments  (including self-assessments...) (2)  incident/event reporting...(4) issues management...including identification, tracking, closure of corrective and actions and verification of effectiveness...(5)  lessons learned programs (6)  performance indicators.  2.a.(5) Self-assessment results will be documented...e. ...must have a structured, documented, self-assessment program for emergency management...
DOE O 414.1C states:  addresses assessments from HS-63 and the corrective action plan requirements supplemented by DOE O 470.2B.
10 CFR 830.122(i) states:  Ensure managers assess their management processes and identify and correct problems that hinder the organization from achieving its objectives.

## TYPES OF ASSESSMENTS

- Self-Assessment
- Independent Assessment
- RA/RV Process
- Quality and Performance Assurance Division
- External Assessment
- HS-63
- DNFSB
- CDNS
- IG

**Notes:**
EMD conducts at least 15 self-assessments on each program element annually.  Additional self-assessments include the Emergency Services Dispatch Center, Nuclear Incident Response Program, and Operations and Safety Systems Monitoring.

Independent assessments take place from internal organizations at the plant.  The primary division responsible for assessments is Quality and Performance Assurance.  PXSO shadows several of the EMD assessments.

Pantex has implemented a Readiness Assurance/Readiness Verification process that requires assessment to confirm readiness for the startup and restarts of nuclear facilities, nuclear or nuclear explosive operations, or hazardous non-nuclear facilities/activities.

EMD coordinates assessments for external organizations, most recently the HS-63 – Office of Emergency Management Oversight (who reports to) Office of Health Safety and Security
Defense Nuclear Facilities Safety Board
Chief of Defense Nuclear Safety
Inspector General

## STEPS PRIOR TO AN ASSESSMENT

- Develop an annual assessment schedule.
- Prepare an assessment plan.
- Assign the assessor/team leader and assemble the team members.
- Conduct an entrance meeting.

**Notes:**
At Pantex we create a schedule that includes all the planned assessments for that fiscal year.  This schedule is reviewed and approved by the Pantex Site Office.

To formalize the assessment plan, Pantex uses a published template.

To maintain consistency, the Team Leader for all EMD assessments at Pantex is the EMD Readiness Assurance Specialist.  Other team members include the EMD Manager, the EMD Program Administrative Specialist, the functional manager, the element lead, and other impacted personnel.

Entrance Meetings are conducted prior to each assessment with participants using a PX-5353.

## STEPS FOR DEVELOPING ASSESSMENT PLAN

- Introduction
- Purpose
- Scope
- Methods
- Participants
- Requirements Table

**Notes:**
The introduction should identify the program/element/activity that will be assessed.

The purpose explains why the program/element/activity is being assessed.

The scope should state the specific boundaries the assessment will encompass.

Methods include procedure reviews, personnel interviews conducted, and performances observed.  During the planning phase, procedures should be gathered and reviewed and interviews should be scheduled.  Pantex develops a fiscal year drill and exercise schedule so assessments can be planned accordingly.

As mentioned, the Team Leader for all assessments is the EMD Readiness Assurance Specialist.  Other team members include the EMD Manager, the EMD Program Administrative Specialist, the functional manager, the element lead, and other impacted personnel.

Requirements for the Emergency Management Program are primarily DOE O 151.1C; with DOE O 153.1 for the Nuclear Incident Response Programs; for the Operations Center it is primarily DOE O 5480.20; and lastly, the Emergency Services Dispatch Center is NFPA 72, 1011061, 1500.  All programs are assessed to 10 CFR 851 and 19 CFR 1910.

## STEPS FOR CONDUCTING AN ASSESSMENT

- Conduct the assessment to include:
- Procedure Reviews
- Record Reviews
- Personnel Interviews
- Performance Observations
- Training Validation
- Corrective Action Verification/Validation
- Identify and document assessment results in an assessment report.
- Track corrective actions to closure.

**Notes:**
The first step of an assessment is to review documentation and procedures, and assess as to the content and presence of requirements.  This is followed by interviews of personnel required to utilize those procedures and then by demonstration of performance.  Assessments can be further carried out to identify and verify training of the individuals performing the work.  Corrective action verification involves a review of the identified corrective actions to ensure they have been completed/implemented.  Corrective action validation involves a review of the completed/implemented corrective action to ensure it is effective and furthermore, prevents recurrence.

The last action to perform in an assessment is to ensure that all the processes above are maintained and captured in a records system.

If any corrective actions have been identified, they are entered into an electronic workflow management tool,  the Problem Evaluation Request/Electronic Suspense and Routing System (PER/ESTARS). The PER system interfaces with the ESTARS.
The PER allows multiple users to access a single database simultaneously.  Submitting a PER results in an ESTARS task being created and routed to the appropriate person.
The ESTARS is used to create, route, track, and report real‐time task coordination. 
ESTARS offers the capability of tasking, requesting, approving, while leveraging your document management and records management tools. 

## STEPS FOLLOWING AN ASSESSMENT

- Identify and document assessment results to include:
- Strengths
- Findings
- Deficiencies
- Weaknesses
- Observations
- Conduct an exit meeting.
- Develop assessment report.
- Identify corrective actions.
- Track corrective actions to closure.

**Notes:**
Strengths are an assessment conclusion that a product, program, or activity is especially effective and it can be supported with clear and observable evidence of effectiveness.

Findings described a failure-related criterion and are identified as either a weakness or deficiency.
Deficiency – an actual or projected failure to meet an evaluation criterion, thereby directly impacting the associated basic emergency management activity. 
Weakness – an actual or projected failure to meet an evaluation criterion, thereby directly impacting or indirectly impacting the associated basic emergency management activity.

Exit Meetings are conducted after each assessment with the assessment participants and the corrective action owner.
Exit Meetings and the results are documented on a PX-5353.

PER/ESTARS is used to enter, track, and document closure of corrective actions.

## STEPS FOR DEVELOPING ASSESSMENT REPORT

- Introduction
- Purpose
- Scope
- Executive Summary
- Results
- Participants
- Methods
- CRADs table

**Notes:**
The introduction should state the program/element/activity that was assessed.
The purpose should be a brief statement concerning the specific reason the program was assessed and the scope should state the specific boundaries that the assessment encompassed.
The executive summary should be brief and able to stand alone, describing the programs, systems, and processes assessed.
The results should describe the strengths and weaknesses affecting the assessed organization, including barriers to performance, so that meaningful action can be taken for improvement.
Participants are listed, with title, and area.
Write a brief statement concerning the methods that were used to gather the information for the evaluation (i.e., documents reviewed with a list to include title and issue; personnel interviewed to include names, titles, and dates; and performance observed to include date observed and location.
It also includes a previous assessments section to include any past findings.
Lastly, the CRADs table that was developed to identify the assessment criteria and requirements in the assessment plan is completed and attached in the assessment report.

## ASSESSMENT CRAD TABLE

**Notes:**
This is an example of the Pantex EMD Assessment CRAD Table.  The first two columns identify the criteria/requirement we are assessing against.  The next column identifies the Integrated Safety Management criteria that is included in the assessment.  The fourth column is broken down into three rows for each criteria/requirement to show what method was used to obtain the assessment information (i.e., procedure review versus interview versus performance observation.  The fifth column indicates what was reviewed, what the result of the review was, and whether or not there were any previous findings.  The sixth column indicates whether the criteria is met or not.  If the criteria was not met, the seventh column is used to indicated the PER/ESTARS tracking number.

This is a new format for the CRAD table due to our recent HS-63 Inspection.  It was determined we did not provide as many performance-based observations in our assessments, so this will become a metric that will be included in next year’s Performance Indicators.

## CLOSE THE LOOP

- Initiate corrective actions for _all_ findings.
- Assure _timely_ completion of corrective actions.
- Document the assessment.
- Maintain assessment records.
- Incorporate assessment information in the annual Emergency Readiness Assurance Plan (ERAP).
- Revisit previous findings in the next applicable assessment.

**Notes:**

From October, 2005 to present, the Pantex Emergency Management Department has issued/identified 1656 corrective actions.  One corrective action was missed when the document was transitioned from a plan to a WI.

Of those 1656 corrective actions, 1448 were closed on time, with 208 being closed late.

Assessments are documented on a PX-5641.  This form lists all the records associated with the assessment from the schedule developed for the fiscal year outlining planned/scheduled assessments to the entrance/exit meetings and through the corrective action closure evidence.
Data from the fiscal year and the annual assessment results are documented in the ERAP, which includes a summary of the activities of the previous year and planned activities projected for the next fiscal year.  The ERAP is updated annually to provide a summary of short-term accomplishments and long-range planning.  The template for the ERAP is sent from the National Nuclear Security Administration (NNSA), Office of Emergency Operations (NA-43).

As mentioned in the previous slide, if there was a finding previously identified in a past assessment, it is documented in the Assessments CRAD Table and validated to determine the effectiveness of the corrective action.

## INCORPORATE OTHERAREAS IN ASSESSMENTS

- Maintain performance metrics
- Include Integrated Safety Management

**EMD CORRECTIVE ACTION STATUS**

**Basis for Performance** **Indicator**
- The Corrective Action Status chart provides information regarding “continuous improvement in the Emergency Management Program from implementation of corrective actions for findings (e.g., deficiencies, weaknesses) in all types of evaluations including both internal and external evaluations.  Corrective Actions must be completed as soon as possible.” DOE O 151.1C CRD 7.b.(1).

**Notes:**
Pantex reports assessment results in weekly and monthly DOE reports and maintains corrective actions in the EMD Performance Indicators.

ISM and HPI
DOE M 450.4-1, provides the guidelines to systematically integrate safety into management and work practices at all levels so that missions are accomplished while protecting the public, the worker, and the environment.  This is accomplished through effective integration of safety management into all facets of work planning and execution.  Whenever possible, Pantex includes an element of the Integrated Safety Management model into the assessment.  ISM is a formal, organized process whereby people plan, perform, assess, and improve the safe conduct of work.  By including ISM into assessments, it provides guiding principles and core functions for ensuring the integration of safety, health, and environmental considerations into all types of work and for all types of potential hazards.
ISO 14001, Section 4.4.6, Emergency Planning and Preparedness, does not require assessments, but does require period testing of emergency procedures.  ISM also pulls in the Human Performance Improvement (HPI) philosophy, which is fundamentally striving for excellence in human performance by continuously reducing event caused by human error.