# Presentation: 110456

## Using Evaluation Criteria to Organize the Planning, Conduct, and Oversight of Readiness Assurance Activities

**Jim Powers**
*Office of Emergency Management*
*National Nuclear Security Administration*
*U.S. Department of Energy*
**Jim Jamison       Greg Martin**
*Science Applications International Corporation (SAIC)*

## Objective

- 	**To promote the use of a ****standard set**** of evaluation criteria to organize, coordinate, and integrate the *****planning*****, *****conduct*****, and *****oversight***** ****of Readiness Assurance activities**

## PrologueSOME THOUGHTS . . . 

- Looking at and using the evaluation criteria as proposed in this presentation is a* ****simple***** **and** *****convenient*** way to organize, coordinate, and integrate readiness assurance activities:
- It will add structure and clarity to seemingly disjointed and unrelated readiness assurance processes
- It will make a difference in how you assess your program
- It will foster program improvements
- It will facilitate DOE oversight by adding transparency
_***It is as simple as it seems***_***!***
***This***** *****is ***_***not***_*** “ROCKET SCIENCE*****!”**

## PART 1

**Background**
**and**
**Requirements**

## Readiness Assurance Activities 

- Provide** ****assurances** that DOE Emergency Management Programs will achieve the ***Performance Goal . . .***
- To be** ****“ready”**** **at all times
- To promptly, efficiently, and effectively **apply**** **the necessary** ****resources**
- To **mitigate**** **consequences and** ****protect**** **its workers, the public, the environment, and national security in the event of an Operational Emergency

## Readiness Assurance Activities

- Each DOE/NNSA emergency management program **must** implement a ***readiness assurance program*** consisting of the following components:
- Evaluations
- Improvements
- Documentation (ERAP)
**	****	****	****	****	****	****DOE O 151.1C**

## Readiness Assurance ActivitiesFramework

**Evaluations consist of . . .**
- Evaluation of plans, procedures, and preparedness activities
- Evaluation of response performance
- Oversight of readiness assurance activities

## Readiness Assurance ActivitiesFramework & Tools

- Evaluation of plans, procedures, and preparedness activities
**Program Evaluations**
- Internal Evaluations (Self-assessments)
- External Evaluations (Field Element, HS-63, DNFSB, line management)
**Performance Indicators**
- Performance Measures/Metrics

## Readiness Assurance ActivitiesFramework & Tools

- Evaluation of response performance
**Facility & site exercise evaluations**
**Limited Scope Performance Tests (LSPT)**
**Evaluated tabletops & drills**
**No-Notice Exercises (NNXs)**
**Performance Indicators**
- Performance Measures/Metrics

## Readiness Assurance ActivitiesFramework & Tools 

- Field Element Oversight of Readiness Assurance Activities Includes...
**Review of Self-Assessments **(required by DOE O 151.1C)
- AND
**Risk Based Oversight (RBO) Model**

## Readiness Assurance Requirements Evaluations & Oversight

***EVALUATIONS***
***Program Secretarial Officer (PSO)*** program evaluation (every 3 years)
***Cognizant Field Element*** assessment (every 3 years)
***Contractor ***
- Self-assessment (annual)
- Site & facility exercises (annual)
- Performance indicators (ongoing)
- No-Notice Exercises (as scheduled)
- External evaluations (as scheduled)
***OVERSIGHT (of Readiness Assurance activities)***
***Cognizant Field Element*** review of contractor self-assessment program (annual)

## Readiness Assurance Requirements Evaluation & Oversight

**WHY SO MANY EVALUATIONS???**
**Evaluation of plans, procedures, preparedness activities**
- Looks at _***planned***_ response to _***any***_ scenario, but not actual response
**Evaluation of response performance**
- Assesses actual response, but _***only***_ for a single  scenario

## Readiness Assurance Requirements Evaluations & Oversight

**Evaluations of**** **
**Plans, procedures, **
**and preparedness activities**
***AND***
**Response performance**
**are** _**COMPLEMENTARY**_

## Readiness Assurance Requirements Evaluations & Oversight

- 	Program and exercise evaluations **must** be based on ***specific standards and criteria***, issued by the Director, Office of Emergency Operations.
- 							**DOE O 151.1C**

## PART 2

**Role of**** ****Standard Evaluation Criteria**** ****in Planning, Conduct, and Oversight of Readiness Assurance Activities**

## Strengths & Limitationsof Evaluation Methods

**Document review**:  Examines _*expected*_* or *_*planned*_ performance only
**Inspection of response tools**: Confirms that tools are in place, but not how they are used
**Examination of simulated or actual response**:  Only one  particular event; after-the-fact look (actual response)
**Observations of training and drills**:  Lack of realism; less stressful environment
**Interviews**:  Less stressful environment than either simulated or actual response

## Each Evaluation Method Has Unique Focus

**LSPT** - Performance of single/few functions
**Tabletops** - Integrated decision-making
**Facility and Site Exercises** - Integrated response
**Site Exercise (Full-Participation)** - Offsite interfaces
**No-Notice Exercises** - Integrated initiation/activation
**Performance Indicators** - Tracking of preparedness activities
**Program Evaluation** - Comprehensive response planning
**Self-Assessment** - Detailed, comprehensive review
***   ******No one evaluation tool provides the complete picture!***

## Role of Standard Evaluation Criteriain Evaluations and Oversight 

**Provide common basis for...**
- Internal & external evaluations
- Validate/enhance reliability of self-assessments
- Exercise Objectives
- Promote complex-wide consistency in exercise planning
- “Exercise Builder” uses the EMG criteria
- Evaluating plans vs. performance
- Program vs. Exercise Evaluations

## Role of Standard Evaluation Criteriain Evaluations and Oversight

**Provide common basis for...**
- Findings, corrective actions, and lessons-learned
- Clear, unambiguous structure for trending, tracking, and otherwise analyzing evaluation results
- Selecting complex-wide performance metrics/measures

## Payoff for the Facility/Contractor...

***Planning & scheduling*** self-assessment efforts
- Estimating needed ***resources and expertise***
- Coordinating use of ***special skills***
***Prioritizing ***efforts based on past findings
***Correlating ***findings from program evaluations with exercise performance
***Documenting ***and linking records & reports
***Explaining*** self-assessment efforts & findings to management and Cognizant Field Element

## Payoff for the Cognizant Field Element ...

***Prioritizing ***oversight activities
- Facilitates the ***coordination ***of oversight activities with internal/external evaluations
- Enhanced ability to ***monitor*** performance trends
- Simplified ***roll-up*** of self-assessment results (in ERAPs)
***Facilitates dialogue*** on external assessment findings (e.g., HS-63, DNSFB, PSO)
- Clear and coherent ***framework*** for supporting ***assertions of program**** ****“readiness”***

## What we are trying to AVOID?

**What we are trying to AVOID?**

**Site/facility Emergency**
**Management**
**Program**

- HS/HA

- ERO

- Admin

- F & E

- Etc,

- Etc,

**External**
**Oversight**

**Inconsistent evaluation efforts and findings at different levels**

**Field Element**
**Oversight**

**Self **
**Assessment**

## What we are trying to ACHIEVE?

**What we are trying to ACHIEVE?**

**Site/facility Emergency**
**Management**
**Program**

- HS/HA

- ERO

- Admin

- F & E

- Etc,

- Etc,

**External**
**Oversight**

**Integration of assessment efforts through systematic use of EMG evaluation criteria**

**Field Element**
**Oversight**

**Self **
**Assessment**

## PART 3

**Recommendations for Use of Standard Evaluation Criteria**

## Different Eyes, Same Criteria

**Self- **
**Assessment**

**Cognizant Field Element**

**HS-63**

**Program Office**

## Using Criteria to Plan Evaluations

**Recognize that individual criteria have different “*****attributes*****” **
- Direct/indirect impact on performance goal
- Observable task or activity & documentation
- Performance level or quality
- Role/responsibility
- Facility or equipment
**Customize criteria for *****program-specific *****use**

## Standard Evaluation CriteriaDOE G 151.1-3, Appendix D

**	**
**	**As published in EMG, criteria are labeled to identify applicability to different evaluations:
**P**:** **_*program*_ evaluation only;
**E**: _*exercise*_ evaluation only;
**P/E**: _*both*_ program and exercise evaluations; and
**CE**: evaluation of the _*conduct*_ of an exercise.

## Recommended Designations as “DIRECT” or “INDIRECT” 

**DIRECT Criterion**:
- 	Failure to meet the criterion, will, ***by itself***, adversely affect the associated output (performance) of the program element
**INDIRECT Criterion**:
- 	Failure to meet the criterion 	may ***contribute*** to a direct 	impact or otherwise ***indirectly ******	******impact*** emergency 	management performance

## Criteria “Attributes”and Evaluation Methods

**Demonstrated performance in**
**Drill**
**Exercise (site only)**
**Exercise (full scale)**
**Walkthrough/demo**
**Planned performance (procedures)**
**Output of task (products)**
**Document reviews (specify)**
**EPIP**
**Checklist**
**Reference docs**
**Training materials**

**TASK:**  The criterion addresses a planning, preparedness or response task that can be observed or analyzed through..
**(Doc):**  A plan, procedure, checklist or other document likely exists that says how the task is to be performed.

## Criteria “Attributes”and Evaluation Methods

**Observe performance**
**Review documents (specify)**
**Observe training**
**Personnel qualifications & selection criteria**
**Etc.**

**PERF (“performance”):**
- The criterion addresses a quantity, quality or other feature of planning, preparedness or response, such as timeliness, accuracy, or effectiveness.

## Criteria “Attributes” and Evaluation Methods

**E-Plan review **
**Other documents (specify)**
**Training materials**
**Personnel selection criteria**
**Observe performance**
**Interviews, discussion**

**R&R (“roles & responsibilities”):**  The criterion addresses the assignment and/or execution of specific roles & responsibilities for planning, preparedness or response.

## Criteria “Attributes”and Evaluation Methods

**Physical inspection**
**Equipment performance specs**
**Calibration & QA records**
**Observe use & performance**
**Records of tests, reliability**

**FEQ (“facilities & equipment”):**  The criterion addresses the availability, adequacy and/or use of facilities, equipment or software for planning, preparedness or response.

## Example 1 (from handout)

| DIRECT Impact Criterion | TASK (Doc) | PERF | R&R | FEQ |
| --- | --- | --- | --- | --- |
| P/E9.11 -Classification of an OE involving the actual or potential airborne release of (or loss of control over) hazardous material is accomplished promptly and accurately using a current set of a facility/site-specific EALs.  | X () |  X |  |  |

- TASK:  A planning, preparedness or response task that can be observed or for which performance can be inferred from procedures or the output (product) of the effort.
-    (Doc) = A controlling document (procedure, checklist) will or may exist.
- PERF: Criterion specifies a quality, quantity or other measure of adequacy.
- R&R:  Criterion addresses staff roles and responsibilities.
- FEQ:  Criterion pertains to facilities, equipment, software, etc.

## Criteria Evaluation Methods Program-Specific

***Performance-Based Evaluations***
_**Emergency Management System**_** *****Performance Goal***
**	****	****	**_***Program Element ***_**Performance Goal****	****	****	****	****     **
**	****	****	****	****    **_***Generic ***_**Criteria (from EMG)****	****	****	****	****	****     ****	**
**	****	****	****	****	****	**_***Program-Specific***_** Criteria**

## Creating Program-Specific Criteria

- Eliminate criteria not applicable to the program
- Identify specific criteria not relevant to a selected exercise scenario
- Insert program-specific terminology, values, and information, where appropriate:
- Identify specific plans, procedures, documents
- Specify time parameters
- Insert organization & position titles
- Specify forms, tools,
- Clarify roles, responsibilities, etc.

## PART 4

**Examples/Exercises**

## PART 5

**Summary**
**And**
**Conclusions**

## SUMMARY:  Evaluation Criteria provide a common basis for...

- Internal & external evaluations
- Validates & enhances reliability of self-assessments
- Exercise Objectives
- Promotes complex-wide consistency in exercise planning
- Evaluating plans vs. performance
- Correlates program findings with vs. exercise observations
- Findings, Corrective Actions, and Lessons-Learned
- Clear, unambiguous structure for trending, tracking, and otherwise analyzing evaluation results
- Selecting complex-wide performance metrics/measures

## Benefits

- For the ***Program***
- Plan & schedule self-assessments
- Estimate resources, expertise
- Coordinate use of special skills
- Prioritize efforts based on past findings
- Correlate findings with exercise performance
- Document and link records, reports
- Explain self-assessment results to mgmt & Field Element

## Benefits

- For the ***Cognizant Field Element***
- Prioritize oversight activity
- Enhance monitoring of performance trends
- Simplified roll-up of self-assessment results (in ERAPs)
- Common basis for dialogue on external assessment findings (HS-63, DNSFB, PSO)

## A Structured Approach Can ProvidePractical & Integrated Answers . . .

**P/E11.16 (Consequence Assessment)**
- When was the last or when is the next . . . response performance or procedure evaluation?
- What “readiness” tests have been conducted over the past two years?
- What findings were assessed?  Corrective actions implemented and validated?
- Is a test required during next exercise?
- Were findings repeated during the program evaluation or exercise?
- ------------------------------------------------
- (P/E11.16 - Field sampling and monitoring activities are used to verify, update, and refine the source term and projected consequences through coordination with those responsible for consequence estimates. )

## A Structured Approach Can ProvidePractical & Integrated Answers . . . (con’d)

**P/E11.16 (Consequence Assessment)**
- Need tests/evaluations to close corrective action?
- NNX is planned and HS-63 inspection next quarter will address this ***TASK***
- DOE oversight will include this ***TASK*** (a Direct impact criterion) in its current annual oversight review of the most important self-assessment results
- Multiple evaluations (tests) reinforce self-assessment conclusions
***We are doing what is necessary to assure “readiness” and improve performance for this TASK.***

## How do the diverse Readiness Assurance pieces fit?

**Field Element **
**Oversight**

**PSO Oversight**

**Performance**
**Indicators**

**No-Notice Exercises**

**External**
**Evaluations**

**Self-Assessment**

**How do the diverse Readiness ****Assurance pieces fit?**

## Reliable “Readiness” Assurance of Emergency Management Program . . .  

**Field Element Oversight**

**PSO Oversight**

**Performance Indicators**

**No-Notice Exercises**

**External**
**Evaluations**

**Self-Assessment**

**Reliable “Readiness” Assurance of Emergency Management Program . . .****  **

**= “READINESS” TO RESPOND**

**Evaluation Criteria**

## ELECT TO PER-R-R-FECT

**A**** ****STANDARD ****set of performance-based****  *****PROGRAM-SPECIFIC***** ****evaluation criteria forms the basis for a cohesive and convincing judgment of ****“*****readiness*****”**** based on the processes and results of a**** *****COORDINATED***** ****Readiness Assurance . . .**
***PROGRAM***