Department of Homeland Security 

Office of Inspector General 

DHS Efforts To Address Lessons Learned 


in the Aftermath of Top Officials Exercises 


OIG-09-53 April 2009 
Office of Inspector General 

U.S. Department of Homeland Security 

Washington, DC 20528 

 April 10, 2009 

Preface 

The Department of Homeland Security (DHS) Office of Inspector General (OIG) was established 

by the Homeland Security Act of 2002 (Public Law 107-296) by amendment to the Inspector 

General Act of 1978. This is one of a series of audit, inspection, and special reports prepared as 

part of our oversight responsibilities to promote economy, efficiency, and effectiveness within 

the department. 

This report addresses the strengths and weaknesses of DHS’ process to determine, formulate, and 

address lessons learned and corrective needs identified during the Top Officials full-scale 

exercise series. It is based on interviews with employees and officials of relevant agencies and 

institutions, direct observations, quantitative analysis, and a review of applicable documents. 

The recommendations herein have been developed to the best knowledge available to our office, 

and have been discussed in draft with those responsible for implementation. We trust this report 

will result in more effective, efficient, and economical operations. We express our appreciation 

to all who contributed to the preparation of this report. 

Richard L. Skinner 


Inspector General 

Table of Contents/Abbreviations
 

Executive Summary.........................................................................................................................1
 

Background......................................................................................................................................2
 

Results of Review ............................................................................................................................6
 

TOPOFF Exercises Did Not Include a Requirement 


for Corrective Action Planning Prior to 2007............................................................................6 


Corrective Action Program Has Not Been Fully Implemented ................................................7 


Recommendations......................................................................................................................8
 

Management Comments and OIG Analysis ..............................................................................9 


Recurring Themes Identified in Previous Exercises and 


Real-World Events Have Not Been Resolved ...........................................................................9 


Recommendations....................................................................................................................10
 

Management Comments and OIG Analysis ............................................................................11 


Participation and Support in TOPOFF Planning and
 

Implementation Need To Be Increased....................................................................................11 


Recommendations....................................................................................................................13
 

Management Comments and OIG Analysis ............................................................................14 


Evaluator Selection and Training Need Improvements...........................................................14 


Recommendations....................................................................................................................15
 

Management Comments and OIG Analysis ............................................................................16 


Dissemination of TOPOFF AARs, Lessons Learned, and Best Practices .............................16 


Recommendations....................................................................................................................18
 

Management Comments and OIG Analysis ............................................................................18 


Appendixes 

Appendix A: Purpose, Scope, and Methodology....................................................................19 


Appendix B: Management Response to the Draft Report ......................................................20 


Appendix C: The Corrective Action Program Process...........................................................22 


Appendix D: Major Contributors to This Report....................................................................23 


Appendix E: Report Distribution............................................................................................24 

Table of Contents/Abbreviations
 

Abbreviations 

AAR after-action report 

CAP Corrective Action Program 

DHS Department of Homeland Security 

ESC Executive Steering Committee 

FEMA Federal Emergency Management Agency 

FOUO For Official Use Only 

HSAS Homeland Security Advisory System 

HSEEP Homeland Security Exercise and Evaluation Program 

HSC Homeland Security Council 

HSPD Homeland Security Presidential Directive 

LLIS Lessons Learned Information Sharing 

NED National Exercise Division 

NEP National Exercise Program 

NIMS National Incident Management System 

NRP National Response Plan 

Sub-IPC Sub-Interagency Policy Committee 

TOPOFF Top Officials 
Page 1 

OIG 


Department of Homeland Security 

Office of Inspector General 

Executive Summary 

The Top Officials exercises test how key government officials respond to 

simulated terrorist attacks. The exercises are intended to strengthen the 

Nation’s capacity to prepare for, prevent, respond to, and recover from 

large-scale terrorist acts. Our review examined the Department of 

Homeland Security’s process to determine, formulate, and address lessons 

learned and corrective needs identified during Top Officials exercises. 

Since the first Top Officials exercises in 2000, neither a process for 

tracking weaknesses and how those weaknesses were resolved, nor a 

method for identifying and analyzing trends in corrective actions or 

significant lessons learned has been established. As a result, federal, state, 

local, and territorial agencies were unclear regarding the implementation 

of suggested improvements following preparedness exercises. 

In 2007, the Federal Emergency Management Agency instituted a federal 

interagency corrective action program to address these problems. This 

program established an interagency process for corrective action 

implementation and validation after Top Officials exercises. However, the 

corrective action process has not been fully implemented. The Federal 

Emergency Management Agency also established a National Exercise 

Program and standardized the terrorism exercise process. However, the 

Department of Homeland Security has not secured adequate participation 

and support from other federal departments and agencies; state, territorial, 

and local entities; or the private sector in planning, implementing, and 

evaluating exercises or in the corrective action processes. In addition, 

after-action reports, best practices, and lessons learned from Top Officials 

exercises have not been disseminated to a broad national audience. 

We are recommending that the Department of Homeland Security seek 

assistance from high-level agency and interagency committees, and amend 

National Exercise Program guidance, as needed, to (1) fully implement the 

corrective action program; (2) increase the level of participation by top 

officials in all phases of the exercise; and (3) disseminate after-action 

reports, best practices, and lessons learned to a broad national audience. 

Federal Emergency Management Agency officials generally agreed with 

our findings and will provide a detailed corrective action plan for 

implementation in their 90-day letter. 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 
Background 

History of the TOPOFF Exercise Series 

In 1998, Congress provided funding for a national terrorism preparedness exercise 

to assess the nation’s crisis and consequence management capability by applying 

plans, policies, procedures, systems, and facilities through federal, state, and local 

responses to terrorist threats and acts. Recognizing that top departmental and 

agency officials rarely participated fully in such exercises, Congress required “the 

participation of all key personnel who would participate in the consequence 

management of such an actual terrorist event.”1

 In May 2000, the Department of 

Justice led the first in a series of national preparedness exercises called Top 

Officials (TOPOFF) 2000. 

TOPOFF 2000 simulated a pneumonic plague outbreak in Denver, Colorado, and 

a mustard gas attack in Portsmouth, New Hampshire. More than 6,500 federal, 

state, and local personnel, including top officials, responded to these simulated 

attacks. Important objectives of the exercise were to examine the interfaces 

between participating agencies and officials to identify seams, gaps, and 

redundancies that affect decision-making, and to identify subsequent actions to 

resolve consequence management issues. 

To build on TOPOFF 2000, Congress mandated the formation of the TOPOFF 

exercise series and the development of a continuous exercise cycle using 

seminars, tabletop exercises, and command post exercises that culminate in a 

biennial, national, full-scale TOPOFF exercise. Each exercise series would 

involve a 2-year cycle of seminars, planning events, and exercises, and would 

culminate in a full-scale assessment of the Nation’s capacity to prevent, prepare 

for, respond to, and recover from large-scale terrorist attacks. 

In May 2003, the Department of Justice and the Federal Emergency Management 

Agency (FEMA) jointly sponsored TOPOFF 2, the second in the series of 

exercises, using the scenarios of a radiological dispersal device in Seattle, 

Washington, and pneumonic plague in Chicago, Illinois. TOPOFF 2 was 

designed to identify vulnerabilities in the domestic incident management 

capability by exercising against a series of integrated terrorist threats and acts. 

The exercise engaged 8,500 responders and top officials from the United States 

and Canada—the first international partner to participate. 

Because TOPOFF 2 was in its final planning stages when DHS was established, 

TOPOFF 3, the third exercise of the series, was the first exercise in which DHS 

maintained full responsibility for planning, conducting, and after-action reporting. 

Conducted in April 2005, TOPOFF 3 was designed to identify vulnerabilities in 

the Nation’s domestic incident management capability. This exercise simulated 

1

 Senate Report 105-235, July 2, 1998. 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 

Page 2 
 

pneumonic plague attacks in Union and Middlesex counties, New Jersey, and a 

mustard gas release in New London, Connecticut. TOPOFF 3 included more than 

10,000 participants, with international exercise play in the United Kingdom and 

Canada as part of a partnership intended to strengthen security, communication, 

and information sharing among the three nations. 

During TOPOFF 3, federal departments and agencies had their first opportunity to 

use the National Response Plan (NRP). The NRP was designed to link nationallevel, hazard-specific contingency plans that could be implemented independently 

during localized incidents or concurrently during incidents of national significance.2 

The foundation for the NRP was the National Incident Management System 

(NIMS), which provides a consistent national doctrinal framework for incident 

management at all jurisdictional levels, regardless of the cause, size, or complexity 

of the incident. 

During 2005, DHS developed a Universal Task List and Target Capabilities List 

to continue to define homeland security tasks, provide a framework to assess 

preparedness, and identify required resources. The Universal Task List describes 

incident management tasks to be performed, and provides a standardized 

reference for all levels of government and the private sector. The Target 

Capabilities List contains capabilities that various levels of government need to 

develop and maintain to prevent, respond to, and recover from a terrorist attack or 

major disaster. The Universal Task List and Target Capabilities List were still in 

the developmental stages during the TOPOFF 3 planning process and were 

subsequently integrated into the TOPOFF 4 exercise design, planning, evaluation, 

and improvement planning considerations. 

Homeland Security Exercise and Evaluation Program Policy and 

Guidance 

The purpose of the Homeland Security Exercise and Evaluation Program 

(HSEEP) is to provide common exercise policy and program guidance that 

constitutes a national standard for exercises.3

 Prior to 2007, HSEEP did not 

fully address exercise planning, evaluation, after-action implementation 

planning, or the corrective action process. In response to NRP and NIMS 

goals to standardize preparedness exercise design, development, and 

evaluation, DHS updated and expanded HSEEP guidelines for intra-agency 

exercises in Volume III, Exercise Evaluation and Improvement Planning, 

dated February 2007. Specifically, the updated 2007 HSEEP guidelines 

required that the TOPOFF 4 exercise include: 

2

 The NRP was in effect for the TOPOFF 3 and TOPOFF 4 exercises but was replaced by the National Response 

Framework in January 2008. 

3

 HSEEP policy and guidance is contained in five volumes: Volume I: Overview and Exercise Program 

Management; Volume II Exercise Planning and Conduct; Volume III: Exercise Evaluation and Improvement 

Planning; Volume IV: Sample Exercise Documents and Formats and Volume V: Prevention Exercises (Draft). 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 

Page 3 
 

�	 

�	 

�	 

�	 

�	 

�	 

�	 

Capabilities-based objectives in exercise design; 

Data collected and analyzed according to HSEEP guidelines; 

Exercise controller/evaluator training and briefings at exercise 

venues; 

An after-action conference to discuss policy observations 


identified during the exercise; 


A final after-action report (AAR) that includes an Improvement 

Plan; 

Adherence to HSEEP security guidance regarding AAR and 


Improvement Plan distribution; and 


Tracking and implementing of corrective actions through the 


Corrective Action Program (CAP). 


Following Homeland Security Council (HSC) guidance and HSEEP 

guidelines to plan exercises according to capabilities-based objectives, the 

October 2007 TOPOFF 4 exercise employed a radiological dispersal 

device scenario that involved coordinated terrorist attacks in Portland, 

Oregon; Phoenix, Arizona; and the U.S territory of Guam. 

Formation of the National Exercise Program 

In the December 2003, Homeland Security Presidential Directive (HSPD)-8: 

National Preparedness, the President directed the Secretary of DHS “in 

coordination with other appropriate Federal departments and agencies, [to] 

establish a national program and a multi-year planning system to conduct 

homeland security preparedness-related exercises that reinforce training 

standards, provide for evaluation readiness, and support the national 

preparedness goal.” Under HSPD-8, federal agencies were directed to 

participate in a collaborative, interagency process to design preparedness 

exercises on a consensus basis, and create a master exercise calendar. 

The HSC, created in October 2001 by HSPD-1: Organization and 

Operation of the HSC, is located in the Executive Office of the President. 

The HSC reports on matters related to homeland security and combating 

terrorism, and consists of cabinet secretaries and White House senior 

officials whose responsibility includes a principal interest in homeland 

security policymaking. The HSC maintains a policy coordination and 

advisory role that functions across all levels of the federal government and 

answers directly to the President. 

In 2006, the HSC established the National Exercise Program (NEP), and 

approved its charter, implementation plan, and 5-year national exercise 

schedule in January 2007. The NEP requires that HSEEP methodology 

and guidelines be incorporated into TOPOFF exercise design, evaluation, 

and documentation activities. The NEP charter addresses White House 

recommendations contained in the Katrina Report, a comprehensive 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 

Page 4 
assessment of the response and recovery efforts as a result of Hurricane 

Katrina, and provides a framework for federal interagency exercise 

coordination. 

FEMA’s National Exercise Division (NED) is the executive agent for the 

NEP. The NEP categorizes exercise activities into four tiers. These tiers 

reflect the relative priority for interagency participation, with Tier I as the 

most important and Tier IV being the least: 

�	 

�	 

�	 

�	 

Tier I: White House–directed exercises with U.S. Governmentwide strategy, policy focus, and participation; 

Tier II: U.S. Government-wide priorities, federal strategies and 

policy focus, and significant exercise simulation; 

Tier III: other federal exercises that have an operational, tactical, or 

organizational focus, and include simulation; and, 

Tier IV: state, territorial, local, tribal, or private sector exercises. 

The NEP chairs the Executive Steering Committee (ESC), which defines 

the exercise schedule, supports coordination of exercises, and ensures that 

exercises are conducted. The ESC coordinates departmental and agency 

roles and responsibilities, and reports to the HSC’s Exercise and 

Evaluation Sub-Interagency Policy Committee (Sub-IPC). The Sub-IPC 

recommends exercise priorities, goals, objectives, schedules, and 

corrective action issues to the HSC Deputies Committee, which is 

composed of deputy cabinet secretaries of the relevant federal agencies. 

The ESC comprises eight departments and agencies: 

�	 

�	 

�	 

�	 

�	 

�	 

�	 

Department of Defense 

o	 Office of the Secretary of Defense/Homeland Defense 

o	 Joint Staff – J7 

Department of Homeland Security 

Department of Energy 

Department of Health and Human Services 

Department of Justice 

o	 Federal Bureau of Investigation 

Department of Transportation 

Office of the Director of National Intelligence 

�	 Federal Bureau of Investigation 

The ESC may also contain up to two rotating members who serve 1-year 

terms. In addition to their program responsibilities, the NED and NEP are 

responsible for the following: 

�	 HSEEP, which develops policy and guidance on all aspects of 

preparedness exercises; 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 

Page 5 
�	 

�	 

�	 

�	 

�	 

�	 

The National Exercise Schedule, which encompasses federal, state, 

territorial, local, tribal, or private sector Tier I through Tier IV 

exercises; 

The CAP system, which provides an interdepartmental system to 

develop, prioritize, and track corrective actions following exercises 

and real-world events; 

The Lessons Learned Information Sharing (LLIS) System, a webbased national network of lessons learned and best practices for the 

emergency response and homeland security community; 

The Regional Exercise Support Program, which supports 

regionally coordinated exercises through assistance in the design, 

development, conduct, and evaluation of preparedness exercises; 

The Terrorism Prevention Exercise Program, which conducts 

exercises and supports activities that increase awareness, 

cooperation, and information sharing among homeland security 

and law enforcement officials at all levels of government; and 

The Senior Official Exercise, which supports the design, 

development, conduct, and evaluation of principal level and intraDHS exercises. 

Results of Review 

TOPOFF Exercises Did Not Include a Requirement for Corrective 

Action Planning Prior to 2007 

Past TOPOFF exercises and actual disasters such as Hurricane Katrina have 

uncovered gaps in the Nation’s preparedness. Federal, state, territorial, and local 

response organizations have recognized the importance of promoting best 

practices and identifying areas for improvement to produce a more effective 

response to acts and threats, as well as natural disasters. However, a process to 

prioritize, track, and validate corrective actions taken to mitigate weaknesses 

identified in exercise planning, development, and execution was not established 

until 2007. 

Prior to 2007, HSEEP’s exercise planning, evaluation, after-action 

implementation planning, and a corrective action process were incomplete. 

During TOPOFF 2 in May 2003, attendees and exercise participants identified 

several lessons learned relating to exercise design and conduct. However, 

exercise evaluation and improvement planning was not performed, as these 

actions were not included as part of the initial HSEEP guidance. 

TOPOFF 3, in 2005, was the first exercise that DHS sponsored with full 

responsibilities for exercise planning, conduct, and after-action activities. The 

exercise results identified operational shortfalls in terrorism prevention, response, 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 

Page 6 
 

Page 7 

and recovery. The TOPOFF 3 after-action process followed 2003 HSEEP 

principles, which focused on determining why a particular event or response 

occurred, and provided participants with information that could be used to 

improve response and recovery efforts. However, improvement planning to 

address remedial needs and corrective action procedures was not a part of 

HSEEP. Therefore, the final AAR recommendations only informed participating 

departments and agencies of existing problems, and encouraged improvements in 

agency prevention, response, and recovery capabilities. 

In the absence of a process to identify and implement corrective actions, problem 

areas identified during and even prior to the TOPOFF 3 exercise recurred during the 

response to Hurricane Katrina in 2005, and again in the 2007 TOPOFF 4 exercise. 

A Corrective Action Program Has Not Been Fully Implemented 

Previous preparedness exercises and real-world incidents did not result in the 

development of corrective action plans or the assignment of responsibility for 

corrective actions to resolve cross-cutting issues, because no department or 

agency had authority to direct the actions of another. Participating agencies 

developed internal corrective action plans, but without a coordinated interagency 

process, issues involving multiple federal entities remained unresolved. 

In February 2007, HSEEP updated the evaluation and improvement planning 

section to include (1) more details on improvement planning, and (2) the 

requirement for a corrective action planning system. In addition, the NEP 

requires the development of a corrective action process to ensure that lessons 

from exercises are either sustained or improved as appropriate. 

The CAP provides the basis for systematically developing, prioritizing, and 

tracking corrective actions following exercises, real-world events, and policy 

discussions. Established as part of the NEP Implementation Plan, the CAP 

represents an improvement toward interagency coordination and accountability 

for corrective action after TOPOFF and other exercises. It requires that all 

departments and agencies maintain a corrective action program that can generate 

data for, and track assignments from, the NEP CAP for unclassified issues. 

Appendix C provides a diagram of the CAP process. 

The full implementation of a CAP system involves the coordination of activities 

between the HSC, ESC, NEP, and participating departments and agencies. The 

HSC maintains a policy coordination and advisory role that functions across all 

levels of the federal government, and answers directly to the President. The HSC 

also maintains responsibility for corrective action issues and communicating the 

overall direction of CAP to department and agency top officials. The ESC 

coordinates departmental and agency roles and responsibilities, and reports to the 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 
DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 


Page 8
 

HSC’s Exercise and Evaluation Sub-IPC. The NEP is responsible for 

incorporating HSEEP guidelines into TOPOFF exercise design and evaluation. 

The TOPOFF 4 AAR, which has not been approved or released by DHS, concludes 

that NEP has established a process for tracking and monitoring the implementation 

of corrective actions. In June 2008, the ESC met to prioritize corrective actions 

identified in the March 2008 AAR Implementation Plan. An NEP official informed 

that in September 2008, the Sub-IPC reviewed the corrective actions identified in 

the AAR Implementation Plan and transmitted them to the HSC Domestic 

Readiness Group for approval. Subsequently, the HSC-approved corrective actions 

were assigned to individual departments and agencies for remediation. The official 

added that NED entered the corrective actions into the CAP system and initiated 

training to ensure that action officers can enter information on the status of assigned 

tasks. However, as of March 2009, the CAP system does not include corrective 

action updates and has not produced required progress reports to validate 

completion of any corrective actions. 

In an August 2007 memorandum to cabinet and agency heads, the Secretary of 

DHS emphasized the NEP Implementation Plan requirement for adherence to a 

disciplined CAP that outlines the process by which shortfalls are identified, tasked 

for action, and tracked to resolution. NEP managers said that generating full 

support and participation for the CAP system from department and agency 

officials is a significant challenge. The support and active participation of the 

Sub-IPC and HSC Deputies Committee are also critical to ensuring that corrective 

actions are assigned, tracked, and validated. 

Recommendations 

We recommend that the Administrator, Federal Emergency Management 

Agency: 

Recommendation #1: Provide the interagency Policy Coordinating 

Committees and the HSC with detailed information regarding challenges 

faced and actions that need to occur for complete implementation of the 

CAP system. 

Recommendation #2: Request that the Sub-IPC and the HSC use the 

information provided to clarify and communicate to federal agencies and 

departments their responsibilities for corrective action assignments, 

tracking, and validation. 

Recommendation #3: Revise the current NEP Implementation Plan by 

including specific program control objectives and procedures that will 

provide management reasonable assurance on CAP implementation. 
DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 


Page 9
 

Management Comments and OIG Analysis 

FEMA Response to Recommendation #1: FEMA agreed with our 

recommendation and will provide a detailed corrective action plan for 

implementation in their 90-day letter following the issuance of our final 

report. 

OIG Analysis: This recommendation is resolved and open pending our 

receipt of FEMA’s 90-day letter. 

FEMA Response to Recommendation #2: FEMA agreed with our 

recommendation and will provide a detailed corrective action plan for 

implementation in their 90-day letter following the issuance of our final 

report. 

OIG Analysis: This recommendation is resolved and open pending our 

receipt of FEMA’s 90-day letter. 

FEMA Response to Recommendation #3: FEMA agreed with our 

recommendation and will provide a detailed corrective action plan for 

implementation in their 90-day letter following the issuance of our final 

report. 

OIG Analysis: This recommendation is resolved and open pending our 

receipt of FEMA’s 90-day letter. 

Recurring Themes Identified in Previous Exercises and Real-World 

Events Have Not Been Resolved 

According to the April 2007, National Exercise Program Implementation Plan, 

“The perennial lament of exercise planners and participants is 

exercise fatigue, when multiple exercises require recurring 

application of a department or agency’s limited time and 

resources. This fatigue factor is exacerbated by frustrations 

that occur when exercise activities do not lead to appreciable 

improvements in policies, plans, or performance.” 

The TOPOFF 4 AAR concluded that the exercise identified (1) where the 

Nation’s preparedness has improved, and (2) areas that need further improvement. 

The areas that need further improvement include recurring themes that have been 

identified in previous exercises and real-world events, and several new areas. 
DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 


Page 10
 

The Homeland Security Advisory System (HSAS) provides an example of a 

policy issue repeatedly identified during TOPOFF exercises that has not been 

resolved. The HSAS was designed to guide protective measures when specific 

threat information from a particular sector or geographic region is received. It 

combines threat information with vulnerability assessments, and provides 

pertinent information to public safety officials and the public. Following 

TOPOFF 2, the AAR reported uncertainty regarding protective actions to be taken 

for each HSAS threat level. Specifically, the purpose, definitions, and 

consequences of the HSAS threat levels were not clear. In addition, coordination 

of actions taken by other agencies or jurisdictions under HSAS was also unclear. 

Participants at all levels of government, as well as international participants, have 

raised questions about the meaning and implications of HSAS level changes. 

In the aftermath of TOPOFF 3, the AAR concluded that the HSAS system was 

still not understood or systematically used by exercise participants. However, no 

corrective actions were included in the AAR, since this was not required by the 

exercise evaluation and improvement process. After TOPOFF 4, the AAR 

concluded again that the purpose, definitions, and consequences of HSAS threat 

levels remain unclear. 

Regarding the reoccurrence of HSAS issues in successive TOPOFF exercises, 

NEP program managers said that actions to address HSAS issues required 

correcting unresolved policy issues beyond the scope and authority of their office. 

Following TOPOFF 4, NEP officials reported that the HSC had prioritized HSAS 

issues for immediate attention. However, a decision has not been made whether 

actions to address HSAS would be included and tracked under the CAP, or 

managed and monitored directly by the HSC. The HSC, through its Domestic 

Readiness Group and Sub-IPC, can prioritize and assign action under CAP. 

Recommendations 

We recommend that the Administrator, Federal Emergency Management 

Agency: 

Recommendation #4: Request that the HSC clarify and document their 

determination whether issues prioritized by the Sub-IPC for immediate 

action after TOPOFF exercises will be included and tracked under the CAP. 
Management Comments and OIG Analysis 

FEMA Response to Recommendation #4: FEMA agreed with our 

recommendation and will provide a detailed corrective action plan for 

implementation in their 90-day letter following the issuance of our final 

report. 

OIG Analysis: This recommendation is resolved and open pending our 

receipt of FEMA’s 90-day letter. 

Participation and Support in TOPOFF Planning and Implementation 

Needs To Be Increased 

The participation of top officials responsible for response and consequence 

management is critical to the success of TOPOFF exercises. Based on the April 

2007 NEP Implementation Plan, all departments and agencies must budget for 

support to NEP Tier I exercises and be prepared to provide Principal- or Deputylevel support. However, FEMA officials said that not all federal departments and 

agencies have (1) established their participation on committees and working 

groups, (2) accounted for the costs associated with participation, and 

(3) committed sufficient resources for TOPOFF exercises. 

For private sector participation, the TOPOFF 4 AAR concludes that exercises 

offer the private sector an opportunity to collaborate with the government and test 

plans for disaster response and recovery. However, planners said that to create 

strong linkages, the private sector needed to be engaged earlier in the planning 

cycle with their federal agency counterparts. 

Federal Participation 

Recognizing that top officials rarely participated in national-level 

preparedness exercises, Congress required that top officials and key staff 

participate fully in exercise planning, implementation, and after-action 

exercise activities. At the federal level, NEP engaged more than 40 

departments, agencies, and offices in its TOPOFF 4 interagency planning 

process. However, all agencies did not commit the required personnel or 

resources to accomplish TOPOFF exercise activities, which affected the 

extent of their participation in the planning, implementing, and evaluating 

processes. 

In August 2007, 10 weeks prior to the start of TOPOFF 4, Secretary 

Chertoff distributed a memorandum for cabinet and agency heads 

requesting those who had not assigned an executive to the ESC to do so. 

However, by the conclusion of the TOPOFF 4 exercise in October 2007, 

not all federal departments and agencies had designated representatives to 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 

Page 11 
Page 12 

the ESC or made budgetary decisions to ensure the availability of 

resources to plan and participate in the exercise. NEP officials said that at 

the conclusion of TOPOFF 4, departmental and agency participation was 

not at the level expected for robust interagency planning and participation. 

As a result, many of the TOPOFF 4 planning and scenario design 

activities did not have a broad representation of exercise planners. 

Although TOPOFF objectives are expected to reflect U.S. Governmentwide priorities, some exercise objects were not emphasized because of the 

need for more inclusive participation in exercise plan, design, or afteraction analysis. In the exercise after-action discussions, ESC members 

said that the need for information and participation by departments and 

agencies has created a perception that the TOPOFF exercise program is 

overly focused on DHS objectives and is of minimal value. 

NEP managers described securing sufficient federal support for TOPOFF 

as a primary challenge to establishing a national exercise program. The 

TOPOFF 4 AAR suggests that HSEEP policy and guidance be amended to 

emphasize the requirements for TOPOFF participation at the highest 

levels. Achieving early and active engagement of all concerned parties 

will help ensure that TOPOFF exercises reflect federal government-wide, 

state, territorial, local, and private sector priorities. FEMA will need the 

support and assistance of federal interagency committees and the HSC to 

achieve this goal. 

Participation at the State, Territorial, Local, and Private Sector 

Levels 

State, territorial, and local officials responsible for TOPOFF 4 planning 

and implementation commented that they: 

�	 

�	 

�	 

Were not engaged early enough in the exercise planning process; 

Were not given clear benchmarks and guidelines for participation; 

and 

Would benefit from more support and mentoring from state-level 

counterparts from previous TOPOFF exercises, which was a part 

of the TOPOFF 3 preparation process. 

NED supported and managed the needs of the private sector working 

group for the TOPOFF 3 exercise. This responsibility was not clearly 

delineated for TOPOFF 4. According to a FEMA official, the NEP did 

not plan or budget for the necessary recruitment and engagement efforts of 

the private sector in TOPOFF 4. 

TOPOFF 4 private sector participants included a broad range of 

representatives from the critical infrastructure/key resources sector, large 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 
Page 13 

corporations, nongovernmental and service organizations, and others. 

Their participation in TOPOFF helped to identify areas where they could 

contribute during or after an actual terrorism event or a major natural 

disaster, and how national-level policies could be integrated into their 

operations. 

Following TOPOFF 4, private sector participants provided the following 

assessments: 

�	 

�	 

Exercise planners needed to distinguish the levels of participation 

of each type of private sector organization and clarify the 

objectives and requirements of participation for each 

organizational type; and 

Private sector participants needed to be recruited earlier in the 

exercise design and plan cycle. 

During TOPOFF 4, there was confusion within the private sector about 

specific roles and defined expectations during the exercise. In the absence 

of a clear delineation of duties and specific roles, private sector entities 

developed expectations that did not coincide with those of the federal 

government. 

Recommendations 

We recommend that the Administrator, Federal Emergency Management 

Agency: 

Recommendation #5: Task FEMA’s Private Sector Office to co-chair a 

working group with DHS private sector entities to identify the requirements 

for private sector TOPOFF participation, and establish a plan to support 

their participation. 

Recommendation #6: Develop and implement a plan to secure the full 

participation and required resources of federal departments and agencies, 

and state, local, and territorial entities to design, plan, and conduct the 

TOPOFF exercise program. 

Recommendation #7: Amend National Exercise Program guidance to 

ensure that it addresses and supports the appropriate level of federal 

departments and agencies, and state, local, and territorial participation 

expected in the TOPOFF exercise program. 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 
Management Comments and OIG Analysis 

FEMA Response to Recommendation #5: FEMA agreed with our 

recommendation, and will provide a detailed corrective action plan for 

implementation in their 90-day letter following the issuance of our final 

report. 

OIG Analysis: This recommendation is resolved and open pending our 

receipt of FEMA’s 90-day letter. 

FEMA Response to Recommendation #6: FEMA agreed with our 

recommendation, and will provide a detailed corrective action plan for 

implementation in their 90-day letter following the issuance of our final 

report. 

OIG Analysis: This recommendation is resolved and open pending our 

receipt of FEMA’s 90-day letter. 

FEMA Response to Recommendation #7: FEMA agreed with our 

recommendation, and will provide a detailed corrective action plan for 

implementation in their 90-day letter following the issuance of our final 

report. 

OIG Analysis: This recommendation is resolved and open pending our 

receipt of FEMA’s 90-day letter. 

Evaluator Selection and Training Needs Improvements 

The TOPOFF 4 Evaluator Handbook outlines evaluator responsibilities that 

include recording observations of player actions and managing data collection at 

all exercise sites. Data collected by evaluators during the TOPOFF 4 exercise 

supported the post-exercise reconstruction and analysis process, the development 

of observations, lessons learned, and the recommendations contained in the 

TOPOFF 4 AAR. More than 650 evaluators participated in the TOPOFF 4 

exercise. 

The TOPOFF 4 Evaluator Handbook requires evaluators to be familiar with the 

roles and responsibilities of the players they observe. NED relied on departments 

and agencies to identify and assign individuals with appropriate agency-specific 

knowledge to perform data collection at key locations during the exercise. 

However, some departments, agencies, and DHS components did not assign 

evaluators to support the evaluation methodology until just before the full-scale 

exercise. Therefore, all evaluators did not complete the NED evaluator training, 

or familiarize themselves with the exercise plan, scenario, missions, policies, and 

procedures specific to their assigned location. 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 

Page 14 
TOPOFF 4 evaluators were offered a training session conducted by NED in 

advance of the exercise. Agency personnel selected as evaluators were expected to: 

�	 

�	 

�	 

�	 

�	 

Review the exercise plan and scenario, the master scenario events list, 

evaluation plan, and control staff instructions; 

Complete evaluator training requirements; 

Familiarize themselves with the missions, plans, policies, procedures, and 

processes applicable to their assigned location; 

Familiarize themselves with data collection forms and exercise evaluation 

guides for their assigned location; and 

Attend the controller/evaluator briefings at their assigned location. 

Some of the evaluators said that they were selected by their departments and 

agencies to serve as evaluators shortly before the start of the TOPOFF 4 exercise. 

Therefore, they were unable to complete the full training program conducted by 

NED. They participated in an abbreviated training session offered the day before 

the exercise started, and did not have the opportunity to review the master 

scenario events list, evaluation plan, or control staff instructions before assuming 

their evaluator duties. This resulted in reduced assurance that evaluators were 

able to obtain the necessary knowledge, skills, and abilities to record required data 

during the exercise. 

Evaluators without sufficient knowledge of the exercise plan, scenario, agency 

plans, policies, and procedures were not as effective in performing their assigned 

duties during the exercise. For example, evaluators collecting data at the Portland 

Joint Information Center and the Joint Federal Office said that they did not have a 

good grasp of the evaluation objectives for their assigned locations, but were 

making their best effort to collect data and record what they observed. 

The TOPOFF 4 AAR Annex 1 includes comments and recommendations for 

improvements from the exercise working groups. The working groups 

acknowledged in their comments that the complexity and specific requirements of 

TOPOFF created a need for more extensive evaluator training tailored to the 

requirements of each exercise venue. They also proposed that evaluators be 

identified earlier in the exercise planning process, and that the training schedule 

be integrated into venue and interagency group meetings. 

Recommendations 

We recommend that the Administrator, Federal Emergency Management 

Agency: 

Recommendation #8: In response to TOPOFF 4 AAR Annex 1 

recommendations, develop and implement a plan to identify, train, and 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 

Page 15 
support department and agency officials and staff who will control, 

evaluate, or participate in TOPOFF exercise programs. 

Management Comments and OIG Analysis 

FEMA Response to Recommendation #8: FEMA agreed with our 

recommendation, and will provide a detailed corrective action plan for 

implementation in their 90-day letter following the issuance of our final 

report. 

OIG Analysis: This recommendation is resolved and open pending our 

receipt of FEMA’s 90-day letter. 

Dissemination of TOPOFF AARs, Lessons Learned, and Best Practices 

According to HSEEP guidelines, the overarching TOPOFF exercise objectives are 

(1) the identification and dissemination of lessons learned, and (2) the promotion 

of best practices to the national community of first responders and emergency 

managers. In addition, HSPD-8: Terrorism Preparedness mandates that DHS 

“develop and maintain a system to collect, analyze, and disseminate lessons 

learned, best practices, and information from exercises, training events, research, 

and other sources.” DHS has not yet developed an effective process to 

disseminate TOPOFF AARs, lessons learned, and best practices to a broad 

national audience. 

Distribution of TOPOFF AARs 

For the TOPOFF 4 exercises ending in November 2007, NEP finalized 

and submitted the AAR to the HSC Sub-IPC for review in March 2008. In 

June 2008, the report was transmitted to FEMA officials for a security 

review and classification. In November 2008, FEMA transmitted the 

AAR to DHS headquarters for final approval and distribution. However, 

as of March 2009, DHS had not approved the TOPOFF 4 AAR for release 

to participating agencies and other interested parties. 

An NEP official said that the current HSEEP timeline requires an AAR to 

be produced 60 days after the exercise. Based on feedback received from 

federal, state, and local officials, the 60-day requirement will be changed 

to 90 days. Also, taking into account the additional interagency 

coordination, the NEP official recommended the following timeframe: 

(1) draft interagency report is created within 120 days after the exercise; 

(2) FEMA approves the report within 150 days after the exercise; and 

(3) DHS approves the report within 180 days after the exercise. 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 

Page 16 
Page 17 

 

To date, TOPOFF AARs have had restricted distribution to a limited 

number of federal, state, territorial, and local departments and agencies on 

a loan basis because DHS considers that the reports may contain sensitive 

information. All TOPOFF AARs have been designated For Official Use 

Only (FOUO), which precludes their dissemination to a broad national 

audience. 

Distribution of TOPOFF Lessons Learned 

The Lessons Learned Information Sharing (LLIS) website is DHS’ online 

repository for information dedicated to the emergency response and 

homeland security communities. NEP managers describe the LLIS 

website as the primary method to disseminate TOPOFF exercise lessons 

learned and best practices. 

The TOPOFF 4 exercise plan did not encompass specific activities or a 

budget to disseminate lessons learned immediately following the exercise. 

However, to bring the results of the exercise to a wider national audience, 

senior FEMA officials directed the NED to conduct a 1-day National 

After-Action Conference for state homeland security advisors and 

emergency management directors. NED invited officials from 56 states 

and territories to attend the March 2007 event in Oklahoma City, 

Oklahoma. The conference goals were to report on the TOPOFF 4 

exercise, present lessons learned, and describe the new CAP process. The 

TOPOFF 4 National After-Action Conference attracted 142 participants 

from federal, state, and territorial agencies; 47 attended in person and 95 

participated via a webcast of the conference. Conference participants 

represented 26 states and territories. 

The conference included a 30-minute presentation and a FOUO document 

that described lessons learned from the TOPOFF 4 exercise. NED 

officials said that while participation was less than envisioned, the 

conference achieved its goals, and was an effective way to disseminate 

TOPOFF lessons learned to a national audience of state homeland security 

officials. 

According to the GAO Standards for Internal Control in the Federal 

Government, information should be communicated to management and 

others who need it within a timeframe that enables them to carry out their 

responsibilities. NED managers said that the process to disseminate 

TOPOFF AARs and lessons learned is evolving, and progress will be 

made as HSEEP-directed exercise evaluation and improvement planning 

becomes more standardized under the NEP and CAP. 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 
DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 


Page 18
 

Recommendations 

We recommend that the Administrator, Federal Emergency Management 

Agency: 

Recommendation #9: With assistance from DHS senior officials and 

offices, develop and implement a process and timeframe to distribute final 

TOPOFF AARs. 

Recommendation #10: To communicate information to a national 

audience of first responders and emergency management officials, develop 

and implement a process and timeframe to populate LLIS.gov with 

TOPOFF exercise lessons learned and best practices. 

Management Comments and OIG Analysis 

FEMA Response to Recommendation #9: FEMA agreed with our 

recommendation, and will provide a detailed corrective action plan for 

implementation in their 90-day letter following the issuance of our final 

report. 

OIG Analysis: This recommendation is resolved and open pending our 

receipt of FEMA’s 90-day letter. 

FEMA Response to Recommendation #10: FEMA agreed with our 

recommendation, and will provide a detailed corrective action plan for 

implementation in their 90-day letter following the issuance of our final 

report. FEMA noted in their response memorandum that at the conclusion 

of TOPOFF 4, DHS/FEMA produced and disseminated three Quick Look 

reports for participating departments and agencies and the response 

community, as well as a public version. 

OIG Analysis: This recommendation is resolved and open pending our 

receipt of FEMA’s 90-day letter. 
DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 


Page 19
 

Appendix A 

Purpose, Scope, and Methodology 

Our objective was to determine whether the DHS is achieving its goals to 

identify, distribute, and address lessons learned and corrective needs that 

surfaced during the biennial national TOPOFF terrorism preparedness 

exercises. We examined these processes in relation to the TOPOFF 3 and 

TOPOFF 4 exercises. 

We conducted interviews with DHS/FEMA National Preparedness 

Directorate staff who manage the TOPOFF exercise series. We interviewed 

other DHS officials, managers, and staff; personnel from other federal 

departments and agencies; and state, local, and private sector exercise 

participants who played an instrumental role in TOPOFF exercise design, 

planning, and evaluation. We collected, reviewed, and analyzed documents 

related to exercise planning, training, evaluation, after-action process, and 

the corrective action program. 

We observed and collected data during the TOPOFF 4 exercise in 

Phoenix, Arizona, and Portland, Oregon; and at the exercise master 

control cell established in Northern Virginia. We observed the TOPOFF 4 

exercise after-action briefings (hot washes) conducted in Phoenix and 

Portland, and in Virginia; the December 2007 Long-term Recovery Table 

Top Exercise; the January 2008 After-Action Conference; and the April 

2008 National After-Action Conference in Oklahoma City, Oklahoma. 

We conducted our review between October 2007 and August 2008 under 

the authority of the Inspector General Act of 1978, as amended, and 

according to the Quality Standards for Inspections issued by the 

President’s Council on Integrity and Efficiency. 
DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 


Page 20
 

Appendix B 

Management Response to the Draft Report 
DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 


Page 21
 

Appendix B 

Management Response to the Draft Report 
DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 


Page 22
 

Evaluators observe and 

analyze exercise, and 

develop preliminary 

recommendations to 

address observed issues

Event Evaluation

DRAFT AAR with 

preliminary 

recommendations

Exercise Participants

review DRAFT AAR and 

develop proposed CAs to 

address preliminary 

recommendations

After-Action 

Conference

Final AAR/IP with specific 

CAs, action officers, start 

dates and completion dates

NEP ESC prioritizes CAs

and sends to DRG for 

finalization; DRG sends 

CAs to affected PCC for 

formal assignment

Issue Prioritization

List of pr oritized CAs

Affected PCC formally 

assigns priority CAs to 

appropriate D/A via 

ExecSec channels

Issue Assignment

Formal notification of CA 

assignments to 

appropriate D/As

Assigned D/A develops 

Implementation Plan for 

assigned CA

Solution 

Development

Implementation Plan 

with defined timelines 

and milestones

Affected PCC reviews 

implementation plan and 

provides feedback

Solution Review

Final Implementation 

Plan reflecting PCC 

feedback

Assigned D/A

implements CAs in 

accordance with proposed 

timelines and milestones

Solution 

Implementation

Progress toward CA 

implementation tracked in 

CAP System

Assigned D/A validates 

implemented CA 

through metrics defined 

in implementat on plan

Solution 

Validation

E.g., Exercise 

objectives testing 

and validating CAs

NEP ESC complies and 

distributes repots to 

D/As, PCCs, and HSC 

Deputies Committee

Solution 

Reporting

CA implementation 

summarized and 

reported

- 

 

 

 

Appendix C 

The Corrective Action Program Process 

Step 1

Agent/Action

Output

Step 2

i

Step 3 Step 4 Step 5

Step 6 Step 7 Step 8

Key Acronyms

Step 9

E+15 E+30 E+60 E+90 E+130

E+160 E+170 Ongoing

Monthly (D/As)

Quarterly (PCCs)

Biannually (Deputies)

DRAFT AAR with 

preliminary 

recommendations 

Step 1 

Agent/Action 

Output 

Final AAR/IP with specific 

CAs, action officers, start 

dates and completion dates 

Step 2 

List of prioritized CAs 

Step 3 

Formal notification of CA 

assignments to 

appropriate D/As 

Step 4 

Implementation Plan 

with defined timelines 

and milestones 

Step 5 

Final Implementation 

Plan reflecting PCC 

feedback 

Step 6 

Assigned D/A 

implements CAs in 

accordance with proposed 

timelines and milestones 

Solution 

Implementation 

Progress toward CA 

implementation tracked in 

CAP System 

Step 7 

E.g., Exercise 

objectives testing 

and validating CAs 

Step 8 

Key Acronyms 

CA implementation 

summarized and 

reported 

Step 9 

E+15 E+30 E+60 E+90 E+130 

E+160 E+170 Ongoing 

Monthly (D/As)

Quarterly (PCCs)

Biannually (Deputies) 

Event Evaluation 

Evaluators observe and 

analyze exercise, and 

develop preliminary 

recommendations to 

address observed issues 

After Action 

Conference 

Exercise Participants 

review DRAFT AAR and 

develop proposed CAs to 

address preliminary 

recommendations 

Issue Prioritization 

NEP ESC prioritizes CAs 

and sends to DRG for 

finalization; DRG sends 

CAs to affected PCC for 

formal assignment 

Issue Assignment 

Affected PCC formally 

assigns priority CAs to 

appropriate D/A via 

ExecSec channels 

Solution 

Development 

Assigned D/A develops 

Implementation Plan for 

assigned CA 

Solution Review 

Affected PCC reviews 

implementation plan and 

provides feedback 

Solution 

Validation 

Assigned D/A validates 

implemented CA 

through metrics defined 

in implementation plan 

Solution 

Reporting 

NEP ESC complies and 

distributes repots to 

D/As, PCCs, and HSC 

Deputies Committee 

After-Action Report (AAR)

Improvement Plan (IP)

Corrective Action (CA)

Department/Agency (D/A)

Homeland Security Council (HSC)

Executive Secretary (Exec Sec)

DRG (Domestic Readiness Group)

After-Action Report (AAR) 

Improvement Plan (IP) 

Corrective Action (CA) 

Department/Agency (D/A) 

Homeland Security Council (HSC) 

Executive Secretary (Exec Sec) 

DRG (Domestic Readiness Group) 
Page 23 

Appendix D 

Major Contributors to This Report 

Deborah Outten-Mills, Chief Inspector, Department of Homeland 

Security, Office of Inspections 

Jim O’Keefe, Senior Inspector, Department of Homeland Security, Office 

of Inspections 

Lawrence Anderson, Inspector, Department of Homeland Security, Office 

of Inspections 

Jessica Barnes, Inspector, Department of Homeland Security, Office of 

Inspections 

DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 
DHS Efforts To Address Lessons Learned in the Aftermath of Top Officials Exercises 


Page 24
 

Appendix E 

Report Distribution 

Department of Homeland Security 

Secretary 

Acting Deputy Secretary 

Chief of Staff for Operations 

Chief of Staff for Policy 

Acting General Counsel 

Executive Secretariat 

Director, GAO/OIG Liaison Office 

Assistant Secretary for Policy 

Assistant Secretary for Public Affairs 

Assistant Secretary for Legislative Affairs 

Administrator, Federal Emergency Management Administration 

Audit Liaison, Federal Emergency Management Administration 

Office of Management and Budget 

Chief, Homeland Security Branch 

DHS OIG Budget Examiner 

Congress 

Congressional Oversight and Appropriations Committees, as appropriate 
ADDITIONAL INFORMATION AND COPIES 

To obtain additional copies of this report, please call the Office of Inspector General (OIG) at (202) 254-4199, 

fax your request to (202) 254-4305, or visit the OIG web site at www.dhs.gov/oig. 

OIG HOTLINE 

To report alleged fraud, waste, abuse or mismanagement, or any other kind of criminal or noncriminal 

misconduct relative to department programs or operations: 

• Call our Hotline at 1-800-323-8603; 

• Fax the complaint directly to us at (202) 254-4292; 

• Email us at DHSOIGHOTLINE@dhs.gov; or 

• Write to us at: 

DHS Office of Inspector General/MAIL STOP 2600, 

Attention: Office of Investigations - Hotline, 

245 Murray Drive, SW, Building 410, 

Washington, DC 20528. 

The OIG seeks to protect the identity of each writer and caller.