GAO 

United States Government Accountability Office

Report to Congressional Requesters

HOMELAND 

DEFENSE 

U.S. Northern 

Command Has a 

Strong Exercise 

Program, but 

Involvement of 

Interagency Partners 

and States Can Be 

Improved 

September 2009 

GAO-09-849 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

September 2009

HOMELAND DEFENSE

U.S. Northern Command Has a Strong Exercise 

Program, but Involvement of Interagency Partners

Highlights of GAO-09-849, a report to and States Can Be Improved 

Congressional Requesters 

U.S. Northern Command 

(NORTHCOM) exercises to test 

preparedness to perform its 

homeland defense and civil support 

missions. GAO was asked to assess 

the extent to which NORTHCOM is 

(1) consistent with Department of 

Defense (DOD) training and 

exercise requirements, (2) 

involving interagency partners and 

states in its exercises, (3) using 

lessons learned and corrective 

actions to improve preparedness, 

and (4) integrating its exercises 

with the National Exercise 

Program (NEP). To do this, GAO 

reviewed NORTHCOM and NEP 

guidance and postexercise 

documentation, assessed 

NORTHCOM compliance, and 

compared DOD and NEP exercise 

requirements. 

What GAO Recommends

GAO is making recommendations 

to DOD to direct NORTHCOM to 

consistently involve the states in 

planning, executing, and assessing 

exercises and improve oversight of 

corrective actions. GAO is also 

recommending that DOD define 

when NORTHCOM should use NEP 

planning and documentation 

requirements. DOD agreed or 

partially agreed with the 

recommendations and cited 

ongoing and future efforts to 

satisfy the recommendations’ 

intent. DOD did not fully address a 

recommendation on training to 

NORTHCOM staff on specific state 

emergency management structures. 

GAO believes such training would 

benefit NORTHCOM personnel in 

advance of a crisis and for exercise 

planning. 

NORTHCOM’s exercise program is generally consistent with the requirements 

of DOD’s Joint Training System, but its exercise reporting is inconsistent. 

Since the command was established in 2002, NORTHCOM has conducted 13 

large-scale exercises and generally completed exercise summary reports 

within the required time frame. However, those reports did not consistently 

include certain information, such as areas needing improvement, because 

NORTHCOM lacks guidance that specifies exercise reports’ content and 

format, potentially impacting its ability to meet internal standards for planning 

and execution of joint exercises, and to compare and share exercise results 

over time with interagency partners and states. 

Nineteen federal agencies and organizations and 17 states and the District of 

Columbia have participated in one or more of the seven large-scale exercises 

that NORTHCOM has conducted since September 2005. However, 

NORTHCOM faces challenges in involving states in the planning, conduct, and 

assessment of its exercises, such as adapting its exercise system and practices 

to involve other federal, state, local, and tribal agencies that do not have the 

same practices or level of planning resources. Inconsistencies with how 

NORTHCOM involves states in exercises are occurring in part because 

NORTHCOM officials lack experience dealing with states and do not have a 

consistent process for including states in exercises. Without such a process, 

NORTHCOM increases the risk that its exercises will not provide benefits for 

all participants, impact the seamless exercise of all levels of government, and 

potentially affect NORTHCOM’s ability to provide civil support capabilities. 

NORTHCOM has a systematic lessons learned and corrective action program 

to improve preparedness, but gaps remain with collecting and sharing lessons 

with agency and state partners and managing corrective actions. Access to the 

system NORTHCOM uses for managing exercise observations is limited for 

non-DOD participants, and DOD believes that the Department of Homeland 

Security’s system is not adequately protected from unauthorized users. 

NORTHCOM’s mitigation steps have not resolved the issues. In addition, 

about 20 percent of the corrective actions tracked by NORTHCOM were being 

closed prematurely due to gaps in oversight. Closing issues prematurely 

increases the risk that issues will reoccur and limits the knowledge gained and 

value of the exercise. 

NORTHCOM has taken steps to integrate its exercises with the NEP, but 

guidance is not consistently applied. NORTHCOM has participated in several 

NEP exercises and is leading its first major NEP exercise in the fall of 2009. 

However, NORTHCOM has used DOD’s Joint Training System planning and 

documentation requirements rather than DHS’s requirements, because NEP 

guidance is not clear on what exercise planning standard should be used and 

DOD guidance does not address the issue. The states we visited use NEP 

guidance. Differences between NEP and DOD guidance could affect the ability 

of all participants to develop effective working relationships. 

View GAO-09-849 or key components.

For more information, contact Davi M. 

D'Agostino, (202) 512-5431, 

dagostinod@gao.gov. 
Page i GAO-09-849 

Contents 

Letter 1

Results in Brief 6 

Background 11 

NORTHCOM’s Exercise Program Is Consistent with DOD’s Joint 

Training System, but Exercise Reporting is Inconsistent 26 

Interagency Partners and States Have Participated In 

NORTHCOM’s Exercises, but Challenges Remain with Including 

States in Planning, Conducting, and Assessing Exercises 32 

NORTHCOM has a Systematic Lessons Learned and Corrective 

Action Program, but Gaps Remain 42 

NORTHCOM Has Taken Steps to Integrate Its Exercises with the 

National Exercise Program, but Guidance Is Not Consistently 

Applied 51 

Conclusions 58 

Recommendations for Executive Action 60 

Agency Comments and Our Evaluation 61 

Appendix I Scope and Methodology 67 

Appendix II Comparison of DOD’s Joint Training System with 

DHS’s Homeland Security Exercise and Evaluation 

Program 71 

Appendix III Comments from the Department of Defense 75 

Appendix IV GAO Contact and Staff Acknowledgments 80 

Related GAO Products 81 

Homeland Defense 
Tables 

Table 1: Major Documents Related to the National Exercise 

Program and NORTHCOM 22 

Table 2: Large-Scale NORTHCOM Exercises Since 2002 27 

Table 3: Interagency Partner and State Participation in Large-scale 

NORTHCOM Exercises Since 2006 33 

Table 4: Emergency Management Structures of the States We 

Visited 38 

Table 5: Number of Observations Input into Lessons Learned 

Management System for Large-Scale Exercises Since 2006 43 

Table 6: NORTHCOM Participation in National Level and Tier II 

Exercises Since 2007 52 

Table 7: Comparison of Areas of Application between DOD’s Joint 

Training System and DHS’s Homeland Security Exercise 

Evaluation Program 55 

Table 8: Comparison of DOD Joint Training System and Homeland 

Security Exercise and Evaluation Program 71 

Figures 

Figure 1: NORTHCOM’s Area of Responsibility 14 

Figure 2: Pathways for Requesting Military Assistance for Incident 

Response 16 

Figure 3: DOD’s Joint Training System 19 

Figure 4: National Exercise Program and U.S. Northern Command 

Exercise Program Time line: Major Events and Milestones 

(2001-2009) 21 

Figure 5: Description of NEP Tiers 23 

Figure 6: Homeland Security Exercise and Evaluation Program 

Tool Kit 25 

Figure 7: NORTHCOM Lessons Learned and Corrective Action 

Process 45 

Page ii GAO-09-849 Homeland Defense
Abbreviations 

DOD Department of Defense 

DHS Department of Homeland Security 

FEMA Federal Emergency Management Agency 

JLLIS Joint Lessons Learned Information System 

NEP National Exercise Program 

NORTHCOM U.S. Northern Command 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately. 

Page iii GAO-09-849 Homeland Defense
Page 1 GAO-09-849 

 

United States Government Accountability Office

Washington, DC 20548 

 

September 9, 2009 

Congressional Requesters 

The U.S. homeland continues to face complex and dynamic threats and 

vulnerabilities from terrorism and other catastrophic incidents, including 

natural disasters, industrial accidents, infrastructure failures, and 

infectious diseases.1

 These incidents can produce devastating 

consequences as observed during the September 11, 2001, terrorist attacks 

and in the aftermath of Hurricane Katrina, which required an effective and 

coordinated national effort, with shared goals and responsibilities for 

protecting and defending the homeland. The complexity of national-level 

coordination makes preparing for an incident all the more crucial. 2

According the Quadrennial Defense Review, unified interagency efforts 

among the Department of Defense (DOD), Department of Homeland 

Security (DHS), and other federal, state, and local agencies are required to 

address threats to the homeland.3

 Exercises play an instrumental role in 

preparing these agencies to respond to an incident by providing 

opportunities to test plans, improve proficiency, assess capabilities and 

readiness, and clarify roles and responsibilities.4

 Short of performance in 

actual operations, exercises provide the best means to assess the 

effectiveness of organizations in achieving mission preparedness. 

Exercises also provide an ideal opportunity to enhance preparedness by 

collecting, developing, implementing, and disseminating lessons learned 

and verifying corrective action taken to resolve previously identified 

issues.5

1

Homeland Security Council, National Strategy for Homeland Security (Washington, D.C: 

October 2007), 1, 9, 13. 

2

GAO, National Preparedness: FEMA Has Made Progress, but Needs to Complete and 

Integrate Planning, Exercise, and Assessment Efforts, GAO-09-369 (Washington, D.C.: 

Apr. 30, 2009). 

3

DOD, Quadrennial Defense Review Report (Washington, D.C.: Feb. 6, 2006), 87. 

4

An exercise is a training event where scenarios are enacted to assess training proficiency 

and identify lessons learned to improve preparedness. Exercises range from tabletop 

computer simulations to the full-scale deployment of personnel and equipment. 

5

U.S. Northern Command, Commander’s Training Guidance for FY09-10 (Colorado 

Springs, Colo.: Apr. 8, 2008), A-20. 

Homeland Defense 
 

Following the September 11 terrorist attacks, DOD realized the need for a 

more integrated civilian and military response capability for any future 

attack on the homeland. In response, DOD established the U.S. Northern 

Command (NORTHCOM) to provide and manage homeland defense and 

civil support. These two activities are among DOD’s contribution to 

homeland security. DHS is the lead federal agency for homeland security, 

which is a national effort to prevent terrorist attacks within the United 

States, reduce America’s vulnerability to terrorism, and minimize the 

damage and recover from attacks that do occur.6

 DOD is the lead federal 

agency for homeland defense—defined as the protection of U.S. 

sovereignty, territory, domestic population, and critical defense 

infrastructure against external threats and aggression against the United 

States.7

 Civil support is defense support of civil authorities—such as DHS 

or another federal agency or state—for domestic emergencies and for 

designated law enforcement and other activities.8

 NORTHCOM differs 

from other combatant commands in that, in addition to Canada and 

Mexico, its area of responsibility includes the 48 contiguous states, Alaska, 

the District of Columbia, Puerto Rico, and the U.S. Virgin Islands. 

Numerous local, state, tribal, and federal agencies and organizations—

including DOD, DHS, and the Federal Emergency Management Agency 

(FEMA)—have jurisdiction over or can coordinate resources within the 

homeland and, therefore, may be involved in the response to an incident. 

According to the National Defense Strategy, a whole-of-government 

approach is only possible when every government department and agency 

understands the core competencies, roles, missions, and capabilities of its 

partners and works together to achieve common goals.9

 Ensuring an 

effective response to an incident will require that federal departments and 

agencies, states, and local governments conduct integrated disaster 

response planning and test these plans by exercising together. 10 To 

achieve its goal of being ready to execute joint operations and ensure a 

seamless operating environment, NORTHCOM’s training goals include 

maximizing exercise participation with federal, state, and local agencies 

 

6

DOD, Strategy for Homeland Defense and Civil Support, 5. 

7

DOD, Strategy for Homeland Defense and Civil Support, 5. 

8

DOD, Strategy for Homeland Defense and Civil Support, 5-6. 

9

DOD, National Defense Strategy (Washington, D.C.: June 2008), 17-18. 

10Defense Science Board, Unconventional Operational Concepts and the Homeland 

(Washington, D.C.: March 2009), 37, 48, 50. 

Page 2 GAO-09-849 Homeland Defense
 

and National Guard units.11 NORTHCOM uses long-established DOD 

practices for training and exercises to test and enhance the preparedness 

of its personnel and forces to perform its homeland defense and civil 

support missions. As a result, DOD has more experience than other federal 

agencies in terms of the scope and range of its exercises and the 

robustness of the process for conducting them. DOD’s goal is to improve 

the homeland defense and consequence management capabilities of its 

interagency partners by leveraging this comparative advantage through 

shared training and exercises.12

The National Exercise Program (NEP) was established in April 2007 under 

the leadership of the Secretary of Homeland Security to prioritize and 

coordinate the various federal, regional, and state exercise activities and 

serves as the principal mechanism for examining the preparation and 

efficiency of the federal government to respond to an incident.13 The NEP 

includes a series of national exercises projected on a 5-year exercise 

schedule and has established a four-tier system to determine the relative 

priority of interagency participation in each exercise. DOD and other 

federal agencies, states, and local organizations participate in these 

national-level exercises. FEMA administers the NEP and has developed 

Homeland Security Exercise and Evaluation Program guidance to provide 

standardized policy, methodology, and terminology for exercise design, 

development, conduct, evaluation, and improvement planning. 

In our previous work following Hurricanes Katrina and Rita, we identified 

a need to improve, among other things, the nation’s disaster response 

capabilities. Overall, capabilities are built upon the appropriate 

combination of people, skills, processes, and assets. Ensuring that needed 

capabilities are available requires effective planning and coordination in 

conjunction with training and exercises in which the capabilities are 

realistically tested and problems identified and subsequently addressed in 

partnership with other federal, state, and local stakeholders.14 Specific to 

 

11U.S. Northern Command, Commander’s Training Guidance for FY09-10, A-4. 

12DOD, Quadrennial Defense Review Report, 26-27. 

13DHS, National Exercise Program Implementation Plan (Washington D.C.: Apr. 11, 2007). 

DHS issued a revised Implementation Plan in June 2008. 

14GAO, Homeland Security: Observations on DHS and FEMA Efforts to Prepare for and 

Respond to Major and Catastrophic Disasters and Address Related Recommendations 

and Legislation, GAO-07-835T (Washington, D.C.: May 15, 2007). 

Page 3 GAO-09-849 Homeland Defense
 

DOD, we reported that prior to Hurricane Katrina, disaster plans and 

training exercises involving DOD were insufficient, and did not 

incorporate lessons learned from past catastrophes to fully delineate the 

military capabilities that could be needed to respond to a catastrophic 

natural disaster. Moreover, disaster plans had not been tested and refined 

with a robust exercise program. As a result of the inadequate plans—and 

the lack of realistic exercises to test those plans—a lack of understanding 

existed within DOD and among federal, state, and local responders as to 

the types of assistance and capabilities that DOD might provide, the timing 

of this assistance, and the respective contributions of the active-duty and 

National Guard components. We recommended that DOD establish 

milestones and expedite the development of detailed plans and exercises 

to fully account for the unique capabilities and support that the military is 

likely to provide to civil authorities in response to an incident.15 In 

addition, we recently found that although FEMA has developed plans, 

guidance, and systems to design exercises, gather lessons learned, and 

track corrective actions to implement the NEP, it faces challenges in 

ensuring that the exercises are carried out in accordance with Homeland 

Security Exercise and Evaluation Program guidance. We recommended, 

among other things, that FEMA establish a program management plan and 

better ensure that exercises follow Homeland Security Exercise 

Evaluation Program guidance, for example, by revising its grant 

monitoring guidance to include a checklist of specific requirements for 

states that receive grant funds.16

In order to better understand NORTHCOM’s efforts to effectively prepare 

for its homeland defense and civil support missions, we were asked to 

review NORTHCOM’s exercise program. This review is the final part of a 

broader congressional request to review NORTHCOM’s efforts to plan and 

coordinate its homeland defense and civil support missions. We have 

issued two previous reports in support of that request.17 For this report, 

our objectives were to determine the extent to which (1) NORTHCOM’s 

 

15GAO, Hurricane Katrina: Better Plans and Exercises Needed to Guide the Military’s 

Response to Catastrophic Natural Disasters, GAO-06-643 (Washington, D.C.: May16, 2006). 

16GAO-09-369. 

17GAO, Homeland Defense: U.S. Northern Command Has Made Progress but Needs to 

Address Force Allocation, Readiness, Tracking Gaps, and Other Issues, GAO-08-251

(Washington, D.C.: Apr. 16, 2008); and Homeland Defense: Steps Have Been Taken to 

Improve U.S. Northern Command’s Coordination with the States and the National 

Guard Bureau, but Gaps Remain, GAO-08-252 (Washington, D.C.: Apr. 18, 2008). 

Page 4 GAO-09-849 Homeland Defense
 

exercise program is consistent with DOD training and exercise 

requirements; (2) NORTHCOM involves relevant interagency partners and 

states in planning, conducting, and assessing exercises; (3) NORTHCOM is 

using lessons learned and corrective actions during exercises to improve 

mission preparedness; and (4) NORTHCOM is integrating its exercises 

with the NEP. 

In conducting this review, we generally focused our scope on 

NORTHCOM’s large-scale exercises18 conducted since Hurricane Katrina 

made landfall in August 2005. To determine the extent to which 

NORTHCOM’s exercise program is consistent with DOD exercise 

requirements and includes relevant exercise partners, we reviewed 

exercise documentation for all 13 large-scale exercises the command 

performed since it was established in 2002; evaluated NORTHCOM’s 

compliance with requirements established in DOD guidance; and assessed 

the level of interagency and state participation in NORTHCOM’s largescale exercises. We also interviewed Office of the Secretary of Defense, 

Joint Staff, NORTHCOM, FEMA, FEMA regional offices, and state officials 

with knowledge of and experience with NORTHCOM’s exercise program. 

We selected a nongeneralizable sampling of states based on the extent to 

which they have participated in major NORTHCOM exercises since 

Hurricane Katrina and the varying scenarios of the exercises. The states 

we selected played a major role in NORTHCOM exercises by having a 

portion of the exercise conducted in their state and having various state 

agencies and officials participate. States we selected include Arizona, 

California, Michigan, Oregon, Rhode Island, and Washington. We also met 

with Nevada officials who participated in a NORTHCOM exercise prior to 

Hurricane Katrina—Determined Promise 03—to provide context to the 

extent that changes may have been made to NORTHCOM’s exercise 

program and help develop our state selection methodology. To determine 

the extent to which NORTHCOM is using lessons learned during exercises 

to improve mission preparedness, we reviewed DOD, NORTHCOM, and 

DHS’s NEP guidance for lessons learned. Based on this guidance, we 

assessed the management of all unclassified exercise observations and 

issues identified from its last 6 large-scale exercises by determining each 

record’s status (open or closed), type (issue or lesson learned), and 

disposition after NORTHCOM staff have acted on these records. Finally, to 

 

18Large-scale exercises train and evaluate actual forces deployed in a field environment 

under simulated scenarios. NORTHCOM generally conducts two large-scale exercises 

annually. 

Page 5 GAO-09-849 Homeland Defense
 

determine the extent to which NORTHCOM is integrating its exercises 

with the NEP we reviewed DOD and DHS guidance regarding the NEP and 

how DOD and its subordinate commands are required to or should 

participate. We conducted our review between June 2008 and September 

2009 in accordance with generally accepted government auditing 

standards. Those standards require that we plan and perform the audit to 

obtain sufficient, appropriate evidence to provide a reasonable basis for 

our findings and conclusions based on our audit objectives. We believe 

that the evidence obtained provides a reasonable basis for our findings 

and conclusions based on our audit objectives. 

NORTHCOM’s exercise program is generally consistent with the overall 

policies and practices of DOD’s Joint Training System, but the way it 

reports on exercises is inconsistent. DOD and NORTHCOM guidance 

require that the command establish an exercise program consistent with 

the Joint Training System and complete postexercise documentation 

within an established time frame.19 NORTHCOM has developed a 

comprehensive exercise program consistent with DOD’s Joint Training 

System, conducted 13 large-scale exercises since it was created in 2002, 

and generally completed exercise summary reports within the required 

time frame. However, those reports did not consistently include certain 

information; for example, only 5 of the 11 exercise summary reports 

included an identified section on lessons learned. DHS currently has a 

template for exercise documentation that includes guidance on content 

and format; however, NORTHCOM does not follow this template and 

neither DOD nor NORTHCOM has similar guidance that specifies content 

and format. Without a consistent record of what occurred during an 

exercise, NORTHCOM cannot ensure that it has met internal standards for 

planning and execution of joint exercises, compare exercise results over 

time, and share lessons learned with interagency partners and states. We 

are recommending that NORTHCOM’s Commander establish criteria for 

the format and content of postexercise documentation. 

Results in Brief 

Interagency partners and states have participated in NORTHCOM 

exercises, but NORTHCOM faces challenges involving interagency 

partners and states in planning, conducting, and assessing exercises. 

 

19Chairman of the Joint Chiefs of Staff, Joint Training Manual for the Armed Forces of the 

United States, 3500.03B (Washington, D.C.: Aug. 31, 2007), A-1; U.S. Northern Command, 

Operations: Exercise Program, Instruction 10-156 (Colorado Springs, Colo.: June 2, 2008). 

Page 6 GAO-09-849 Homeland Defense
 

NORTHCOM guidance requires that NORTHCOM maximize exercise 

participation with interagency partners and states to achieve preparedness 

goals and ensure a seamless operating environment.20 Seventeen federal 

agencies and organizations and 17 states and the District of Columbia have 

participated in one or more of the seven large-scale exercises that 

NORTHCOM has conducted since September 2005. We found that the 

states we visited derived benefits from their participation in these 

exercises. For example, officials from 3 states told us this was their first 

opportunity to interact with federal military forces. However, we found 

that challenges remain which have resulted in inconsistencies in the way 

that NORTHCOM involves the states in its exercises. One of DOD’s 

challenges is adapting its exercise system and practices to accommodate 

the coordination and involvement of other federal, state, local, and tribal 

agencies that do not have the same kinds of practices or level of planning 

effort. NORTHCOM also faces the challenge of balancing its training 

objectives with those of state agencies and organizations, particularly 

given the limited resources and funding states have available to exercise. 

While state and local governments seek to exercise their first responder 

capabilities before having their resources overwhelmed and needing to 

seek federal assistance, NORTHCOM’s goal is to exercise its capability to 

provide support to civil authorities when local, state, and other federal 

resources are overwhelmed. As a result of this challenge, officials from 5 

states told us that all of their needs were not fully met during the 

exercises, for example, due to large-scale, unrealistic scenarios that 

overwhelmed the states’ resources before they had the opportunity to 

exercise their training objectives. Inconsistencies with how NORTHCOM 

involves states in planning, conducting, and assessing exercises are 

occurring in part because NORTHCOM officials lack experience in dealing 

with the differing emergency management structures, capabilities, and 

needs of the states. Inconsistencies are also occurring because 

NORTHCOM has not established a process for including states in 

exercises, such as consistent procedures for requesting state involvement 

in exercises through DHS/FEMA or the National Guard Bureau. Without an 

informed and consistent process, NORTHCOM increases the risk that its 

exercises will not provide benefits for all participants, impacting the 

seamless exercise of all levels of government and potentially affecting 

NORTHCOM’s ability to provide support to civil authorities. We are 

recommending that (1) DOD work with DHS to establish guidance and 

procedures for requesting state participation and involving states in 

 

20U.S. Northern Command, Commander’s Training Guidance FY 09-10, A-4. 

Page 7 GAO-09-849 Homeland Defense
 

planning, executing, and assessing exercises and (2) NORTHCOM’s 

Commander develop a training plan for NORTHCOM staff on state 

emergency management structures and relevant issues related to working 

with civilian state and local emergency management officials. 

NORTHCOM has a systematic lessons learned and corrective action 

program to improve preparedness, but gaps remain with collecting and 

sharing lessons with agency and state partners and managing corrective 

actions. DOD and NORTHCOM guidance requires that NORTHCOM 

identify lessons learned during the course of normal operations, exercises, 

and real-world events; share valid observations and findings as widely as 

possible; and track them until the corrective actions are verified during 

subsequent events or exercises.21 NORTHCOM uses DOD’s Joint Lessons 

Learned Information System (JLLIS), an automated tool for managing and 

tracking observations, and a corrective action process to manage issues 

requiring action for resolution. Access to this system is limited for nonDOD participants, and DOD believes that DHS’s system is not adequately 

protected from potential unauthorized users. NORTHCOM has taken steps 

to address these issues, such as placing a template on NORTHCOM’s 

restricted access portal to provide a venue for non-DOD officials to record 

exercise observations, but these efforts have not been successful. Because 

NORTHCOM is not fully involving other federal agencies and states in its 

lessons learned process, it is missing opportunities to learn lessons from 

an exercise. For example, officials from two states did not provide 

NORTHCOM with observations from exercises because they did not 

attend the command’s postexercise reviews for varying reasons, such as 

limited resources. As a result, NORTHCOM risks the reoccurrence of these 

issues. We also found that about 20 percent of the corrective actions 

tracked by NORTHCOM were being closed prematurely because, under 

existing procedures, NORTHCOM exercise directorate officials do not 

have oversight over issues that are resolved within other directorates or 

are not giving long-standing issues the sustained management attention to 

ensure resolution. A NORTHCOM official told us that they do not have the 

staff necessary to oversee the actions on records handled within the other 

directorates. Closing issues requiring corrective actions prematurely could 

increase the risk that they could reoccur and limits the knowledge gained 

 

21Chairman of the Joint Chiefs of Staff, Joint Lessons Learned Program, Instruction 

3150.25D (Washington, D.C.: Oct. 10, 2008), A-1 and A-6; U.S. Northern Command, 

Commander’s Training Guidance FY 09-10 , A-4, A-8, A-20, A-2-8; U.S. Northern 

Command, Lessons Learned Program and Corrective Action Board Process, Instruction 

16-166 (Colorado Springs, Colo.: Mar. 16, 2009), 2. 

Page 8 GAO-09-849 Homeland Defense
 

and value of the exercise. We are recommending that (1) NORTHCOM’s 

Commander establish and publicize procedures for non-DOD exercise 

participants to submit observations, (2) the Chairman of the Joint Chiefs 

of Staff direct the Joint Staff to work with DHS to either resolve 

information assurance issues so that combatant commands can post 

Exercise Summary Reports with lessons learned on DHS’s Lessons 

Learned Information Sharing system or establish an alternative method to 

systematically collect and share lessons learned, and (3) the Chairman of 

the Joints Chiefs of Staff direct the Joint Staff to revise the joint lessons 

learned operating instruction to include procedures to ensure that 

corrective actions are implemented and verified in a subsequent exercise 

or operation before being closed. 

NORTHCOM has taken steps to integrate its exercises with the NEP, such 

as participating in several NEP exercises, including combining two of its 

large-scale exercises with NEP exercises. However, NORTHCOM has not 

consistently applied NEP planning and documentation requirements, 

because NEP guidance is unclear on the extent to which federal agencies 

should follow NEP requirements and DOD guidance does not specifically 

address this issue. NEP guidance provides policies and tools for the 

design, planning, conduct, and evaluation of exercises—known as the 

Homeland Security Exercise and Evaluation Program.22 FEMA requires 

that entities, such as states, receiving homeland security grant funding for 

their exercises use Homeland Security Exercise and Evaluation Program 

tools, such as reporting templates. We reviewed NEP guidance, such as the 

Implementation Plan, and found it does not clearly state the extent to 

which federal agencies are required to follow the Homeland Security 

Exercise and Evaluation Program planning and documentation guidance. 

As a result of this unclear guidance, we found that agency officials have 

varying interpretations of the requirements. A DOD and a Joint Staff 

official told us that NEP guidance does not require agencies to use the 

Homeland Security Exercise Evaluation Program even for NEP exercises; 

therefore, NORTHCOM uses the Joint Training System as the basis for 

planning, conducting, and assessing exercises. FEMA officials told us that 

federal agencies should use the Homeland Security Exercise Evaluation 

Program for exercises that require some level of interagency participation, 

 

22DHS, National Exercise Program Implementation Plan (Washington, D.C.: June 20, 

2008). 

Page 9 GAO-09-849 Homeland Defense
 

so that consistent approaches are used by the various participants.23

Neither DOD nor NORTHCOM guidance specifically addresses this issue.24

We recognize that DOD and NORTHCOM must meet their own mission 

and exercise requirements and the Joint Training System may be best 

suited for NORTHCOM’s exercises; however, all of the states we visited 

use Homeland Security Exercise and Evaluation Program guidance. We 

found that having differing sets of guidance makes exercise development 

more difficult and potentially limits the benefits for participating states. 

Further, inconsistent approaches to the development and content of 

postexercise documentation may affect the ability of organizations to fully 

learn lessons identified in exercises.25 We believe that achieving national 

preparedness requires a whole-of-government approach and is a shared 

responsibility among federal, state, and local governments and 

organizations and an integration of their various standards, policies, and 

procedures into the national system.26 In the absence of clear NEP 

guidance on this issue, DOD should ensure that its components clearly 

understand when use of NEP planning and reporting guidelines is 

appropriate. We are recommending that the Secretary of Defense and the 

Joint Staff provide specific guidance that defines the conditions under 

which the combatant commands should follow the Homeland Security 

Exercise and Evaluation Program planning and documentation 

requirements or the DOD’s Joint Training System should be modified for 

those civil support exercises. 

DOD provided written comments on a draft of this report and provided 

technical comments which we incorporated into the final report as 

appropriate. DOD agreed or partially agreed with all our recommendations 

and described actions it is taking or plans to take to implement them. DOD 

 

23FEMA officials stated that federal departments and agencies should be held accountable 

for meeting key Homeland Security Exercise Evaluation Program requirements, but that 

FEMA lacks the authority to ensure compliance. While we recognize that FEMA’s role is 

generally to coordinate, guide, and support, we believe that FEMA’s expanded leadership 

role under the Post-Katrina Act of 2006 provides FEMA opportunities to instill a shared 

sense of responsibility and accountability on the part of all agencies. GAO-09-369. The PostKatrina Act was enacted as Title VI of the Department of Homeland Security 

Appropriations Act, 2007, Pub. L. No. 109-295, (2006). 

24NORTHCOM guidance states the Vigilant Guard Program should use Homeland Security 

Exercise and Evaluation Program formats to encourage civilian agency participation. U.S. 

Northern Command, Commander’s Training Guidance: Fiscal Year 2009-2010, p. A-18. 

25Defense Science Board, Unconventional Operational Concepts and the Homeland. 

26GAO-09-369. 

Page 10 GAO-09-849 Homeland Defense
 

generally agreed with the recommendation to work with DHS to establish 

guidance and procedures for requesting state participation and involving 

states in planning, executing, and assessing exercises. DOD agreed that 

better coordination for interfacing with state officials can be achieved. 

However, DOD noted that NORTHCOM is the primary training audience 

for its exercises and that the recommendation should also be addressed to 

an interagency exercise planning committee. We noted NORTHCOM’s 

considerable efforts to involve state and interagency exercise partners. 

However, we believe that the effective involvement of and interaction with 

state and other federal partners is a critical component of improving and 

maintaining NORTHCOM’s preparedness for providing support to civil 

authorities regardless of whether it is a strictly NORTHCOM-sponsored 

exercise or an event conducted under the National Exercise Program. 

Therefore, developing procedures to improve coordination with the states 

can only assist DOD and all its interagency and state partners. We also 

noted that the interagency exercise planning committee is one venue at 

which DOD can effectively coordinate with its interagency partners. 

Although DOD agreed with the recommendation on a training plan for 

NORTHCOM staff and discussed current and future NORTHCOM training 

efforts, its response did not address the need for NORTHCOM staff to be 

provided more thorough training on specific emergency management 

organizations and structures of each state. We continue to believe that 

such training is needed since it is an important element of preparedness 

for exercises and for actually providing civil support in the event of a 

major incident. A summary of DOD’s comments and a summary of our 

response to these comments follow the Recommendations for Executive 

Action section of this report. DOD’s written comments are reprinted in 

appendix III. DHS also reviewed a draft of this report and provided 

technical comments, which we incorporated into the final report as 

appropriate. 

The U.S. homeland continues to face an uncertain, complex security 

environment with the potential for terrorist incidents and natural disasters 

which can produce devastating consequences. Ensuring an effective 

response will require that federal departments and agencies, states, and 

local governments conduct integrated disaster response planning and test 

these plans by exercising together. 

Background 

Training and Exercises Exercises play an instrumental role in preparing the nation to respond to 

an incident by providing opportunities to test emergency response plans, 

evaluate response capabilities, assess the clarity of established roles and 

Page 11 GAO-09-849 Homeland Defense
 

responsibilities, and improve proficiency in a simulated, risk-free 

environment.27 Short of performance in actual operations, exercises 

provide the best means to assess the effectiveness of organizations in 

achieving mission preparedness. Exercises provide an ideal opportunity to 

collect, develop, implement, and disseminate lessons learned and to verify 

corrective action taken to resolve previously identified issues.28 Sharing 

positive experiences reinforces positive behaviors, doctrine, and tactics, 

techniques, and procedures, while disseminating negative experiences 

highlights potential challenges in unique situations or environments or 

identifies issues that need to be resolved.29 According to the National 

Response Framework, well-designed exercises improve interagency 

coordination and communications, highlight capability gaps, and identify 

opportunities for improvement.30 There are various types of exercises 

ranging from tabletop exercises that involve key personnel discussing 

simulated scenarios in informal settings to a full-scale exercise, including 

many agencies, jurisdictions, and disciplines and a “boots on the ground” 

response, such as firefighters decontaminating mock victims. 

Military Mission and 

Organization 

DOD established the Office of the Assistant Secretary of Defense for 

Homeland Defense and Americas’ Security Affairs to oversee homeland 

defense activities for DOD, under the authority of the Under Secretary of 

Defense for Policy, and, as appropriate, in coordination with the Chairman 

of the Joint Chiefs of Staff. This office develops policies, conducts 

analysis, provides advice, and makes recommendations on homeland 

defense, defense support of civil authorities, emergency preparedness, and 

domestic crises management matters within the department. The assistant 

secretary assists the Secretary of Defense in providing policy directions to 

NORTHCOM and other applicable combatant commands to guide the 

 

27A training event focuses primarily on improving individual or collective ability to perform, 

such as academic or field training. An exercise focuses primarily on evaluating capability 

or an element of capability, such as a plan or policy. Necessary training should take place 

prior to an exercise. 

28U.S. Northern Command, Commander’s Training Guidance for FY09-10, A-20. 

29U.S. Northern Command, Lessons Learned Program and Corrective Action Board 

Process, Instruction 16-166 (Colorado Springs, Colo.: Mar. 16, 2009), 2. 

30 The framework is a guide to how the nation conducts all-hazards response and is 

intended to capture specific authorities and best practices for managing incidents that 

range from the serious but purely local, to large-scale terrorist attacks or catastrophic 

natural disasters. DHS, National Response Framework (Washington, D.C.: January 2008). 

Page 12 GAO-09-849 Homeland Defense
 

development and execution of homeland defense plans and activities. This 

direction is provided through the Chairman of the Joint Chiefs of Staff. 

The Chairman of the Joint Chiefs of Staff, as principal military advisor to 

the President and Secretary of Defense, has numerous responsibilities 

relating to homeland defense and civil support, including providing advice 

on operational policies, responsibilities, and programs. Furthermore, the 

Chairman of the Joint Chiefs of Staff and the Joint Staff are responsible for 

formulating joint training policy and doctrine.31 The Joint Staff assists the 

Chairman by facilitating implementation of the Chairman’s joint training 

programs, including the Joint Training System, Chairman’s sponsored 

exercise program, and joint exercise program. 

NORTHCOM NORTHCOM is the military command responsible for planning, organizing, 

and executing DOD’s homeland defense and civil support missions within 

its area of responsibility—the continental United States (including Alaska) 

and territorial waters (see fig. 1).32 Homeland defense is the protection of 

U.S territory, sovereignty, domestic population, and critical defense 

infrastructure against external threats and aggression. DOD is the primary 

federal agency responsible for homeland defense operations, such as air 

defense, and NORTHCOM is the combatant command responsible for 

commanding and coordinating a response to a homeland defense 

incident.33 To carry out its homeland defense mission, NORTHCOM is to 

conduct operations to deter, prevent, and defeat threats and aggression 

aimed at the United States. 

 

31Chairman of the Joint Chiefs of Staff, Instruction 3500.01E, Joint Training Policy and 

Guidance for the Armed Forces of the United States (Washington, D.C.: May 31, 2008). 

32DOD, Strategy for Homeland Defense and Civil Support, 8. 

33Homeland defense is considered DOD’s portion of the broader area of homeland security. 

DHS is the lead primary federal agency for homeland security issues, which is a concerted 

national effort to prevent terrorist attacks within the United States, reduce America’s 

vulnerability to terrorism, and minimize the damage and recover from attacks that do 

occur. Homeland Security Council, National Strategy for Homeland Security, 3; DOD, 

Strategy for Homeland Defense and Civil Support, 5. 

Page 13 GAO-09-849 Homeland Defense
 

Figure 1: NORTHCOM’s Area of Responsibility 

Canada

United States

Mexico

Sources: GAO analysis; Map Resources (map).

Alaska falls within

NORTHCOM’s area

of responsibility, but 

military forces in the 

state are assigned to 

U.S. Pacific Command.

NORTHCOM’s second mission is civil support or defense support of civil 

authorities. Civil support is DOD support to U.S. civilian authorities, such 

as DHS, for domestic emergencies, both natural and man-made, and 

includes the use of DOD personnel—federal military forces and DOD’s 

career civilian and contractor personnel—and DOD agency and 

component resources. Because these missions are complex and 

interrelated, they require significant interagency coordination. Civil 

support missions include domestic disaster relief operations for incidents 

such as fires, hurricanes, floods, and earthquakes. Such support also 

includes counterdrug operations and management of the consequences of 

a terrorist incident employing a weapon of mass destruction. DOD is not 

the primary federal agency for such missions (unless so designated by the 

Page 14 GAO-09-849 Homeland Defense
 

President) and thus provides defense support of civil authorities only 

when (1) state, local, and other federal resources are overwhelmed or 

unique military capabilities are required; (2) assistance is requested by the 

primary federal agency; and (3) NORTHCOM is directed to do so by the 

President or the Secretary of Defense. 34 See fig. 2 for the pathway for 

requesting DOD and NORTHCOM assistance during an incident. 

 

34DHS, National Response Framework; Joint Chiefs of Staff, Joint Pub. 3-28, Civil Support 

(Washington, D.C.: Sept. 14, 2007). 

Page 15 GAO-09-849 Homeland Defense
 

Figure 2: Pathways for Requesting Military Assistance for Incident Response 

Incident commander and 

local first responders

Local elected and 

emergency management 

officials

Governor and state officials, 

including emergency 

management officials

Secretary of Homeland Security/ 

FEMA Administrator 

Federal 

Coordinating 

Officer 

State 

Coordinating 

Officer 

Defense 

Coordinating 

Officer 

Designated forces support 

as directed 

Joint Field Office (Temporary)

Declares incident as either a major disaster or 

emergency under the Stafford Act 

President of the United States

Requests assistance directly to DOD 

Appoints Federal Coordinating Officer 

Delivers assistance 

Legend 

Request for federal assistance 

Request for federal assistance after Stafford 

declaration

Approves assistance 

Directs assistance (when assigned) 

Other federal 

departments

Incident occurs

Joint Task 

Force 

Commander 

(Temporary)

NORTHCOM 

Secretary of Defense 

Approves FEMA requests for 

assistance

Source: GAO analysis of DOD and DHS documentation. 

Established to coordinate federal 

support for response, but does not 

manage on-scene operations

FEMA Regional 

Administrator 

Page 16 GAO-09-849 Homeland Defense
 

NORTHCOM conducts or participates in exercises to improve readiness to 

perform its assigned missions. The command annually conducts 2 largescale exercises35—Ardent Sentry and Vigilant Shield—and participates in 

over 30 smaller command, regional, state, and local exercises. Each Ardent 

Sentry and Vigilant Shield training event emphasizes one of the key 

missions while at the same time including elements of the other. Ardent 

Sentry emphasizes the civil support missions; Vigilant Shield the homeland 

defense missions. The basis for NORTHCOM’s exercises is DOD’s Joint 

Training System. NORTHCOM’s Training and Exercise Directorate is 

responsible for planning and executing joint training, exercises, and 

education programs to ensure NORTHCOM is prepared to accomplish its 

assigned missions. 

DOD’s Joint Training 

System 

Due to the need to prepare for and conduct military operations to defend 

the United States and fight the nation’s wars, DOD has developed an 

established, authoritative, time-tested process for planning, conducting, 

and evaluating exercises in order to test and improve preparedness to 

meet its wide range of critical missions.36 NORTHCOM uses DOD’s Joint 

Training System as the basis to design, develop, and conduct exercises.37

The Joint Training System provides an integrated, requirements-based 

method for aligning training programs with assigned missions consistent 

with command priorities, capabilities, and available resources. The joint 

system consists of four phases beginning with the identification of critical 

capabilities required based on assigned missions, proceeding through the 

planning and scheduling of training events, the execution and evaluation 

of required training, and assessing training proficiency against required 

capability (see fig. 3). This process is designed to ensure that an 

organization’s training program is linked to the Joint Mission Essential 

 

35For purposes of this report, large-scale exercises are those involving multiple agencies 

and organizations, venues, and events, such as NORTHCOM’s Vigilant Shield and Ardent 

Sentry exercises. 

36DOD established the Joint Training System in fiscal year 1994. GAO, Military Training: 

Actions Needed to Enhance DOD’s Program to Transform Joint Training, GAO-05-548

(Washington, D.C.: June 21, 2005) 3, 19. 

37The documents that provide the written framework for the Joint Training System are 

Chairman of the Joint Chiefs of Staff, Instruction 3500.01E, Joint Training Policy and 

Guidance for the Armed Forces of the United States (Washington, D.C.: May 31, 2008); 

Manual 3500.03B, Joint Training Manual for the Armed Forces of the United States 

(Washington, D.C.: Aug. 31, 2007); and Manual 3500.04E, Universal Joint Task Manual 

(Washington, D.C.: Aug. 25, 2008). 

Page 17 GAO-09-849 Homeland Defense
 

Task List,38 the personnel executing the tasks are properly trained, and 

shortfalls in training are identified and corrected in order to improve 

readiness. The Joint Training Information Management System is an 

automated system specifically designed to assist users in managing 

elements of each of the four phases of the Joint Training System.39 During 

the execution phase, commanders and directors focus on executing and 

evaluating planned training events, which can be accomplished through 

academic training, exercises, or a combination of these activities.40 During 

the execution stage of the Joint Training System, the Joint Event Life 

Cycle provides a five-stage methodology for joint-event development 

design, planning, preparation, execution, and evaluation. For example, 

DOD components prepare for the execution of an exercise by conducting 

five conferences, such as the Concept Development Conference where 

exercise and training objectives are discussed and scenarios developed. 

Activities for the Joint Event Life Cycle are managed through the Joint 

Training Information Management System. 

 

38The Joint Mission Essential Task List outlines those tasks that are essential to a 

combatant command’s ability to perform assigned missions. The list is a key element in 

readiness assessment and reporting. 

39Specifically, the Joint Training Information System provides the joint community with an 

integrated software capability to identify, collect, analyze, store, and disseminate the data 

required to execute the Joint Training System. 

40Joint exercises have been characterized by some form of multiechelon, computer-assisted 

exercises which embody complex simulation. These may be either a field training exercise, 

command postexercise, or a tabletop exercise. 

Page 18 GAO-09-849 Homeland Defense
 

Figure 3: DOD’s Joint Training System 

Joint Training System

Identify 

capability 

requirements

Determine training 

resources

Identify 

training

requirements &

plan required 

training

Execute and

evaluate planned 

training

Execution

Evaluation

Design 

Joint event Planning

life cycle

Preparation

Source: DOD. 

Evaluating lessons learned and identifying issues for corrective actions are 

fundamental components of DOD’s training and exercise process. The 

Chairman of the Joint Chiefs of Staff provides policy, direction, and 

guidance for DOD’s Joint Lessons Learned Program.41 The objectives of 

this program are to collect and analyze observations from exercises and 

real world events; disseminate validated observations and findings to 

appropriate officials; identify and implement corrective actions; and track 

corrective actions until reobserved in a subsequent exercise or event to 

 

41Chairman of the Joint Chiefs of Staff, Joint Lessons Learned Program, 3150.25D, C-1. 

Page 19 GAO-09-849 Homeland Defense
 

ensure that the issue has been successfully resolved. Combatant 

commands, including NORTHCOM, execute lessons discovery, knowledge 

development, and implementation activities scaled to meet the command’s 

requirements while supporting and feeding into the Chairman’s Joint 

Lessons Learned Program by identifying lessons applicable across 

combatant commands and the services. 

The National Exercise 

Program 

The NEP was established in April 2007 under the leadership of the 

Secretary of Homeland Security to prioritize and coordinate federal, state, 

and local exercise activities and serves as the principal mechanism for 

examining the preparation of the federal government to respond to an 

incident and adopting policy changes to improve such preparation.42 The 

day-to-day staff-level coordination of the NEP is managed by the NEP 

Executive Steering Committee—a working group of the White House’s 

Domestic Readiness Group Exercise and Evaluation Sub-Policy 

Coordination Committee—and is chaired and facilitated by FEMA’s 

National Exercise Division. The steering committee is also responsible for 

framing issues and recommendations for the full coordination committee 

on exercise themes, goals, objectives, scheduling, and corrective actions.43

Figure 4 illustrates the major events and milestones of the NEP and 

NORTHCOM’s exercise program and table 1 provides information on 

related major documents. 

 

42DHS, National Exercise Program Implementation Plan. 

43Other steering committee members include DOD, the Office of the Director of National 

Intelligence, the Federal Bureau of Investigation, and the Departments of State, Justice, 

Energy, Transportation, and Health and Human Services. 

Page 20 GAO-09-849 Homeland Defense
 

Figure 4: National Exercise Program and U.S. Northern Command Exercise Program Time line: Major Events and Milestones 

(2001-2009) 

9/11 Terrorist

attacks

Homeland 

Security 

Presidential 

Directive 8

issued

Hurricane 

Katrina

Post Katrina

Act enacted 

Joint Chiefs of 

Staff Lessons

Learned 

Instructions

issued 

DOD NEP 

Instruction 

issued 

2001

2002

2003

2004

2005

2006

2007

2008

2009

NORTHCOM Established 

FEMA

• United Defense 03 Exercise

• Determined Promise 03 Exercise

• United Defense 04

• Determined Promise 04

• Vigilant Shield 06 Exercise

• Ardent Sentry 06 Exercise

• Vigilant Shield 07 

• Ardent Sentry 07

• NORTHCOM exercise

 guidance issued

• Vigilant Shield 09

National Exercise Program (NEP) Charter signed

National Exercise Program Implementation Plan approved

National Level Exercise 1-08 (Top Officials 4 + Vigilant Shield 08)

National Level Exercise 2-08 (Ardent Sentry 08)

• Revised Homeland Security Exercise and 

 Evaluation Program Guidance published

• National Preparedness Guidelines published

• National Response Framework published

FEMA merges with DHS DHS Established

FEMA reorganizes

Source: GAO Analysis of DOD and DHS documentation. 

• National Planning Scenarios Issued

• National Response Plan Published

Page 21 GAO-09-849 Homeland Defense
 

Table 1: Major Documents Related to the National Exercise Program and NORTHCOM 

Title Description 

Homeland Security Presidential Directive-8 Requires the development of the national preparedness goal; preparedness 

priorities; an assessment system; a training and exercise program; a system to 

collect, analyze, and disseminate information from exercises, training events, and 

actual incidents; equipment standards; and a federal response capability 

inventory. 

National Response Framework Establishes a comprehensive, national, all-hazards approach to domestic incident 

response. Replaced the National Response Plan. 

National Preparedness Guidelines Establishes a vision for national preparedness and provide a systematic approach 

for prioritizing preparedness efforts across the United States 

National Planning Scenarios Provides planning tools that represent a minimum number of credible scenarios 

depicting the range of potential terrorist attacks and natural disasters and related 

impacts facing the United States They form a basis for coordinated federal 

planning, training, and exercises. 

U.S. North American Aerospace Defense 

Command (NORAD)-NORTHCOM Instruction 

10-156 

Establishes the NORAD and NORTHCOM exercise program, provides an 

overview of the exercise program, describes the principal NORAD and 

NORTHCOM-sponsored exercises, and outlines the process for scheduling, 

planning, execution, and afteraction review of exercises. 

Chairman of the Joint Chiefs of Staff Instruction 

3150.25D 

Codifies the Joint Lessons Learned Information System as the DOD system of 

record for the Joint Lessons Learned Program. 

DOD Instruction 3020.47 Establishes policy, assigns responsibilities, and provides instructions for DOD 

participation in the National Exercise Program. 

Source: DOD and DHS. 

The NEP includes a series of national exercises projected on a 5-year 

exercise schedule. These exercises are organized into four tiers with each 

tier reflecting different requirements for interagency participation (see fig. 

5). 

Page 22 GAO-09-849 Homeland Defense
 

Figure 5: Description of NEP Tiers 

Tier I

Tier II

Tier III

Tier IV

White House directed, U.S. Government-wide 

strategy and policy focus, full federal agency 

participation. 5 Tier I excercises conducted annually 

(1 national-level exercise and 4 principal-level 

exercises).

Federal Department/Agency directed, 

U.S. government-wide strategy and policy focus. Full 

Federal Agency participation in person or through 

simulation. No more than 3 Tier II exercises conducted 

annually.

Other federal exercises, regional operational, tactical or 

organizational-specific focus. Full federal agency 

participation is optional. The number of exercises

conducted annually depends upon individual entities’ 

needs and requirements. 

Non-Federal Exercises, State, Territorial, Local and or 

tribal governments and/or private sector focus. Federal 

Agency participation is optional. The number of 

exercises conducted annually depends upon individual 

entities’ needs and requirements.

Source: FEMA.

FEMA administers the NEP and maintains the Homeland Security 

Exercise and Evaluation Program—a capabilities and performance-based 

exercise program—to provide standardized policy, methodology, and 

terminology for exercise design, development, conduct, evaluation, and 

improvement planning. DHS maintains policy and guidance for this 

program.44

Similar to DOD’s Joint Training System, the Homeland Security Exercise 

Evaluation Program uses an exercise life cycle with five phases: 

foundation, design and development, conduct, evaluation, and 

improvement planning. This program also provides document templates 

for exercise planning and evaluation and a collection of interactive, on-line 

systems for exercise scheduling, design, development, conduct, 

evaluation, and improvement planning, referred to as the Homeland 

 

44DHS, Homeland Security Exercise and Evaluation Program, Volumes 1-5 (Washington, 

D.C.: February 2007). 

Page 23 GAO-09-849 Homeland Defense
 

Security Exercise Evaluation Program Tool Kit (see fig. 6).45 FEMA also 

has additional resources to support exercises. For example, exercise 

stakeholders can access FEMA’s Lessons Learned Information Sharing 

system, an interagency Web site for posting lessons learned and sharing 

best practices, to learn about promising practices that could facilitate 

exercise activities.46

 

45https://hseep.dhs.gov/pages/1001_Toolk.aspx. 

46https://llis.gov. 

Page 24 GAO-09-849 Homeland Defense
 

Figure 6: Homeland Security Exercise and Evaluation Program Tool Kit 

Source: GAO analysis of FEMA documentation.

Design and Development System

A project management tool and comprehensive tutorial for the design, 

development, conduct, and evaluation of exercises.

Exercise Evaluation Guide Builder (Beta)

An online application that enables users to customize exercise evaluation 

guides and templates.

Master Scenario Events List Builder (Beta)

A tool that enables users to create customized master scenario events list 

formats by selecting from a list of data fields.

Corrective Action Program System

An online application that enables users to prioritize, track, and analyze 

improvement plans developed from exercises and real-world events.

National Exercise Schedule System

A national online comprehensive tool that facilitates scheduling and 

synchronization of national-level, federal, state, and local exercises.

Exerci

se pro

gram mana

gement

Foundation

Design, development, and conduct

Evaluation and Improvement Planning

Page 25 GAO-09-849 Homeland Defense
 

NORTHCOM’s 

Exercise Program Is 

Consistent with 

DOD’s Joint Training 

System, but Exercise 

Reporting is 

Inconsistent 

NORTHCOM’s Exercise 

Program is Consistent with 

DOD’s Joint Training 

System 

NORTHCOM’s Commander’s Training Guidance requires that NORTHCOM 

establish a training and exercise program consistent with the Joint 

Training System and establishes that training efforts and resources will be 

focused on two large-scale exercises annually.47 The Joint Training System 

requires, among other things, that an organization’s training objectives be 

linked to its Joint Mission Essential Task List and include the use of the 

Joint Events Life Cycle for planning, conducting, and assessing exercises.48

We found that NORTHCOM has developed a comprehensive exercise 

program consistent with DOD’s Joint Training System. For example, 

NORTHCOM uses the Joint Training Information Management System to 

link training objectives with its Joint Mission Essential Task List. 

NORTHCOM officials enter information on task performance of exercise 

participants into the Joint Training Information Management System to 

evaluate the extent to which the command is trained based on 

performance requirements in the Joint Mission Essential Task List.49

NORTHCOM also uses the Joint Training Information Management System 

to manage the Joint Events Life Cycle for its large-scale exercises, 

including planning exercise milestones and developing a time line that 

 

47U.S. Northern Command, Commander’s Training Guidance for FY 09-10, A-11. 

48The Joint Events Life Cycle provides a sequence of the inputs, process steps, and outputs 

necessary to execute any size exercise. This process consists of five stages including 

design, planning, preparation, execution, and evaluation. A series of planning conferences 

provide discrete breakpoints between each stage. Chairman of the Joint Chiefs of Staff, 

Joint Training Manual for the Armed forces of the United States, E-6. 

49NORTHCOM is given an overall rating in the system for the command’s current status of 

“trained”, “needs practice,” or “untrained” for each Joint Mission Essential Task. U.S. 

Northern Command, Memorandum on NORAD and USNORTHCOM Joint Training Plan 

FY08-09(Colorado Springs, Colo.: July 30, 2007, A-32-A-35; Chairman of the Joint Chiefs of 

Staff, Joint Training Manual for the Armed Forces of the United States, F-5-F-6. 

Page 26 GAO-09-849 Homeland Defense
 

allows exercise planners to see where they are in the event life-cycle 

process. For example, NORTHCOM holds five planning conferences for 

each exercise, including a concept development conference, where 

exercise and training objectives are discussed and scenarios developed.50

We also found that NORTHCOM has conducted 13 large-scale exercises 

since it was created in 2002, generally including 2 exercises each year (see 

table 2). Vigilant Shield is held in the fall and focuses primarily on 

NORTHCOM’s homeland defense mission, and Ardent Sentry is generally 

conducted in the spring and focuses on defense support of civil 

authorities. 

Table 2: Large-Scale NORTHCOM Exercises Since 2002 

Exercise Dates Scenarios 

Ardent Sentry 09 June 18-24, 2009 • Flooding along Mississippi River 

• Domestic terrorist organization releases foot and mouth disease at 4-H expo 

• Anthrax released into the food supply 

• Train derailments in Iowa and Kansas resulting in chemical releases 

• Domestic terrorist organization launches attack against a Wyoming Air Force 

Base 

• U.S. and Canadian maritime incidents 

Vigilant Shield 09 Nov. 12-18, 2008 • Earthquake along San Andreas Fault 

• Homeland defense vignettes associated with mission areas 

Ardent Sentry 08 May 1-8, 2008 • Category 4 hurricane strike and terrorist threat affecting National Capital 

Region 

• Accidental VX nerve gas release in Oregon 

• Multiple terrorist attacks in Washington to exercise both response to chemical 

weapons attack and specific defense support of civil authorities capabilities 

Vigilant Shield 08 Oct. 15-20, 2007 • Series of exploded radiological dispersal devices 

• Aircraft accidents 

• Reverse no-fly list 

• Deployment of a command assessment element 

Ardent Sentry 07 April 30 - May 17,2007 • Critical infrastructure protection 

• Maritime operations 

• Detonation of a 10-kiloton improvised nuclear device in Indiana 

• Category III hurricane impacting the New England area 

 

50Chairman of the Joint Chiefs of Staff, Joint Training Manual for the Armed Forces of the 

United States, E-6-E-12. 

Page 27 GAO-09-849 Homeland Defense
 

Exercise Dates Scenarios 

Vigilant Shield 07 Oct. 16-27, 2006 and 

Dec. 4-15, 2006 

• Crash and explosion of plane carrying nuclear weapons 

• Maritime attacks/events 

• Rogue aircraft and North American Air Defense Command Air and Strategic 

Threat events 

• Strategic Ballistic Missile Defense Force Protection 

• Defense against cyber attacks 

Ardent Sentry 06 May 8-18, 2006 • Hurricane strike on New Orleans, Louisiana 

• Explosion of a railcar carrying phosgene 

• U.S. and Canadian Infrastructure attack 

• Chemical attack 

• Radiological incident 

• Avian flu pandemic outbreak scenario 

Vigilant Shield 06 Nov. 1-10, 2005 • Chemical scenario 

• Critical infrastructure protection of a nuclear power plant 

• Biological scenario 

• Ballistic missile defense threat 

• Aerospace defense 

Ardent Sentry 05 Apr. 4-9, 2005 • Exercise deployment and employment of Joint Task Force-Civil Support in 

support of FEMA Region I mission assignment requests in a chemical 

response 

• Response to a biological event 

• Protection of critical infrastructure (nuclear power plant) 

Determined Promise 04 Aug. 5-10, 2004 • Simultaneous terrorist air events (hijacked aircraft, aircraft with terrorists on 

watch list, low-altitude defense and swarm attack by business jet aircraft) 

• Explosion of container with radiological dispersal device in Long Beach, 

California harbor 

• Train derailment and chemical spill 

• Multiple sarin, mustard, and chlorine gas attacks in Virginia 

• Maritime intercept operations 

• Attack by unmanned aerial vehicles with high-explosive warheads fired from 

oil platform 

• Cyber security event 

• Ballistic missile defense 

Unified Defense 04 Feb. 19-25, 2004 • Air threats to Alaska, Canada, and the United States 

• Category IV+ hurricane in Corpus Christi, Texas 

• Nuclear detonation in Texas 

• Quick reaction force deployments to Texas and Alaska 

• Transportation security and U.S. Coast Guard incidents in Alaska 

• Maritime intercept operations events 

• Ballistic missile defense threat 

Page 28 GAO-09-849 Homeland Defense
 

Page 29 GAO-09-849 Homeland Defense 

Exercise Dates Scenarios 

Determined Promise 03 Aug. 18-28, 2003 • Terrorist attack of a biological weapon of mass destruction in Clark County, 

Nevada 

• Wildfires in the western United States 

• 2003 World Gymnastics Championship in California 

• Ongoing operational plan load-out operations 

• Sustained, diverse maritime events 

• Air threat to Alaska 

• High-interest vessel threat 

• Strategic infrastructure threats in the northwest United States 

• Category III hurricane in the southeast United States 

• Train derailment in Kentucky 

• Shipping and security issues in Alaska 

Unified Defense 03 Feb. 6-13, 2003a • National special security event - Super Bowl 

• Joint Task Force-Civil Support held in strategic reserve for State of the Union 

Address and Super Bowl 

• Nuclear threat along southwest U.S. border 

• Border surveillance 

• Threat to port 

• Consequence management in northeast United States 

• Natural disaster in Alaska 

Source: GAO analysis of NORTHCOM diocuments. 

a

Unified Defense 03 started late due to the Columbia Shuttle disaster and ended early due to a level 

Orange alert in the Homeland Security Alert System. 

NORTHCOM guidance outlines the postexercise documentation required 

to be completed for each exercise, including quick look, after-action, and 

exercise summary reports; provides a time line for the completion of these 

documents; and includes general direction that these documents follow 

the same focus areas as the collection management plan—the source 

document from which exercise analysts identify, examine, and 

recommend emerging issues and trends.51 We found that NORTHCOM has 

generally completed exercise summary reports for its exercises; however, 

neither NORTHCOM nor Joint Forces Command officials could locate an 

exercise summary report for Unified Defense 03. 52 In addition, 

 

51U.S. Northern Command, Operations: Exercise Program, Instruction 10-156 (Colorado 

Springs, Colo.: June 2008), 20. 

52While NORTHCOM was unable to provide exercise summary or quick-look reports for 

United Defense 03, they did provide pre-exercise briefings. NORTHCOM officials told us 

that the exercise started late due to the Columbia shuttle disaster and ended early due to a 

level Orange alert in the Homeland Security Alert System. 

NORTHCOM’s 

Documentation of 

Exercises Is Generally 

Timely but Inconsistent 
 

postexercise documentation is not consistently included on NORTHCOM’s 

portal or the Joint Training Information Management System. NORTHCOM 

guidance issued in June 2008 provides a time line for the completion of 

postexercise documents and has been applicable to 2 subsequent 

exercises–Ardent Sentry 08 and Vigilant Shield 09. According to the 2008 

guidance, the exercise summary report is to be submitted to the 

NORTHCOM Commander within 90 days of completing an exercise.53 The 

Ardent Sentry 08 and Vigilant Shield 09 exercise summary reports were 

issued 99 days and 92 days, respectively, after the completion of each 

exercise. Overall, we reviewed exercise summary reports for 11 of 

NORTHCOM’s large-scale exercises that have taken place since 2003. 

Seven of the 11 exercise summary reports were issued within 100 days. 

Four of the reports were issued later than 100 days, and 1 of 

NORTHCOM’s earlier reports was issued in less than 30 days. 

NORTHCOM guidance states that exercise summary reports should 

provide the official description of the exercise, identify significant lessons 

learned, and be targeted toward a national audience.54 Guidance also 

requires that exercise summary reports follow the same focus areas as the 

collection management plan—the source document from which exercise 

analysts identify, examine, and recommend emerging issues and trends.55

We found that NORTHCOM’s exercise summary reports generally included 

an executive summary, training objectives, and the exercise’s major 

scenarios and events, but did not consistently include lessons learned, 

exercise strengths and weaknesses, or clear recommendations. The 

exercise summary reports that included a section on lessons learned 

lacked details. For example, 6 of the 11 exercise summary reports we 

reviewed included an identified lessons learned section, and just 1 of these 

6 reports—Unified Defense 04—provided additional information on 

lessons learned beyond identifying the title of each observation and the 

status of the observation in the lessons learned management system. As 

 

53U.S. Northern Command, Operations: Exercise Program. The instruction, issued in June 

2008, also requires exercise quick look reports be issued within 30 days after the 

completion of an exercise. The instruction is only applicable to Ardent Sentry 08 and 

Vigilant Shield 09, as the post-exercise documents for these reports were issued after June 

2008. Post-exercise documents for all other exercises reviewed by GAO were issued prior 

to June 2008. 

54U.S. Northern Command, Operations: Exercise Program, 20. 

55U.S. Northern Command, Operations: Exercise Program, 20. 

Page 30 GAO-09-849 Homeland Defense
 

discussed later in this report, access to this system is required in order to 

obtain any additional information on the lesson learned.56

We also found that NORTHCOM exercise summary reports have not 

followed the same focus areas as collection management plans.57 For 

example, none of the seven exercise summary reports for NORTHCOM 

exercises conducted since Hurricane Katrina in 2005 reported on the 

information identified in the collection management plans’ focus areas. 

Inconsistencies in exercise documentation may be occurring because DOD 

and NORTHCOM guidance do not require a standard format or specific 

content for postexercise documentation. Although NORTHCOM uses other 

methods to document exercises, such as the Joint Training Information 

Management System, this system does not include a complete record of 

each exercise. For example, the Joint Training Information Management 

System does not include the lessons learned from an exercise. In addition, 

access to this system is generally limited to DOD officials. Recognizing the 

need for a complete and consistent record of each exercise, DHS’s 

Homeland Security Exercise Evaluation Program provides a template for 

exercise documentation, including format and content.58 NORTHCOM 

used this template for National Level Exercise 2-08, but does not use

template for its own exercises.

 the 

 

59 Despite differences in the requirements 

and complexities of NORTHCOM’s and DHS’ exercise programs, the lack 

of a complete and consistent record of each exercise lessens the extent to 

which NORTHCOM can ensure it has trained to key focus areas. Further, it 

deprives the command of a key source of historical information upon 

which to base current and future assessments of exercises and a 

56Access to JLLIS is restricted in order to safeguard sensitive information because of 

concerns that adversaries may be able to exploit any weaknesses contained in those 

records. For users to access the unclassified portion of the JLLIS they must have a DOD 

Common Access Card, a card reader, and appropriate computer software prior to 

requesting access. 

57Collection Management Plans are generally linked to the Joint Mission Essential Tasks 

and exercise objectives. Focus areas include a wide range of operations, such as 

coordinating and integrating regional interagency activities and coordinating consequence 

management in theater. U.S. Northern Command, Operations: Exercise Program, 20. 

58DHS, Homeland Security Exercise Evaluation Program, Volume III: Exercise 

Evaluation and Implementation Plan, (Washington, DC: February 2007), A-1 – B-2. 

59U.S. Northern Command, North American Aerospace Defense Command and United 

States Northern Command National Level Exercise 2-08: 1-8 May 2008 Exercise 

Summary Report, (Colorado Springs, Colo.: Aug. 15, 2008)1-25. 

Page 31 GAO-09-849 Homeland Defense
 

consistent venue for sharing lessons learned with interagency partners and 

states. 

Interagency Partners 

and States Have 

Participated In 

NORTHCOM’s 

Exercises, but 

Challenges Remain 

with Including States 

in Planning, 

Conducting, and 

Assessing Exercises 

Federal Agencies and 

States Have Participated in 

NORTHCOM Exercises 

NORTHCOM recognizes the importance of exercising with key partners in 

all its missions and that, in order to achieve its goal of being trained and 

ready to execute joint operations and ensure a seamless operating 

environment, NORTHCOM should maximize exercise participation with 

federal, state, and local agencies and National Guard units.60 NORTHCOM 

has included interagency partners, such as DHS, FEMA, and the U.S. Coast 

Guard, and several states in its large-scale exercises (see table 3). We 

found that 17 civilian federal agencies and organizations have participated 

to varying degrees in one or more of the seven large-scale NORTHCOM 

exercises that have occurred since Hurricane Katrina made landfall in 

August 2005. Seventeen states have participated in NORTHCOM exercises 

since that time, and 8 of these states—Arizona, California, Connecticut, 

Indiana, Michigan, Oregon, Rhode Island, and Washington—played a 

major role by having a portion of the exercise conducted in the state and 

 

60U.S. Northern Command, Commander’s Training Guidance for FY09-10; Chairman of 

the Joint Chiefs of Staff, Joint Training Manual for the Armed Forces of the United 

States, CJCSM 3500.03B (Aug. 31, 2007). 

Page 32 GAO-09-849 Homeland Defense
 

having various state agencies and officials participate.61 For example, 

Indiana and Rhode Island played major roles in Ardent Sentry 07 for the 

detonation of a 10-kiloton improvised nuclear device and category III 

hurricane impacting the New England region, respectively. Both states 

established emergency operating centers and exercised large numbers of 

state emergency management personnel. 

Table 3: Interagency Partner and State Participation in Large-scale NORTHCOM Exercises Since 2006 

Exercise 

Other federal agencies 

and organizations States 

Vigilant Shield 06 • Department of Homeland Security 

• Other federal organizations in the areas of homeland defense 

and defense support of civil authoritiesb

• Connecticut 

• New Jersey 

Ardent Sentry 06 • Coast Guard Sector Detroit District 9 

• Department of Energy 

• Department of Health and Human Services 

• Department of Homeland Security 

• Department of Justice 

• Department of State 

• Department of Transportation 

• Environmental Protection Agency 

• Federal Aviation Administration 

• Federal Bureau of Investigation 

• FEMA 

• Transportation Security Agency 

• U.S. Coast Guard/Atlantic Area 

• U.S. Department of Agriculture 

• Arizona 

- Division of Emergency 

 Management 

- Joint Terrorist Task Force 

- Tactical Intelligence Center 

• Maine 

• Michigan 

- Detroit Metropolitan Airport 

- State Emergency Operation 

Center 

- State Police 

- Service Emergency 

Preparedness Liaison Officer 

- St. Clair County Regional 

Response Team 

- Wayne County 

 

61We visited all of these states except for Connecticut and Indiana. We visited Indiana 

during a review of NORTHCOM’s planning and interagency coordination in the spring of 

2007. We also met with Nevada officials who participated in Determined Promise 03. The 

remaining 9 of the 17 states played lesser roles in the exercises. 

Page 33 GAO-09-849 Homeland Defense
 

Exercise 

Other federal agencies 

and organizations States 

Vigilant Shield 07 • Department of Energy 

• DHS National Interagency Simulation Cell 

• Department of Justice 

• Department of Transportation 

• FEMA 

• DHS Principal Federal Official 

• U.S. Coast Guard 

• Department of Agriculture 

• Environmental Protection Agency 

• Federal Bureau of Investigation 

• Department of Health and Human Services 

• DHS Immigration and Customs Enforcement 

• National Geospatial Intelligence Agency 

• DHS Transportation Security Administration 

• National Security Agency 

• Veterans Administration 

• Arizona 

- Pima County Emergency 

Management 

- Tucson Police and Fire 

• Washington, DC 

• Alaska 

Ardent Sentry 07 • DHS-National Operations Center 

• FEMA 

• FEMA Region I 

• FEMA Region II 

• FEMA Region X 

• Federal Bureau of Investigation 

• U.S. Coast Guard Pacific Area 

• Indiana 

- Department of Homeland Security

- Indianapolis, Indiana 

- Marion County, Indiana 

• Rhode Island 

• Connecticut 

• Massachusetts 

• Vermont 

• New Hampshire 

• Maine 

• New York 

• Ohio 

• Illinois 

• Alaska 

National Level 

Exercise 1-08/Vigilant 

Shield 08

• FEMA 

• DHS-National Operations Center 

• FEMA Regions IX and X 

• U.S. Coast Guard Pacific Area 

• Oregon 

- City of Portland 

Arizona 

- City of Phoenix 

• Guama

National Level 

Exercise 2-08/Ardent 

Sentry 08

• Department of Homeland Security 

• FEMA National Exercise Division 

• Other federal organizations in the areas of homeland defense 

and defense support of civil authoritiesb

• Washington, DC 

• Oregon 

• Washington 

Page 34 GAO-09-849 Homeland Defense
 

Exercise 

Other federal agencies 

and organizations States 

Vigilant Shield 09 • FEMA Region IX 

• Other federal organizations in the areas of homeland defense 

and defense support of civil authoritiesb

• California 

Ardent Sentry 09 • Dept of Energy 

• Dept of Health and Human Services 

• Dept of Homeland Security 

• Environmental Protection Agency 

• Federal Aviation Administration 

• Federal Bureau of Investigation 

• FEMA Region VII 

• FEMA Region VIII 

• U.S. Coast Guard Operational Command 

• FEMA National Response Coordination Center 

• Iowa 

- Dept of Agriculture 

- Dept of Health 

- Dept of Public Safety 

- Homeland Security & Emergency 

Management Division 

- Iowa State Emergency 

Operations Center 

- City of Des Moines Fire Dept. 

- City of Des Moines Police Dept. 

• Kansas 

- Dept of Emergency Management 

- Dept of Health/Environment 

- Dept of Agriculture 

- Dept of Animal Health 

- Dept of Transportation 

- Highway Patrol 

• Nebraska 

• Missouri 

• Wyoming 

Source: GAO analysis of NORTHCOM exercise documents. 

a

The US territory of Guam is in U.S. Pacific Command’s area of responsibility. 

b

Exercise summary reports for Ardent Sentry 08 and Vigilant Shield 06 and 09 indicate that other 

federal organizations in the areas of homeland defense and defense support of civil authorities 

participated in the exercises, but the reports do not provide specific names of organizations and we 

were unable to identify these organizations in the Joint Training Information Management System. 

State emergency management and National Guard officials told us that 

they participated in NORTHCOM exercises because they wanted to better 

understand the (1) capabilities that NORTHCOM could bring to the 

response to an incident and (2) command and control issues of the troops 

in a state when NORTHCOM is involved. We previously reported that 

states’ participation in NORTHCOM exercises helps to build relationships 

and improve coordination.62 Officials from all of the states we met with 

 

62GAO-08-252. 

Page 35 GAO-09-849 Homeland Defense
 

told us that they derived benefits from their participation in these 

exercises. For example, state emergency management officials from three 

states told us that first-hand interaction with federal military forces and 

the opportunity to observe the federal response to an incident was 

beneficial. In addition, two state emergency management and National 

Guard officials told us that NORTHCOM officials were professional, welltrained, and helpful. Further, officials from five states told us that 

NORTHCOM provided beneficial resources, such as funds for travel to 

attend exercise planning conferences and contractor staff to help state 

officials prepare exercise scripts and injects.63 Finally, officials from two 

states told us that the benefits of working with NORTHCOM included 

gaining an understanding of the resources and capabilities that 

NORTHCOM can provide, as well as understanding how NORTHCOM 

coordinates its response through FEMA. 

NORTHCOM is also attempting to include states in exercises through the 

Vigilant Guard Program. The goal of the Vigilant Guard Program is to 

enhance National Guard and State emergency management agency 

preparedness to perform their homeland defense and Defense Support to 

Civil Authorities roles and responsibilities. It focuses on State Guard Joint 

Force Headquarters coordination with the state emergency management 

agency and Joint Task Force-State operations and involves multiple states 

and agencies. The program began in September 2004 and included one 

exercise in fiscal year 2005. Now the plan is to conduct four exercises 

annually. NORTHCOM was given management responsibility for the 

Vigilant Guard exercises in 2007, although the National Guard Bureau 

retains responsibility for budgeting for these events. Two of the four 

annual Vigilant Guard exercises are to be linked to major combatant 

command exercises, usually NORTHCOM’s Ardent Sentry and Vigilant 

Shield. States hosting a Vigilant Guard exercise determine the objectives 

for these events, and NORTHCOM provides support. Separate planning 

begins for these Vigilant Guard exercises prior to the related planning 

meetings for any linked NORTHCOM exercise. NORTHCOM’s Ardent 

Sentry 09 is linked with a Vigilant Guard exercise in Iowa with scenarios 

including a train derailment and a chemical spill, an epidemic outbreak, 

and terrorism incident. 

 

63 Injects are pieces of information presented to the training audience during an exercise in 

a manner similar to real-world operations to guide the training audience toward the 

accomplishment of the training objectives. 

Page 36 GAO-09-849 Homeland Defense
 

A key element to developing effective working relationships with all states 

is a well-thought out and consistent process for including the states in 

planning, conducting, and assessing exercises. Without such a process, 

states may be unwilling to participate in future NORTHCOM exercises, 

impacting the seamless exercise of all levels of government and potentially 

affecting NORTHCOM’s ability to provide support to civil authorities. We 

found that challenges remain which have resulted in inconsistencies in the 

way that NORTHCOM involves the states in its exercises. One of DOD’s 

challenges is adapting its exercise system and practices to accommodate 

the coordination and involvement of other federal, state, local, and tribal 

agencies that do not have the same kinds of practices or level of planning 

effort. Differences in exercise culture stem from differences in missions, 

experience, authority, scope, and resources available to DOD, interagency 

partners, and states. DOD has an established, authoritative, time-tested 

process for planning, conducting, and evaluating exercises in order to test 

and improve preparedness to meet its wide range of critical missions. 

Within DOD, training and exercises are considered a vital component of its 

overall mission of defending the national interests and significant 

resources are devoted to these activities. In contrast, DHS, as the lead for 

interagency homeland security efforts, is a new agency and has faced 

challenges since it was created due to frequent reorganization and not 

being fully staffed.64 DHS and other civilian agencies and state and local 

governments have day-to-day missions and responsibilities that may take 

priority over exercises and often do not have the resources or experience 

to participate in or conduct exercises. For example, DOD exercises often 

are conducted 24 hours a day, 7 days a week and may last a week or more 

to enhance the realism of the exercise, while civilian agencies generally 

participate 8 hours per day, usually—according to NORTHCOM officials—

during normal business hours, and do not exercise longer than a few days. 

Therefore, DOD exercises are generally longer in duration, more resource 

intensive, and involve more participants than other federal and state 

exercises. Furthermore, DOD views itself as the last line of defense and 

often exercises until resources are exhausted to fully assess capabilities 

and identify areas needing improvement. Civilian agencies and states may 

prefer not to exhaust resources during an exercise in order to avoid 

appearing unprepared for an incident and the associated political 

controversy. 

NORTHCOM Faces 

Challenges Involving 

Interagency Partners and 

States in Planning, 

Conducting, and Assessing 

Exercises 

 

64Defense Science Board, Unconventional Operational Concepts and the Homeland, viii, 

xiii, 38. 

Page 37 GAO-09-849 Homeland Defense
 

Another challenge that NORTHCOM faces is exercising with the various 

states and territories within its area of responsibility considering the legal 

and historical limits of the constitutional federal-state structure. The states 

have a wide range and type of civilian state agencies responsible for 

emergency management, some of which are headed by the Adjutant 

General of the state, who also heads the military department or National 

Guard, and others are completely separate entities (see table 4). 

Table 4: Emergency Management Structures of the States We Visited 

State Exercise(s) Emergency management structure 

Arizona Ardent Sentry 06 

Vigilant Shield 07 

• The Adjutant General is the director of the Department of Emergency and Military 

Affairs, which is a state agency combining Arizona’s Army and Air National Guard 

with Joint Military Programs and Emergency Management. 

California Vigilant Shield 09 

Determined Promise 4 

• The California Emergency Management Agency is a Governor’s cabinet-level 

agency that is led by a Secretary who reports to the Governor. 

• The Adjutant General is the head of the California State Military Department 

under which the National Guard operates. 

Michigan Ardent Sentry 06 • The Civilian Emergency Management and Homeland Security Division is located 

in the State Police Department. 

• The Adjutant General is the director of the Department of Military and Veteran 

Affairs under which the National Guard operates. 

Nevada Determined Promise 03 • The Division of Emergency Management is part of the Department of Public 

Safety (which also includes the Highway Patrol and State Fire Marshall). 

• The Adjutant General is the head of the Nevada Office of the Military which is also 

known as the National Guard. 

Oregon Vigilant Shield 08

National Level 

Exercise 1-08

• The Oregon Office of Emergency Management operates administratively under 

the Adjutant General and the Oregon Military Department. The Adjutant General 

also directs, manages, and supervises the Oregon National Guard. The Office of 

Emergency Management’s operations, however, are kept separate from those of 

the Oregon National Guard. 

Rhode Island Ardent Sentry 07 • The Adjutant General heads the Emergency Management Agency and the 

National Guard. These offices are co-located with the National Guard playing a 

supporting role to the Emergency Management Agency. 

Washington Ardent Sentry 08

National Level 

Exercise 2-08

• The Adjutant General heads the Military Department which houses the 

Washington Emergency Management Division, National Guard, Air National 

Guard, and State Guard. 

Source: GAO analysis of NORTHCOM documents and information provided by states. 

Working with states has been the responsibility of the National Guard and 

is relatively new for a federal military command like NORTHCOM. 

NORTHCOM officials face challenges in dealing with the various civilian 

agencies, differing emergency management structures, capabilities, and 

needs of the states. For example, for Ardent Sentry 08 (linked with 

National Level Exercise 2-08), NORTHCOM planned a scenario involving a 

Page 38 GAO-09-849 Homeland Defense
 

chemical bomb attack in Seattle, Washington without consulting the state 

health department or civil support team—the agencies responsible for 

responding to a chemical or biological attack. State officials told us that 

NORTHCOM invited the health department to participate once state 

officials informed them that they should be involved, but that the scenario 

was already locked in without the input of this key participant. DOD 

officials told us that they rely on FEMA regional offices to provide 

information on state agencies. However, we believe that NORTHCOM 

officials should have determined if all relevant agencies were included in 

the exercise when directly interacting with state officials during the 

scenario development and other planning conferences, before the 

scenarios were locked in. Washington emergency management officials 

told us that this affected the realism of the exercise. 

NORTHCOM also faces challenges in balancing its training objectives with 

those of state agencies and organizations. State and local governments 

seek to exercise their first responder capabilities before having their 

resources overwhelmed and needing to seek federal assistance. On the 

other hand, NORTHCOM seeks to exercise its capability to provide 

support to civil authorities when local, state, and other federal resources 

are exhausted. This necessarily requires scenarios that exceed the states’ 

capabilities and that stress DOD capabilities. Officials from four of the 

seven states we interviewed told us that NORTHCOM’s exercise scenarios 

appeared unrealistic, overwhelmed their states too soon during the 

exercise, or did not allow states to fully exercise their own training 

objectives.65 For example, the scenario for Ardent Sentry 06 included 

multiple improvised explosive devices detonating over a 4-day period in 

various sites, such as the City of Detroit, St. Clair and Wayne Counties, 

Michigan, and Windsor, Ontario, Canada with over 14,000 fatalities and a 

simultaneous pandemic flu outbreak in Michigan.66 State emergency 

management officials told us that such a large number of casualties would 

overwhelm state resources almost immediately, and therefore precluded 

 

65Creating an exercise scenario event that is believable is critical to engaging participants in 

a level of play and experiential learning that will be long lasting. Defense Science Board, 

Unconventional Operational Concepts and the Homeland, 55. 

66U.S. Northern Command, Northern American Aerospace Defense Command and United 

States Northern Command Ardent Sentry 06 Exercise Summary Report (Colorado 

Springs, Colo.: Sept. 27, 2006), 4-6. 

Page 39 GAO-09-849 Homeland Defense
 

fully exercising training objectives for state and local responders.67

Officials from these states told us that because they did not have the 

opportunity to exercise their own training objectives, they believed 

NORTHCOM was using them as a training tool. 68 A NORTHCOM official 

told us that NORTHCOM needs the states to participate in exercises and, 

therefore, will be flexible to accommodate other organizations’ training 

objectives; however, NORTHCOM ultimately has its own objectives to 

exercise. Officials from five of seven states noted that, for example, they 

face budget and staffing limitations, and playing a major role in a 

NORTHCOM exercise often requires establishing a state emergency 

operations center with numerous staff and agencies involved. Given the 

expansive scenarios NORTHCOM uses to guide its exercises and the 

perception of half of the states we visited that this limits the benefits to 

them, we believe that the states may be less likely to expend scarce 

resources to participate in future NORTHCOM exercises. 

Inconsistencies with how NORTHCOM involves states in planning, 

conducting, and assessing exercises is occurring in part because 

NORTHCOM officials lack experience dealing with the various state 

agencies and emergency management structures. Inconsistencies are also 

occurring because NORTHCOM has not established an informed, 

consistent process for including states in its exercises. One aspect of this 

process is the way that NORTHCOM requests state participation in its 

exercises. Currently, NORTHCOM has various processes for requesting 

that other federal departments and agencies participate in its exercises, 

such as making the request through the Joint Staff. FEMA officials told us 

that requests for state participation in NORTHCOM exercises should be 

made through FEMA’s regional offices. However, because NORTHCOM 

does not have an established process for requesting state participation, 

officials from the states we visited told us that NORTHCOM officials made 

requests informally and in a variety of ways, including through the 

National Guard Bureau, the state’s National Guard, or FEMA’s regional 

 

67State emergency management officials told us that they attended planning meetings with 

NORTHCOM and raised their concern about the scenarios, but NORTHCOM officials made 

the decision to use the scenario anyway in order to meet their own training objectives. As a 

result, these officials told us that several other states dropped out of the exercise for this 

scenario during the planning stages. 

68The Defense Science Board recently reported that national exercises typically focus on a 

top-down approach where the supporting organizations are training aids to the senior-level 

players, instead of a bottom-up approach focusing on an integrated and layered response. 

Defense Science Board, Unconventional Operational Concepts and the Homeland, xv. 

Page 40 GAO-09-849 Homeland Defense
 

offices. 69 In some cases, such as when the state emergency management 

agency and state National Guard have a close working relationship, this 

method has been effective for NORTHCOM. However, in other cases, this 

method has led to more limited exercises. For example, emergency 

management officials from one state told us that NORTHCOM does not 

have full state representation if it only exercises with the state National 

Guard. In that case, NORTHCOM therefore misses out on interaction with 

other key state emergency management officials and responders and 

affects the realism of the exercise. 

Another aspect of the lack of a consistent process for requesting state 

participation is potentially missing the opportunity to leverage the existing 

expertise of the National Guard Bureau and defense coordinating officers 

located in each of the 10 FEMA regional offices. As we previously 

reported, the National Guard Bureau and defense coordinating officers 

have knowledge and experience in dealing with states in their region and 

may be a valuable resource for NORTHCOM officials during the planning 

and conduct of exercises.70 The three defense coordinating officers with 

whom we met told us that they participate in NORTHCOM exercises, but 

currently their role does not involve requesting state participation on 

behalf of NORTHCOM or providing state-specific information to 

NORTHCOM exercise officials. Without an informed and consistent 

process for including the states in planning, conducting, and assessing its 

exercises, NORTHCOM increases the risk that its exercises will not 

provide benefits for all participants, impacting the seamless exercise of all 

levels of government and potentially affecting NORTHCOM’s ability to 

provide support to civil authorities. 

 

69In some cases, NORTHCOM’s coordination with the National Guard has been 

inconsistent. For example, National Guard officials from one state told us that it is unclear 

whether communication with NORTHCOM should occur directly or through the National 

Guard Bureau. 

70Defense Coordinating Officers are senior-level military officers who provide liaison 

support and serve as a point of contact for federal, state, and local agencies requiring DOD 

support during an incident. GAO-08-252. 

Page 41 GAO-09-849 Homeland Defense
 

NORTHCOM has a 

Systematic Lessons 

Learned and 

Corrective Action 

Program, but Gaps 

Remain 

NORTHCOM Has a 

Systematic Lessons 

Learned and Corrective 

Action Process 

DOD and NORTHCOM guidance requires that NORTHCOM identify 

observations during the course of normal operations, exercises, and realworld events; capture the detail required to fully understand the problem; 

and share valid lessons learned and issues as widely as possible.71

NORTHCOM has been identifying observations, lessons learned, and 

needed corrective actions from its exercises and operations since the 

command was created in 2002. NORTHCOM collects and tracks 

observations through the Joint Lessons Learned Information System 

(JLLIS)—the automated official DOD system for managing and tracking 

exercise observations and recording lessons learned.72 As of April 2009, 

DOD exercise participants input 94 observations into JLLIS during 

NORTHCOM’s most recent large-scale exercise, Vigilant Shield 09. Table 5 

shows the observations entered into JLLIS or its predecessor for 

NORTHCOM’s major exercises since 2006.73

 

71Chairman of the Joint Chiefs of Staff, Joint Lessons Learned Program, Instruction 

3150.25D, A-1 and A-6; U.S. Northern Command, Commander’s Training Guidance FY 09-

10, A-4, A-8, A-20, A-2-8: U.S. Northern Command, Lessons Learned Program and 

Corrective Action Board Process, 2. 

72The Joint Staff develops joint lessons learned policy and serves as the program manager 

for JLLIS. 

73JLLIS became the system of record in October 2008. Prior to that NORTHCOM’s lessons 

learned system was the Advanced Lesson Management System. 

Page 42 GAO-09-849 Homeland Defense
 

Table 5: Number of Observations Input into Lessons Learned Management System for Large-Scale Exercises Since 2006 

NORTHCOM exercise 

Unclassified

observationsa

Classified 

observations 

Total

observations

per exercise

Vigilant Shield 09 75 19 94

Ardent Sentry 08

(National Level Exercise 2-08) 46 0 46

Vigilant Shield 08

(National Level Exercise 1-08) 34 0 34

Ardent Sentry 07 113 4 117

Vigilant Shield 07 95 12 107

Ardent Sentry 06 54 2 56

Vigilant Shield 06 12 1 13

Source: Joint Lessons Learned Information System. 

Note: This includes active records on the Non-Secure Internet Protocol Router Network as of April 9, 

2009; and on the Secure Internet Protocol Router Network (classified and unclassified) as of May 4, 

2009 

a

Includes unclassified active records from the Secure Internet Protocol Router Network. 

The philosophy and approach of NORTHCOM’s Lessons Learned Program 

have been largely the same since NORTHCOM published its first 

instruction for the program in 2003,74 although the requirement to reobserve corrective actions in a subsequent exercise or operation before 

closing them was not established until 2005.75 We found that NORTHCOM 

generally has a systematic lessons learned and corrective action program, 

based on clear procedures and a regular process. Observations are 

assigned to an office of primary responsibility within NORTHCOM and 

categorized as either a lesson learned—a positive finding—-or an issue 

which requires corrective action. NORTHCOM’s intent is to manage and 

resolve issues requiring corrective action at the lowest organizational level 

possible. This responsibility is generally within NORTHCOM’s various 

directorates, component commands, or a Joint Task Force.76 Issues may 

be closed at the directorate level without external approval or oversight. 

 

74U.S. Northern Command, Lessons Learned Policy and Guidance, Operating Instruction 

36-104, (Colorado Springs, Colo.: Oct. 1, 2003). 

75Chairman of the Joint Chiefs of Staff, Instruction 3150.25B, Joint Lessons Learned 

Program, A-2 –A-3 (Washington, D.C.: February 15, 2005).

76Those component commands, such as Army Forces North or Air Force North, may also 

have a lessons learned program of their own. 

Page 43 GAO-09-849 Homeland Defense
 

Broader scope or more sensitive issues requiring the involvement of more 

than one directorate or subcommand go into the formal corrective action 

board process for review, tracking, and approval as necessary.77 This 

formal process includes two boards—the Corrective Action Board and the 

Executive Corrective Action Board—to review and resolve issues.78 Figure 

7 illustrates the flow of NORTHCOM’s lessons learned and corrective 

action process. 

 

77The Corrective Action Board process is intended to provide a means for addressing 

commandwide issues and ensuring these issues are tracked, resolved, and verified. 

78The Corrective Action Board Working Group is chaired by the head of the Joint Training 

and Exercises Directorate, and includes representatives from each NORTHCOM 

directorate and subcommand and reviews issues requiring extensive coordination across 

directorates and subordinate commands or action by an organization(s) external to 

NORTHCOM. The Executive Corrective Action Board is chaired by NORTHCOM’s Chief of 

Staff and comprised of the heads of the directorates and subcommands and reviews 

sensitive issues requiring decisions by senior leaders or actions of organizations outside of 

NORTHCOM, such as the Joint Staff. For example, lessons learned that are shared with 

DHS are reviewed by the Executive Corrective Action Board before they are forwarded to 

the Joint Staff. 

Page 44 GAO-09-849 Homeland Defense
 

Figure 7: NORTHCOM Lessons Learned and Corrective Action Process 

Observation 

recorded

Responsible

office 

assigned

Valid 

finding

Lesson 

learned or

issue?

Record 

activated

Corrective 

action 

developed 

Corrective 

action 

verified 

Issue becomes

lesson learned 

Corrective 

action 

implemented 

Verify corrective

action in a

subsequent

exercise or

operation

Responsible 

office does

not concur

Record 

closed

End

End

End

Record 

activated

No

No

Yes

Yes

Issue Lesson 

learned

Source: GAO analysis of NORTHCOM guidance.

Page 45 GAO-09-849 Homeland Defense
 

JLLIS was intended to make lessons learned more widely available; 

however, we found that non-DOD exercise participants have limited 

access to this system, which presents challenges for NORTHCOM in 

collecting observations from and sharing lessons learned with interagency 

partners and states.79 Joint Staff and NORTHCOM officials told us that 

only a small number of interagency staff and no state representatives 

currently have access to JLLIS and therefore are not able to directly input 

exercise observations or review NORTHCOM’s lessons learned thro

this system.

Gaps Remain with 

Collecting Observations 

From and Sharing Lessons 

Learned with Interagency 

Partners and States 

ugh 

e 

s 

s 

 

80 DOD officials told us that access to JLLIS is restricted in 

order to safeguard sensitive information because of concerns that 

adversaries may be able to exploit any weaknesses contained in those 

records. On the other hand, NORTHCOM officials told us that most of th

observations collected from exercises are unclassified, and we found this 

to be true for more than 90 percent of the observations from 

NORTHCOM’s six most recent exercises. The vast majority of the 

observations we reviewed are listed as unclassified and few records 

indicate they have a “For Official Use Only” restriction. For users to acces

the unclassified portion of the JLLIS they must have a DOD common 

access card, a card reader, and appropriate computer software prior to 

requesting access. Special security arrangements must be made for 

granting JLLIS access to civilian federal officials who do not posses

common access cards but have other federally issued personal 

identification verification cards. These officials cannot access JLLIS 

because they also lack the appropriate software to operate the personal 

identification verification cards within the system. In an attempt to 

address this gap and gather more input from non-DOD exercise 

participants, in April 2008 NORTHCOM officials placed a Word document 

template on its internet portal to provide a venue for non-DOD exercise 

participants to record exercise observations. Any observations provided in 

this format would be input by NORTHCOM Joint Training and Exercise 

79NORTHCOM has a process for sharing lessons learned from National Level Exercises that 

are applicable to the interagency to DHS through the Joint Staff. Thus far, NORTHCOM has 

provided three3 issues requiring corrective action from National Level Exercise 2-08 

conducted in May 2008. 

80Joint Staff officials told us that about 10Department of State officials have access to 

JLLIS, and DHS and Department of Energy officials have requested access for some of their 

staff. Other departments and agencies, such as the Federal Bureau of Investigation and 

Department of Health and Human Services, have received briefings about JLLIS, but have 

not requested access. The Joint Staff official also told us that his office does not have 

enough staff to support a large number of non-ommon access card users requesting JLLIS 

access, and granting access would be a lengthy process due to the software and security 

requirements that must be addressed. 

Page 46 GAO-09-849 Homeland Defense
 

Directorate officials into JLLIS. However, NORTHCOM’s portal

the same DOD-issued card to gain entry.

 requires 

nd 

 in the 

officials.82

ours 

s to 

ding 

in 

old us 

that they did not attend NORTHCOM’s postexercise reviews for the 

 

81 In addition, NORTHCOM’s 

lessons learned manager told us no one has submitted observations using 

the template since it was put on the portal. This may be because the 

command has not actively publicized how to access the template a

underscored the value to the command of obtaining observations from 

interagency partners and states. In response to our inquiries in May 2009, 

NORTHCOM’s lessons learned manager told us that the command is

process of adding a link to DHS’s Homeland Security Information Network

so that interagency partners and states will be able to submit lessons 

learned which can subsequently be transferred to JLLIS by NORTHCOM 

In addition to collecting observations using JLLIS, NORTHCOM can obtain

lessons learned from interagency partners and states during postexercise 

meetings. NORTHCOM conducts a review called a Hotwash within h

of completing the exercise so that exercise participants can discuss 

observations that significantly impacted their mission and recommend 

emergent themes for discussion during a subsequent review known as the 

facilitated after-action review. This review, generally held 7 days after the 

exercise is completed, provides an opportunity to present major issue

senior leaders and obtain the Commander’s guidance for resolution. 

However, the extent to which interagency and state officials are atten

and participating in NORTHCOM’s postexercise meetings is unclear. 

Based on NORTHCOM’s documentation, only two states (out of the last 

six major exercises) participated in a Facilitated After Action Review—

California in Vigilant Shield 09 and Alaska in Ardent Sentry 07. Officials 

from three states in addition to California told us that they participated 

the after-action meeting for the exercises they participated in, but they 

may have participated in the regional or national-level meeting rather than 

NORTHCOM’s.83 Officials from two of the seven states we met with t

81NORTHCOM’s lessons learned manager told us that those without cards that have a valid 

need for access to the portal must work with NORTHCOM’s Help Desk staff to register for 

an account. 

82DHS’s Web-based Homeland Security Information Network facilitates information sharing 

and collaboration between federal, state, local, tribal, private sector, and international 

partners involved in homeland security. 

83Depending on the exercise, for example if it is linked to a national level or regional 

exercise, multiple Facilitated After Action Reviews may take place. 

Page 47 GAO-09-849 Homeland Defense
 

exercises in which they participated at least partly due to staffing and 

budget limitations. 

NORTHCOM has also attempted to share lessons learned with other 

federal agencies and states by using FEMA’s lessons learned sharing 

system. For example, NORTHCOM has posted six reports onto FEMA’s 

lessons learned system, including four recent exercise reports and two 

reports from operations in 2008.84 However, with one exception, the 

documents that NORTHCOM has made available on this system (1) 

include only lists of observations and, in some cases, record-tracking 

numbers from JLLIS and previous lessons learned systems, and (2) lack 

detailed information on individual lessons learned and corrective actions.85

Joint Staff and NORTHCOM officials told us that they do not post detailed 

information on the unclassified Lessons Learned Information Sharing 

system Web site, because it is not adequately protected from the potential 

for unauthorized access to records. As a result, the security of the 

information cannot be assured. According to these officials, if an 

adversary nation or terrorist group gained access to this information, it 

may be possible for them to identify weaknesses in NORTHCOM’s 

operations that can be exploited. In a recent exercise summary report 

NORTHCOM stated that it will post lessons learned, best practices, and 

reports that may benefit their non-DOD mission partners in FEMA’s 

Lessons Learned Information Sharing system, which the report describes 

as a secure, restricted-access information system.86 Because security 

concerns are preventing NORTHCOM from openly sharing all its 

unclassified lessons learned with its interagency partners and the states, 

the information NORTHCOM does provide may be of limited value for 

helping its partners improve the nation’s disaster responsiveness. Because 

NORTHCOM is not fully involving other federal agencies and states in its 

lessons learned process, it is missing opportunities to learn lessons from 

an exercise. For example, officials from two states did not provide 

NORTHCOM with lessons learned from exercises because they did not 

 

84NORTHCOM reports posted on FEMA’s Lessons Learned Information Sharing system 

include exercise summary reports for National Level Exercise 1-08 (Vigilant Shield 08), and 

Ardent Sentry 2007; two copies of the Quick Look Report for National Level Exercise 2-08; 

NORTHCOM Support for 2008 Hurricane Season Operations Summary Report; and 

NORTHCOM Support for 2008 Political Conventions Operations Summary Report. 

85We found one document where details were provided on 12 observations. 

86U.S. Northern Command, National Level Exercise 2-08 Exercise Summary Report 

(Colorado Springs, Colo.: Aug. 15, 2008), 16. 

Page 48 GAO-09-849 Homeland Defense
 

attend the command’s post exercise reviews. As a result, NORTHCOM 

risks the reoccurrence of potential problems that were not identified in its 

process. 

NORTHCOM Faces 

Challenges Managing 

Corrective Actions 

DOD and NORTHCOM guidance requires that issues requiring corrective 

actions be tracked and remain open until the solutions are completed and 

verified as effective—through training, operations, or exercises.87 We 

found that NORTHCOM directorates and subcommands are closing some 

issues prematurely, without confirming that corrective actions were made 

or verifying in a subsequent exercise or operation that the corrective 

action is effective. We reviewed unclassified records in JLLIS from 

NORTHCOM’s previous six large-scale exercises and found at least 77 of 

the 375 records or about 20 percent required corrective actions but were 

either closed prior to completing the corrective action or closed without 

verifying the effectiveness of the corrective action.88 For example, an 

observation was made during Ardent Sentry 07 that NORTHCOM did not 

have a process for addressing a foreign nation’s offer of military-to-military 

assistance in a major disaster. The issue was validated and the corrective 

action developed, but the issue was closed by the originating organization 

before the corrective action could be verified or reobserved in a 

subsequent exercise.89 The record was closed even though the Executive 

Corrective Action Board directed that it remain open until an exercise of 

suitable scope to require significant military support was developed.90

Another example of a record being closed without verification or reobservation is an observation made during Ardent Sentry 07 raising 

concerns that NORTHCOM personnel could arrive to assess a disaster site 

 

87Chairman of the Joint Chiefs of Staff, Instruction 3150.25D, Joint Lessons Learned 

Program (Washington, D.C.: Oct. 10, 2008), A-8; U.S. Northern Command, Lessons Learned 

Program and Corrective Action Board Process, Instruction 16-166 (Colorado Springs, CO: 

Mar. 16, 2009), 14; Prior Chairman of the Joint Chiefs of Staff guidance also required that 

corrective actions be tracked and remain open until the solutions are completed and 

verified. 

88We reviewed active records from NORTHCOM’s large-scale exercises that were 

conducted between May 2006 and November 2008 for Ardent Sentry 2006, 2007, and 2008 

and Vigilant Shield 2007, 2008, and 2009. 

89The corrective action included a set of checklists and processes added to the command’s 

battle staff operating procedures. 

90The scenarios developed for NORTHCOM’s next two applicable major exercises were not 

of sufficient scope to induce another nation, other than Canada, to offer military 

assistance. 

Page 49 GAO-09-849 Homeland Defense
 

without alerting state officials they would be coming. As a result, 

NORTHCOM developed a new Command Assessment Element Concept of 

Execution91 in July 2007 to promote better command and control and 

situational awareness; however, the issue was closed before the procedure 

could be observed in a subsequent exercise or operation to verify effective 

resolution. 

These issues are likely being closed without verification or re-observation, 

because NORTHCOM Training and Exercise Directorate officials do not 

have oversight over the disposition of open issues that are resolved within 

directorates or are unable to give long-standing issues the sustained 

management attention needed to ensure resolution. NORTHCOM’s lessons 

learned manager told us that the command does not have the staff 

necessary to oversee the actions on records handled within the other 

directorates. In addition, while the checkbox format in JLLIS makes it easy 

to see whether an issue is open, awaiting verification, or closed, entries 

made in JLLIS regarding corrective actions required, implementation date, 

and plan for verification are primarily in a narrative format, which may 

make the review and oversight process more time consuming. Without 

sufficient oversight, NORTHCOM cannot ensure that corrective actions 

are verified and reobserved in a subsequent exercise or operation before 

the issue is closed, so that the command knows the solution is effective. 

We recognize that such oversight should be addressed without 

significantly stressing NORTHCOM’s staff. However, if NORTHCOM does 

not ensure that corrective actions are fully resolved, it increases the risk 

that these issues may occur again, possibly during crucial, real-world 

situations. This lack of oversight, coupled with the lack of a well-thought 

out and consistent process for including the states in assessing exercises 

as discussed earlier in this report, further limits the knowledge gained and 

value of the exercise for all participants. 

 

91The Command Assessment Element is a team supporting NORTHCOM’s commander by 

providing situational awareness from a disaster or incident site. 

Page 50 GAO-09-849 Homeland Defense
 

NORTHCOM Has 

Taken Steps to 

Integrate Its 

Exercises with the 

National Exercise 

Program, but 

Guidance Is Not 

Consistently Applied 

NORTHCOM Has 

Participated in National 

Exercises and Taken Steps 

to Integrate with the NEP 

Since the NEP Charter was approved in January 2007, NORTHCOM has 

participated in the major national exercise held under the NEP and taken 

steps to integrate its exercises into the national program.92 NEP guidance 

requires that heads of departments and agencies actively participate in tier 

I exercises and recommends participation in tier II exercises either 

through the National Exercise Simulation Center or as determined by 

agency leadership.93 Departments or agencies can participate in the NEP 

by combining an existing exercise with a NEP exercise, taking part in a 

tier II exercise sponsored by a different department or agency, or 

requesting to lead a tier II exercise to obtain greater interagency 

participation and support. DOD guidance requires that components 

participate in or lead planning efforts of NEP exercises as appropriate 

given the scenario or as tasked by the Assistant Secretary of Defense, 

Homeland Defense and America’s Security Affairs or the Chairman of the 

Joint Chiefs of Staff.94 NORTHCOM’s training guidance specifies the NEP 

exercises in which the command plans to participate during the following 

2 fiscal years.95 NORTHCOM combined two of its large-scale exercises—

Vigilant Shield 08 and Ardent Sentry 08—with major national exercises 

 

92For the purpose of this report, we are defining major national exercises as operationsbased NEP exercises involving the deployment of personnel or forces to conduct 

operations from the federal, state, and local levels. 

93DHS, National Exercise Program Implementation Plan. 

94DOD, DOD Participation in the National Exercise Program, Instruction 3020.47, 

(Washington, D.C.: Jan. 29, 2009), 10. 

95The command’s Joint Training Plan, which includes the commander’s training guidance, is 

revised and updated annually. U.S. Northern Command, Commander’s Training Guidance 

for FY 09-10. 

Page 51 GAO-09-849 Homeland Defense
 

and has taken part in two additional exercises sponsored by other 

departments (see table 6). For example, National Level Exercise 1-08, a 

tier I exercise, and NORTHCOM’s Vigilant Shield 08 were conducted 

October 15 -20, 2007, in parallel with Top Officials 496 and several other 

exercises.97 These exercises were linked together by the use of common 

scenarios and objectives intended to test existing plans, policies, and 

procedures to identify planning and resource gaps and develop corrective 

actions to improve preparedness against a weapons of mass destruction 

attack. 

Table 6: NORTHCOM Participation in National Level and Tier II Exercises Since 2007 

Exercise and tier Date Main scenario Primary sponsor/lead 

Description of 

NORTHCOM 

participation 

National Level 

Exercise 1-08 (I) 

October 2007 Radiological dispersal device 

attack 

FEMA Linked Vigilant Shield 08

to exercise 

National Level 

Exercise 2-08 (II) 

May 2008 Hurricane, terrorist chemical 

attack, accidental chemical 

release 

FEMA Linked Ardent Sentry 08

to exercise 

Diablo Bravo (II) July 2008 Nuclear weapons incident Department of Energy/ 

National Nuclear Security 

Administration 

Provided initial DOD 

response support for 

Department of Energy 

Empire 2009 (II) June 2009 Radiological dispersal device 

Attack 

National Nuclear Security 

Administration 

Provided Defense 

Coordinating Officer 

support 

Source: GAO analysis of DHS and NORTHCOM documentation. 

Note: We limited the scope of our finding to major national exercises sponsored at the federal level 

and planned after National Level Exercise 1-08, the first major exercise conducted under the NEP. 

NORTHCOM officials told us that they generally would like to participate 

in NEP exercises to achieve the benefits of exercising with interagency 

partners, but in some cases it is not beneficial to do so. For example, the 

officials told us NORTHCOM decided not to combine Ardent Sentry 09 

 

96The Top Officials exercise, better known as TOPOFF, is the nation’s terrorism 

preparedness exercise involving officials at every level of government and representatives 

from the international community and private sector. Sponsored by DHS, TOPOFF 4 

shared the same radiological dispersal device scenario in Vigilant Shield 08 and included 

more than 100 organizations. 

97National Level Exercise 1-08 included Global Lightning 08 at U.S. Strategic Command, 

Positive Response 08-1 for the Chairman, Joint Chiefs of Staff, Blue Flame in the United 

Kingdom, Pacific Challenge in Australia, a U.S. Southern Command Response Cell due to 

Able Warrior 08-1 date change, and an unnamed exercise at U.S. Pacific Command. 

Page 52 GAO-09-849 Homeland Defense
 

with National Level Exercise 09—a tier 1 exercise scheduled for July 

2009—because the objectives and scenarios for the exercises did not meet 

their training needs. Although NORTHCOM officials will conduct Ardent 

Sentry 09 separately, they are using the National Exercise Simulation 

Center—FEMA’s newly established training and exercise facility—to 

provide a test run for the center’s use in National Level Exercise 09.98

DOD and NORTHCOM have taken steps to integrate exercises with the 

National Exercise Program, including posting the command’s exercises on 

DHS’s National Exercise Schedule, successfully applying to lead a tier II 

exercise, and publishing guidance on integration with the NEP. The NEP 

Implementation Plan recommends that federal departments and agencies 

post exercises on the NEP’s National Exercise Schedule so that exercises 

and planning meetings can be synchronized across the federal 

government.99 NORTHCOM has posted its annual Ardent Sentry and 

Vigilant Shield exercises for the first 4 of 5 fiscal years on the national 

schedule, while FEMA’s National Exercise Division has posted exercises 

for the first 3 fiscal years. As of June 2009, neither the Joint Staff nor any 

other combatant commands have posted exercises on the national 

schedule. In addition, NORTHCOM recently requested and was granted 

approval to lead Vigilant Shield 10 as a tier II exercise scheduled for 

November 2009. Vigilant Shield 10 should have greater interagency 

participation than it would have received as a tier 3 exercise, since federal 

departments and agencies will be required, at a minimum, to participate in 

the National Exercise Simulation Center. As of May 2009, the participants 

of Vigilant Shield 10 include the DHS and the Departments of Justice, 

Energy, Transportation, Health and Human Services, and Veteran Affairs; 

the Joint Chiefs of Staff and U.S. Joint Forces Command; and other 

government and nongovernment organizations.100 This exercise will be the 

first time that NORTHCOM will share planning responsibilities with 

FEMA’s National Exercise Division. This exercise will also be linked to a 

 

98FEMA’s National Exercise Simulation Center opened on January 12, 2009, to provide a 

national, shared facility to train and exercise and improve coordination for all-hazards 

preparedness. The goal was to pool resources, maximize efficiency, and provide sustained 

exercise and training support to all participants. 

99FEMA officials told us that only tier 1 and 2 exercises are required to be included in the 

national schedule, but encouraged the inclusion of Tier 3 exercises. 

100The list of participating departments or agencies may change as the planning for Vigilant 

Shield 10 progresses. 

Page 53 GAO-09-849 Homeland Defense
 

Canadian government an exercise to demonstrate its readiness for the 

2010 Olympics in Vancouver. 

NORTHCOM Has Not 

Consistently Applied NEP 

Guidance 

NEP guidance includes policies and tools for the design, planning, 

conduct, and evaluation of exercises—known as the Homeland Security 

Exercise and Evaluation Program, which creates a common exercise 

policy and consistent terminology for exercise planners and serves as the 

foundation of NEP exercises.101 FEMA requires that entities, such as state 

and local governments, receiving homeland security grant funding for their 

exercises adhere to specific Homeland Security Exercise Evaluation 

Program guidance for exercise program management, design, conduct, 

evaluation, and improvement planning.102 We reviewed key program 

documents, such as the Implementation Plan, and found that this guidance 

is unclear about the extent to which federal agencies should use the 

Homeland Security Exercise Evaluation Program. For example, the 

Implementation Plan states that the NEP does not displace a preexisting 

exercise program, and none of the NEP guidance requires that federal 

agencies use the Homeland Security Exercise Evaluation Program. 

However, the Implementation Plan states that the Homeland Security 

Exercise Evaluation Program will serve as the doctrinal foundation for 

NEP exercises. 

FEMA officials told us that federal agencies should use this program when 

participating in tier I and tier II exercises so that the various exercise 

participants have consistency when planning, conducting, and assessing 

exercises. We found that NORTHCOM generally has used DOD’s Joint 

Training System guidance for planning NEP exercises, defining 

capabilities, and reporting exercise results.103 NORTHCOM officials told us 

that the Joint Training System is consistent with the NEP and served, in 

part, as the basis for the Homeland Security Exercise and Evaluation 

 

101This plan was revised in June 2008, but has not been formally approved by the current 

administration. However, FEMA officials told us that they intend to execute the NEP using 

the latest revision of the plan. DHS, DRAFT National Exercise Program Implementation 

Plan (Washington, D.C.: June 20, 2008). 

102GAO-09-369, p. 35. 

103NORTHCOM officials told us that they have used Homeland Security Exercise Evaluation 

Program guidance in one case when they drafted the exercise plan for NLE 2-08, because 

their state and local exercise partners were required to use this guidance. 

Page 54 GAO-09-849 Homeland Defense
 

Program.104 We found that these sets of guidance have similar processes 

but use different methods for defining the tasks and capabilities that are 

performed and validated in an exercise. The primary differences between 

these sets of guidance are that (1) DOD’s task list, which serves as the 

basis for its exercises, includes tasks that are specific to military missions, 

such as troop movements and sealifts; (2) DHS guidance provides more 

detailed criteria for the postexercise documentation, such as content and 

format; and (3) DHS’s planning cycle is generally shorter—9 to 15 months 

versus 12 to 18 months for DOD. (See table 7.) See app. II for a more 

detailed comparison. 

Table 7: Comparison of Areas of Application between DOD’s Joint Training System and DHS’s Homeland Security Exercise 

Evaluation Program 

Areas of application Joint Training System Homeland Security Exercise Evaluation Program 

Methods for defining tasks 

and capabilities 

• Derived from Joint Mission Essential Task 

Lists. 

• Users may also develop tasks using the 

Universal Joint Task List. 

• Derived from Target Capabilities List or 

Universal Task List. 

• Users may also develop tasks using the 

Universal Task List. 

Reporting requirements • Issue a Quick Look Report and an 

Executive Summary Report. 

• No detailed guidelines on content for either 

report. 

• No detailed guidelines on content found in 

the required Quick Look or Exercise 

Summary Report. 

• Issue a Quick Look Report and an After Action 

Report/Improvement Plan. 

• Provides detailed guidelines on content for both 

reports, including templates. 

• Provides detailed guidelines on content for both 

the required Quick Look Report and the required 

After Action Report/Improvement Plan. 

Planning cycle • Uses the Joint Event Life Cycle, has five 

stages: design, planning, preparation, 

execution, and evaluation. 

• Intended to be flexible and can be modified 

to any type of event. Can span a period of 

12 to 18 months for exercises. 

• Uses the Homeland Security Exercise Evaluation 

System’s Exercise Cycle, employs five phases: 

foundation, design and development, conduct, 

evaluation, and improvement planning. 

• Intended to be adaptable to any type of exercise. 

Can span a period of 9 to 15 months. 

Lessons learned/corrective 

actions systems 

• Joint Lessons Learned Information System 

manages and tracks Lessons Learned and 

corrective actions. 

• FEMA’s Lessons Learned Information Sharing 

System manages and disseminates lessons 

learned. 

• DHS’s Corrective Action Program system 

manages corrective actions. 

Source: GAO analysis of DOD and DHS documentation. 

 

104NORTHCOM’s training guidance for fiscal years 2008-2009 stated that the command will 

adhere to the guiding principles of the Homeland Security and Exercise Evaluation 

Program, but this reference has been removed from guidance for fiscal years 2009-2010. 

Page 55 GAO-09-849 Homeland Defense
 

According to both the Homeland Security Exercise and Evaluation 

Program and Joint Training System guidance, it is important to link tasks 

and capabilities with exercise objectives to ensure that participants 

exercise or train as they would perform in a real-world event. The 

Homeland Security Exercise and Evaluation Program recommends using 

DHS’s Target Capabilities List or the Universal Task List to formulate the 

tasks and capabilities that underlie the objectives for an exercise.105 These 

lists describe the capabilities government entities need and tasks they are 

expected to perform to prevent, protect against, respond to, and recover 

from incidents of national significance. In contrast, NORTHCOM derives 

its tasks and capabilities from the Universal Joint Task List to formulate 

Joint Mission Essential Tasks. According to NORTHCOM guidance, the 

command is required to include in its exercises the Joint Mission Essential 

Tasks associated with its Joint Training Plan, which is updated annually.106

These tasks are identified by joint force commanders as most essential to 

their assigned or anticipated missions with priority given to their wartime 

missions and describe their priority wartime requirements. We found that 

DOD’s operating instruction for participation in the NEP does not provide 

guidance on how DOD components should incorporate tasks and 

capabilities derived from sources recommended by the Homeland Security 

Exercise and Evaluation Program when participating in NEP exercises.107

The primary differences between DHS’ and DOD’s lists are that DOD’s task 

lists generally incorporate more descriptive metrics and criteria to assess 

performance and include tasks that are specific to military missions, such 

as troop movements and sealifts. In some cases, state National Guards 

officials have had to translate DOD task lists into DHS tasks lists when 

working with their civilian partners and vice versa. 

We also found that neither DOD’s nor NORTHCOM’s guidance for 

developing postexercise reports includes the same degree of specificity 

recommended in the Homeland Security Exercise Evaluation Program. 

For example, both sets of exercise guidance require postexercise reports; 

however, the Homeland Security Exercise Evaluation Program provides 

templates and guidance for these documents, including requiring an 

improvement plan to clearly outline the corrective actions needed, which 

 

105DHS, Homeland Security Exercise and Evaluation Program, Volume 1: HSEEP 

Overview and Exercise Program Management (Washington, D.C.: February 2007). 

106U.S. Northern Command, Operations: Exercise Program, pp. 4-5.

107DOD, DOD Participation in the National Exercise Program, Instruction 3020.47, 7-8. 

Page 56 GAO-09-849 Homeland Defense
 

are not included in DOD’s or NORTHCOM’s guidance. In addition, 

NORTHCOM’s exercise summary reports for National Level Exercise 1-08 

and 2-08 did not contain all information recommended by the Homeland 

Security Exercise Evaluation Program. For example, NORTHCOM did not 

include the recommended analyses regarding the capabilities and tasks 

tied to the exercises’ objectives. 

As stated above, we reviewed NEP guidance such as the Implementation 

Plan and found it does not clearly state the extent to which federal 

agencies are required to follow the Homeland Security Exercise and 

Evaluation Program. As a result of this unclear guidance, we found that 

agency officials have varying interpretations of the requirements. For 

example, a DOD and a Joint Staff official told us that NEP guidance does 

not require agencies to use the Homeland Security Exercise Evaluation 

Program even for NEP exercises. Therefore, NORTHCOM uses the Joint 

Training System rather than the Homeland Security Exercise and 

Evaluation Program as the basis for planning, conducting, and assessing 

exercises. However, officials from FEMA’s National Exercise Division told 

us that all participating agencies should use the Homeland Security 

Exercise and Evaluation Program guidance for tier I and tier II NEP 

exercises.108 FEMA officials stated that federal departments and agencies 

should be held accountable for meeting key requirements, but that FEMA’s 

authority is limited to guiding, supporting, and coordinating with, but not 

directing other federal departments and agencies to comply with guidance. 

As we have previously reported, we believe that FEMA’s expanded 

leadership role under the Post-Katrina Act provides FEMA opportunities 

to instill a shared sense of responsibility and accountability on the part of 

all agencies.109

Neither DOD nor NORTHCOM guidance specifically addresses the extent 

to which DHS’s Homeland Security Exercise and Evaluation Program 

planning and documentation requirements should be followed.110 We 

recognize that NORTHCOM and DOD must meet their own mission and 

 

108According to FEMA officials, since tier I and II exercises are interagency exercises, the 

Homeland Security Exercise and Evaluation Program should be used because it was 

designed for the interagency community. 

109GAO-09-369. 

110NORTHCOM guidance states the Vigilant Guard Program should use Homeland Security 

Exercise and Evaluation Program formats to encourage civilian agency participation. U.S. 

Northern Command, Commander’s Training Guidance: Fiscal Year 2009-2010, A-18. 

Page 57 GAO-09-849 Homeland Defense
 

exercise requirements and the Joint Training System may be best suited 

for NORTHCOM’s exercises; however, all of the states we visited use 

Homeland Security Exercise and Evaluation Program guidance. We found 

that having differing sets of guidance, such as DOD’s and DHS’ capabilities 

task lists and postexercise documentation requirements, makes exercises 

more difficult and potentially limits the benefits for participating states. 

For example, officials from three states we visited told us that using 

NORTHCOM’s exercise planning and reporting requirements rather than 

Homeland Security Exercise Evaluation Program guidance has made the 

processes more difficult. Further, the Defense Science Board found that 

inconsistent approaches to the development and content of postexercise 

documentation may affect the ability of organizations to fully learn lessons 

identified in exercises.111 We also reported that when other federal entities 

carry out processes that do not specifically follow the Homeland Security 

Exercise and Evaluation Program, FEMA managers do not have the 

necessary data to measure progress, identify gaps in preparedness, and 

track lessons learned—key objectives of the NEP.112 We believe that 

achieving national preparedness requires a whole-of-government approach 

and is a shared responsibility among federal, state, local, and tribal 

governments and organizations and an integration of their various 

standards, policies, and procedures into the national system.113

There is an increasing realization within the federal government that an 

effective, seamless national response to an incident requires a strong 

partnership among federal, state, and local governments and 

organizations, including integrated planning, training, and the exercise of 

those plans. For DOD, the effective execution of civil support, especially 

amid simultaneous, multijurisdictional disasters, requires ever-closer 

working relationships with other departments and agencies and at all 

levels of government. NORTHCOM’s use of DOD’s Joint Training System 

has provided a robust process for planning and conducting exercises to 

improve preparedness to achieve its homeland defense and civil support 

missions, and its efforts to involve its interagency partners and the states 

in exercises have helped to reduce uncertainty about the process for 

responding to an incident. However, without a consistent record of what 

Conclusions 

 

111Defense Science Board, Unconventional Operational Concepts and the Homeland. 

112GAO-09-369, p. 42. 

113GAO-09-369, p. 68. 

Page 58 GAO-09-849 Homeland Defense
 

has occurred during an exercise that is accessible by all exercise 

participants, including those from other federal agencies and states, 

NORTHCOM cannot ensure that it has met internal standards, trained to 

key focus areas, or compared the goals and results of exercises over time. 

Further, a key element to developing effective working relationships with 

all states is a consistent process for including states in planning and 

executing NORTHCOM’s exercises that incorporates state-specific 

knowledge and information. By coordinating consistently with 

organizations, like FEMA and NGB, that have knowledge and experience 

dealing with states, NORTHCOM can improve the value and effectiveness 

of exercises for all of the participants involved. 

Exercises provide an opportunity to enhance preparedness by collecting, 

developing, implementing, and disseminating lessons learned and verifying 

corrective action taken to resolve previously identified issues. 

NORTHCOM’s clear procedure for capturing observations in JLLIS and 

identifying issues needing corrective action has helped to improve its 

capabilities to complete its missions. However, by not providing federal 

agencies and states greater access to its lessons learned process, 

NORTHCOM will lose opportunities to learn valuable lessons from an 

exercise, particularly observations from the states that could enhance 

coordination and build more effective interagency relationships. Further, 

the risk that issues may reoccur will be increased, particularly when 

interagency partners are not aware of key issues or concerns that might 

impede the government’s overall responsiveness to a natural or man-made 

disaster. In addition, when corrective actions remain open until fully 

implemented and verified in a subsequent exercise, NORTHCOM will have 

greater assurance that issues raised during exercises are being adequately 

addressed and the corrections are in fact solving the problems identified. 

NEP policies and tools for the design, planning, conduct, and evaluation of 

exercises are intended to create a common exercise policy and consistent 

terminology for exercise planners across all levels of government to 

improve the federal government’s ability to evaluate national 

preparedness. The steps DOD and NORTHCOM have taken to integrate 

exercises with the NEP have helped DHS to prioritize and coordinate 

federal exercise activities and enhance the federal government’s ability to 

respond to an incident. We recognize that NORTHCOM and DOD must 

meet their own mission and exercise requirements and the Joint Training 

System may be best suited to meet the high standards required for 

NORTHCOM’s exercises. However, achieving national preparedness 

requires shared responsibility among federal, state, and local governments 

and organizations and an integration of their various standards, policies, 

Page 59 GAO-09-849 Homeland Defense
 

and procedures into the national system. We also recognize that the NEP 

continues to evolve and become more useful to federal and state partners. 

IHowever, in the absence of clear guidance from DHS on the extent to 

which agencies should use Homeland Security Exercise Evaluation 

Program planning and documentation guidance, DOD should ensure that 

its components clearly understand when the use of this guidance is 

appropriate so that both DOD and its exercise partners, such as other 

federal agencies and states, derive the most benefits from exercises. This, 

in turn, contributes to the ultimate success of a whole-of-government 

approach to national preparedness. 

To improve NORTHCOM’s consistency with exercise documentation, we 

recommend that the Secretary of Defense direct NORTHCOM’s 

Commander to develop guidance with specific criteria for postexercise 

documentation, particularly the Exercise Summary Report as the official 

exercise record, including the content and format to be included in such 

reports that will allow the results and lessons learned of exercises to be 

easily reviewed and compared. 

Recommendations for 

Executive Action 

To improve NORTHCOM’s involvement of interagency partners and states 

in its exercises, we recommend that 

• the Secretary of Defense, in consultation with the Chairman of the 

Joint Chiefs of Staff, the Commander, U.S. Northern Command, and 

other relevant combatant commanders, coordinate with the 

Department of Homeland Security and Federal Emergency 

Management Agency to develop guidance and procedures for 

consistently involving state officials in planning, executing, and 

assessing exercises that incorporate relevant state-specific 

information, and 

• the Secretary of Defense direct NORTHCOM’s Commander to develop 

a training plan for NORTHCOM headquarters staff on state emergency 

management structures and relevant issues related to working with 

civilian state and local emergency management officials. 

To improve NORTHCOM’s involvement of interagency partners and states 

in its lessons learned and corrective action process and its management of 

corrective actions, we recommend that the Secretary of Defense direct: 

• NORTHCOM’s Commander to establish and publicize valid and easily 

accessible procedures for non-DOD exercise participants to submit 

observations relevant to NORTHCOM, such as placing a template on 

Page 60 GAO-09-849 Homeland Defense
 

NORTHCOM’s publicly accessible Web site or DHS’s Homeland 

Security Information Network, so that NORTHCOM officials have a 

clear, secure avenue to obtain observations and assess potential 

lessons that originate with its exercise partners; 

• the Chairman of the Joint Chiefs of Staff, in consultation and 

coordination with DHS, to either resolve information assurance issues 

so that the combatant commands, including NORTHCOM, can post 

Exercise Summary Reports with lessons learned and observations 

from NEP exercises on DHS’s Lessons Learned Information Sharing 

system to make them easily accessible to interagency partners and 

states or establish an alternative method to systematically collect and 

share lessons learned; and 

• the Chairman of the Joint Chiefs of Staff to revise the joint lessons 

learned operating instruction to include procedures to ensure that 

appropriate corrective actions are implemented and verified in a 

subsequent exercise or operation before being closed and that the 

reasons for closure are documented. Possible procedures might be 

adding a verification checkbox on JLLIS’s issue management page or 

requiring that the directorates and subordinate commands within the 

combatant commands provide a status report when a correction is 

implemented and reobserved or closed for reasons other than reobservation. 

To improve NORTHCOM’s ability to work with interagency partners on 

major national exercises and further achieve the objectives of the NEP, we 

recommend that the Secretary of Defense revise the instruction on DOD 

participation in the NEP and/or direct the Chairman of the Joint Chiefs of 

Staff to revise the operating instruction regarding DOD participation in the 

NEP to provide the general conditions under which the combatant 

commands are expected to follow the Homeland Security Exercise and 

Evaluation Program planning and documentation requirements or the 

DOD’s Joint Training System should be modified for those civil support 

exercises. 

In comments on a draft of this report, DOD generally agreed with the 

intent of our recommendations and discussed steps it is taking or plans to 

take to address these recommendations. DOD also provided technical 

comments, which we have incorporated into the report where appropriate. 

DHS also reviewed a draft of this report and provided technical comments, 

which we have incorporated where appropriate. 

Agency Comments 

and Our Evaluation 

Page 61 GAO-09-849 Homeland Defense
 

In response to our recommendation that NORTHCOM develop guidance 

with specific criteria for postexercise documentation to allow the results 

and lessons learned of exercises to be reviewed and compared, DOD 

agreed that such information should be provided in a standardized format 

that can be easily accessed and understood by authorized organizations 

which might benefit from such knowledge. DOD cautioned that any 

actions in response to this recommendation must accommodate 

constraints regarding classified information. We agree that properly 

securing classified information is a critical responsibility and believe this 

can easily be accomplished without undermining the intent of the 

recommendation, which is to improve the consistency and completeness 

of formal exercise documentation and thereby its overall value. 

In response to our recommendation that DOD coordinate with DHS and 

FEMA to develop guidance and procedures for consistently involving state 

officials in planning, executing, and assessing exercises that incorporate 

relevant state-specific information, DOD agreed that better coordination 

for interfacing with state officials can be achieved. DOD also pointed out 

that NORTHCOM continues to expand its efforts to work through defense 

coordinating officers, existing state National Guard relationships, and 

FEMA regional headquarters partners to ensure that states are able to 

benefit from participation in DOD-sponsored exercises. However, DOD 

also said that while NORTHCOM has continuously engaged and 

encouraged state participation in NORTHCOM-sponsored exercises, the 

primary audience for such training is and must remain NORTHCOM. DOD 

also suggested that our recommendation has applicability to other federal 

interagency partners and that the issue should be addressed to the 

Exercise and Evaluation Sub-Interagency Planning Committee as a 

revision to the National Exercise Program Implementation Plan. As our 

report indicates, we agree that NORTHCOM has sought to engage and 

involve the states in its comprehensive exercise program. NORTHCOM 

plans for and conducts major exercises both inside and outside the 

construct of the National Exercise Program. Particularly for NORTHCOMsponsored exercises focused on the command’s civil support mission, the 

effective involvement of and interaction with state and other federal 

partners is a critical component of improving and maintaining 

NORTHCOM’s preparedness. For NORTHCOM’s participation in nationallevel exercises, the preparedness goals and objectives of all participants 

are equally important. We believe that in developing procedures to 

improve coordination with the states, DOD can (1) avoid situations where 

exercises meant to improve preparedness are not fully coordinated with 

the necessary partners; (2) capitalize on the structures and organizations it 

already has in place, such as the defense coordinating officers and 

Page 62 GAO-09-849 Homeland Defense
 

relationships with state National Guard headquarters; and (3) coordinate 

with DHS and FEMA to improve the military-civilian interface. With regard 

to the latter, the Exercise and Evaluation Sub-Interagency Planning 

Committee may indeed be one of the venues at which DOD can effectively 

coordinate with its interagency partners. 

With respect to our recommendation that NORTHCOM develop a training 

plan for NORTHCOM headquarters staff on state emergency management 

structures and relevant issues related to working with civilian state and 

local emergency management officials, DOD agreed and noted that 

headquarters training is required for all newly assigned NORTHCOM staff. 

Further, DOD noted that NORTHCOM sponsors three versions of its 

defense support of civil authorities seminar that are targeted to staff at 

different seniority levels. We agree that NORTHCOM has continued to 

improve the level of awareness and training it provides staff on the 

complexities of providing defense support to civilian authorities in the 

United States. However, this does not fully address our recommendation. 

While training on the general procedures of the national response 

framework, the nature of state-federal government relations, and DOD’s 

proper role in providing military support to civil authorities is invaluable 

for NORTHCOM staff, we continue to believe that this should be 

supplemented by the kinds of state-specific information that would 

provide both exercise officials and all other staff with an understanding of 

the key differences between states. These differences are possibly as 

numerous as the number of states and play a role in all routine 

interactions between the individual states and DOD officials as well as for 

effective coordination for exercise planning and coordination during a 

natural disaster or some other no-notice incident requiring defense 

support to civil authorities. 

DOD agreed with our recommendation that NORTHCOM establish and 

publicize valid and easily accessible procedures for non-DOD exercise 

participants to submit observations relevant to NORTHCOM, such as 

placing a template on NORTHCOM’s Web site or DHS’ Homeland Security 

Information Network, so that NORTHCOM officials have a clear, secure 

avenue to obtain observations and assess potential lessons that originate 

with its exercise partners. DOD indicated that collecting exercise 

information from all perspectives would provide additional opportunities 

to improve NORTHCOM’s ability to accomplish its mission tasks. 

DOD also agreed with our recommendation that it work with DHS to 

either resolve information assurance issues so that NORTHCOM can post 

Exercise Summary Reports with lessons learned on DHS’ Lessons Learned 

Page 63 GAO-09-849 Homeland Defense
 

Information Sharing system or establish an alternative method to 

systematically collect and share lessons learned. DOD cautioned that 

while wide dissemination of information approved for release would be of 

great benefit to homeland security entities it continues to adhere to the 

Joint Training System and cannot mandate that DHS alter its Lessons 

Learned Information System to make accommodations. DOD also noted 

that it has procedures in place to allow specifically cleared individuals 

from outside DOD access to information contained in Exercise Summary 

Reports. We agree that DOD cannot mandate alterations to the Lessons 

Learned Information System. We also agree that the Joint Training System 

should remain the chief guidance for the conduct of DOD exercises. 

However, we continue to believe that in working with DHS on the proper 

level and mode of information sharing, DOD may be able to improve the 

dissemination of relevant exercise-related information to all appropriate 

officials. 

DOD agreed with our recommendation that the Chairman of the Joint 

Chiefs of Staff revise the joint lessons learned operating instruction to 

include procedures to ensure that appropriate corrective actions are 

implemented and verified in a subsequent exercise or operation before 

being closed and that the reasons for closure are documented. DOD 

indicated that the Chairman of the Joint Chiefs of Staff Instruction 

3150.25D could be expanded to provide more guidance and the Joint 

Lessons Learned Information System could be updated to provide a 

technological solution to address the issue once the process and 

procedures are in place. DOD also indicated that the process of verifying 

corrective action and closing issues will become more effective with the 

modifications it outlined in response to the recommendation. 

In response to our recommendation that DOD revise guidance on DOD 

participation in the National Exercise Program to provide the general 

conditions under which the combatant commands are expected to follow 

the Homeland Security Exercise and Evaluation Program planning and 

documentation requirements or the DOD’s Joint Training System should 

be modified for those civil support exercises, DOD recognized the 

importance of ensuring effective interaction with interagency partners for 

homeland security-related exercises. However, DOD noted that the 

National Exercise Program Implementation Plan contains language placed 

there at DOD’s insistence that establishes a process to resolve doctrinal 

differences during exercise planning. DOD indicated that together with 

provisions in the implementation plan establishing the administration, 

scope, and hierarchy of multiagency homeland security exercises and the 

5-year National Exercise Program schedule, this should address our 

Page 64 GAO-09-849 Homeland Defense
 

recommendation. DOD further noted that The Joint Training System 

remains the Secretary of Defense’s guidance on DOD exercises and that 

the National Exercise Program Implementation Plan stipulates that 

individual department or agency exercise programs should not be 

replaced. We agree that the Joint Training System is and should be DOD’s 

primary guidance for ensuring that DOD components train and exercise 

according to standards. However, because interagency exercises are 

becoming an ever larger part of the national preparedness effort, and to 

the extent that effective exercise planning is bolstered by common 

procedures, our recommendation is intended to help DOD clarify for its 

components the circumstances under which the specific planning and 

documentation requirements for the Homeland Security Exercise and 

Evaluation Program can be followed without detriment to DOD’s high 

training and exercise standards or compromise of the Joint Training 

System. 

DOD’s written comments are reprinted in appendix III. 

We are sending copies of this report to the Secretary of Defense, Secretary 

of Homeland Security, and other interested parties. In addition, the report 

will be available at no charge on GAO’s Web site at http://www.gao.gov. 

If you or your staff have any questions about this report, please contact me 

at (202) 512-5431 or dagostinod@gao.gov. Contacts points for our Office of 

Congressional Relations and Public Affairs may be found on the last page 

Davi M. D’Agostino 

of this report. Key contributors to this report are listed in appendix IV. 

Director, Defense Capabilities 

 and Management 

Page 65 GAO-09-849 Homeland Defense
 

List of Requesters 

The Honorable Joseph I. Lieberman 

Chairman 

The Honorable Susan M. Collins 

Ranking Member 

Committee on Homeland Security and Governmental Affairs 

United States Senate 

The Honorable Bennie G. Thompson 

Chairman 

Committee on Homeland Security 

House of Representatives 

The Honorable Kit Bond 

United States Senate 

The Honorable Patrick Leahy 

United States Senate 

The Honorable Gene Taylor 

House of Representatives 

Page 66 GAO-09-849 Homeland Defense
Appendix I: Scope and Methodology Appendix I: Scope and Methodology 

In conducting this review, we generally focused our scope on U.S. 

Northern Command’s (NORTHCOM) large-scale exercises1

 conducted 

since Hurricane Katrina made landfall in August 2005. To determine the 

extent to which NORTHCOM’s exercise program is consistent with 

Department of Defense (DOD) training and exercise requirements and 

includes relevant exercise partners, we evaluated NORTHCOM’s 

compliance with exercise reporting and documentation requirements 

established in DOD and command guidance. We reviewed available 

guidance to determine requirements for timing, format, and content. We 

also compared these requirements with guidance contained in the 

Department of Homeland Security’s (DHS) Homeland Security Exercise 

and Evaluation Program documentation. We reviewed exercise 

documentation for all large-scale exercises the command performed since 

it was established in 2002 to determine the extent to which the command 

complied with the guidance. 

We also performed an assessment of the experiences and level of 

participation from some interagency organizations and states in 

NORTHCOM’s large-scale exercises. We initially met with Nevada officials 

who participated in a NORTHCOM exercise prior to Hurricane Katrina—

Determined Promise 03—to provide context to the extent that changes 

may have been made to NORTHCOM’s exercise program and help develop 

our state selection methodology. We selected a nongeneralizable sample of 

six states based on the extent to which they have participated in major 

NORTHCOM exercise since Hurricane Katrina and the varying scenarios 

of the exercises. The states we selected played a major role in 

NORTHCOM exercises by having a portion of the exercise conducted in 

their state and having various state agencies and officials participate. 

States we selected include Arizona, California, Michigan, Oregon, Rhode 

Island, and Washington. We met with representatives from each state’s 

emergency management organization and state national guard. Because of 

the methodology selected, the resulting data and information from these 

state visits could not be projected to make assumptions about the rest of 

the states and what they may experience exercising with NORTHCOM. We 

also met with officials from three Federal Emergency Management Agency 

(FEMA) regional offices that had exercised with NORTHCOM in three of 

the last six large-scale exercises. We also interviewed officials from the 

 

1

For purposes of this report, large-scale exercises are those involving multiple agencies and 

organizations, venues, and events, such as NORTHCOM’s Vigilant Shield and Ardent Sentry 

exercises. NORTHCOM generally conducts two large-scale exercises annually. 

Page 67 GAO-09-849 Homeland Defense 
Appendix I: Scope and Methodology 

Office of the Assistant Secretary of Defense for Homeland Security and 

America’s Security Affairs, Joint Staff, and NORTHCOM with knowledge 

of and experience with NORTHCOM’s training and exercise program. 

To determine the extent to which NORTHCOM is using lessons learned 

during exercises to improve mission preparedness, we reviewed DOD, 

NORTHCOM, and DHS National Exercise Program (NEP) guidance for 

recording, tracking, and managing lessons learned and assessed 

NORTHCOM’s management of exercise observations and issues identified 

in several of NORTHCOM’s large-scale exercises since Hurricane Katrina 

in 2005. We interviewed NORTHCOM, Joint Staff, and FEMA officials 

regarding the various lessons learned management systems, and how 

interagency and state access to these systems can be accomplished. We 

also spoke with an official in the General Services Administration 

regarding the types of federal personal identification verification cards 

used by DOD and other federal departments and agencies to access 

government computer systems. In reviewing the management of 

NORTHCOM’s lessons learned program we identified and reviewed all 

unclassified exercise observations from its last six large-scale exercises 

that had been activated in NORTHCOM’s area of the Joint Lessons 

Learned Information System (JLLIS). Our review of the records in JLLIS 

entailed determining each record’s status (open or closed), its type (issue 

or lesson learned), and each record’s disposition after NORTHCOM staff 

have acted on these records to respond to the issues or lessons learned 

documented. Based on our review, we generally placed these records into 

one of several categories: open; closed, nonconcur; issue closed with 

reobservation; issue closed with no reobservation; and lesson learned. In 

addition we reviewed several records that had been merged with other 

original records because each related to the same issue; however, the 

original record for that issue was not part of our universe. Therefore, 

without reviewing the lead record the merged records lacked sufficient 

information regarding their disposition and that condition became another 

category. 

Finally, to determine the extent to which NORTHCOM is integrating its 

training and exercises with the NEP we reviewed DOD, NORTHCOM, and 

Department of Homeland Security guidance to identify any differences in 

exercise planning and documentation between DOD’s guidance and that 

for the NEP. We used that analysis to determine under what conditions 

NORTHCOM should apply standards related to the NEP, and how DOD 

and its subordinate commands should participate in the NEP tier I or II 

exercises. We reviewed NORTHCOM documentation from two major 

national exercises conducted during fiscal year 2008 to determine the 

Page 68 GAO-09-849 Homeland Defense
Appendix I: Scope and Methodology 

extent to which NORTHCOM employed the guidance from the Homeland 

Security Exercise Evaluation Program. We determined that national 

exercises that are operations-based exercises in that they involved the 

deployment of personnel would be the best candidates for evaluating 

NORTHCOM’s participation in such exercises. We also interviewed state 

emergency management and National Guard officials from six states that 

have exercised with NORTHCOM since 2005, to understand the extent to 

which NORTHCOM is integrating its exercise planning and conduct with 

the interagency as well as various state governments. 

In addressing our objectives, we reviewed plans and related documents, 

obtained information, and interviewed officials at the following locations: 

• NORTHCOM Headquarters, Peterson Air Force Base, Colorado 

Springs, Colorado 

• Joint Forces Command, Joint Warfighting Center, Suffolk, Virginia 

• The Office of the Secretary of Defense, Washington, D.C. 

• The Joint Staff, Washington, D.C. 

• U.S. Army North, Fort Sam Houston, San Antonio, Texas 

• National Guard Bureau, Arlington, Virginia 

• Department of Homeland Security, Washington, D.C. 

• U.S. Coast Guard, Atlantic Area, Portsmouth, VA 

• FEMA’s National Preparedness Directorate, Washington, D.C. 

• FEMA Region 1, Boston, Massachusetts 

• FEMA Region 9, Oakland, California 

• FEMA Region 10, Bothell, Washington 

• General Services Administration, Washington, D.C. 

• Arizona Division of Emergency Management, Phoenix, Arizona 

• Arizona National Guard, Joint Force Headquarters, Phoenix, Arizona 

• California Emergency Management Agency, Sacramento, California 

• California National Guard, Joint Force Headquarters, Sacramento, 

California 

• Michigan State Police, Emergency Management and Homeland 

Security Division, Lansing, Michigan 

• Michigan National Guard, Lansing, Michigan 

• Nevada State Division of Emergency Management, Carson City, 

Nevada 

• Nevada National Guard, Joint Force Headquarters, Carson City, 

Nevada 

• Rhode Island Emergency Management Agency, Cranston, Rhode Island 

• Rhode Island National Guard, Joint Force Headquarters, Cranston, 

Rhode Island 

• Oregon Military Department, Office of Emergency Management, 

Salem, Oregon 

Page 69 GAO-09-849 Homeland Defense
Appendix I: Scope and Methodology 

• Oregon Military Department, National Guard Joint Force 

Headquarters, Salem, Oregon 

• Washington Military Department, Emergency Management Division, 

Camp Murray, Washington 

• Washington National Guard, Joint Force Headquarters, Camp Murray, 

Washington 

We conducted our review from June 2008 to September 2009 in 

accordance with generally accepted government auditing standards. Those 

standards require that we plan and perform the audit to obtain sufficient, 

appropriate evidence to provide a reasonable basis for our findings and 

conclusions based on our audit objectives. We believe that the evidence 

obtained provides a reasonable basis for our findings and conclusions 

based on our audit objectives. 

Page 70 GAO-09-849 Homeland Defense
Appendix II: Comparison of DOD’s Joint 

Training System with DHS’s Homeland 

Security Exercise and Evaluation Program 

We reviewed the time lines and milestones for developing exercises found 

in the Department of Defense’s Joint Training System and U.S. Northern 

Command’s (NORTHCOM) implementing guidance and compared them 

with the Department of Homeland Security’s Homeland Security Exercise 

and Evaluation Program guidance to determine the similarities and 

differences between them. We used the guidance associated with 

operation-based exercises rather than discussion-based exercises to 

present the full spectrum of Homeland Security Exercise and Evaluation 

Program processes and planning events. 

Table 8: Comparison of DOD Joint Training System and Homeland Security Exercise and Evaluation Program 

DOD Joint Training System and NORTHCOM Exercise 

Program Homeland Security Exercise and Evaluation Program Guidance 

Joint Event Life Cycle 

NORTHCOM uses the Joint Events Life Cycle to design, plan, 

and execute exercises. 

• Consists of 5 stages: design, planning, preparation, 

execution, and evaluation. 

• Is a flexible guide that can be modified to apply to various 

types of events 

• Can span a period of 12 to 18 months for exercise 

development. 

Homeland Security Exercise and Evaluation Program Exercise Cycle 

Homeland Security Exercise and Evaluation Program policy and 

guidance is the doctrine for the development of National Exercise 

Program exercises. 

• Employs 5 phases, which are collectively known as the exercise 

cycle: foundation, design and development, conduct, evaluation, 

and improvement planning. 

• Planning time line for the national level exercise will begin 

approximately 15 months prior to exercise conduct. 

Concept Working Group 

Time frame: 12 to 15 months prior to start of exercise. 

Purpose: 

• Commences start of design stage. 

• Conduct initial planning to develop the candidate 

scenario setting, determine funding constraints, and 

define training requirements. 

• Result in the validation of the exercise purpose, exercise 

objectives, identification of the training audience, 

intelligence concepts, draft exercise event flow, planning 

milestone, and draft initial planning conference agenda. 

Concept and Objective Meeting 

Time frame: Minimum 6 months prior to start of exercise, lasts 2 to 4 

hours. 

Purpose: 

• Marks the start of the planning process and can be conducted in 

conjunction with the initial planning conference. 

• Identify the type, scope, objectives, and purpose of the exercise, 

as well as help planners identify, among other things, exercise 

planning team members. 

• Reach agreement regarding exercise type, scenario, capabilities, 

tasks, objectives, the target exercise time frame, and the date and 

time of the next planning conference. 

Appendix II: Comparison of DOD’s Joint 

Training System with DHS’s Homeland 

Security Exercise and Evaluation Program 

Page 71 GAO-09-849 Homeland Defense 
Appendix II: Comparison of DOD’s Joint 

Training System with DHS’s Homeland 

Security Exercise and Evaluation Program 

DOD Joint Training System and NORTHCOM Exercise 

Program Homeland Security Exercise and Evaluation Program Guidance 

Initial Planning Conference 

Time frame: 10 to12 months prior to start of exercise, lasts 3

to 4 days. 

Purpose: 

• First major planning conference in the Joint Events Life 

Cycle process. 

• Identify levels of participation, develop deployment and 

logistic requirements, validate the exercise milestones, 

develop outline of the exercise plan and other planning 

documents, determine modeling and simulation 

requirements, and develop the mid-planning conference 

agenda. 

Initial Planning Conference 

Time frame: Minimum 6 months prior to start of exercise, lasts 3 to 6 

hours. 

Purpose: 

• Begins planning process. 

• Determine exercise scope by gathering input from the exercise 

planning team, design requirements and conditions (such as 

assumptions and artificialities), objectives, extent of play, and 

scenario variables (such as time, location, hazard selection). 

• Used to develop exercise documentation by obtaining the 

planning team’s input on exercise location, schedule, duration, 

and other relevant details. 

Mid-Planning Conference 

Time frame: 7-8 months prior to start of exercise, lasts 

approximately 3 days. 

Purpose: 

• Develop a draft exercise plan, scenario, modeling and 

simulation architecture, after-action review concept of 

operations, detailed augmentation requirements. 

• Update milestones and identify funding issues. In 

addition, workgroups should have master scenario 

events lists identified with an understanding of who is 

responsible for completing one. 

Mid-Term Planning Conference 

Time frame: Minimum 3 months prior to start of exercise, lasts 

approximately 1 day. 

Purpose: 

• Discuss and determine exercise organization and staffing 

concepts, scenario and time line development, scheduling, 

logistics, and administrative requirements. 

• Review draft documentation (such as scenario, exercise plan, 

controller and evaluator handbook, master scenario events list). 

• Conduct a walkthrough of the proposed exercise site at the end of 

the conference. 

The Master Scenario Events List Development Conference 

Time frame: Conducted before, after, or in conjunction with 

the mid-planning conference. 

Purpose: 

• Forum for the initial development of detailed scenario 

events and the master scenario events list—a detailed 

narrative description of the exercise scenario in 

chronological sequence. 

The Master Scenario Events List Conference 

Time frame: Conducted separately or combined with mid-planning 

conference or final planning conference; length varies from 4 to 8

hours. 

Purpose: 

• Focuses on developing the master scenario events list—a 

chronological list that supplements the exercise scenario with 

event synopses; expected participant responses; capabilities, 

tasks, and objectives to be addressed; and responsible 

personnel. 

Page 72 GAO-09-849 Homeland Defense
Appendix II: Comparison of DOD’s Joint 

Training System with DHS’s Homeland 

Security Exercise and Evaluation Program 

DOD Joint Training System and NORTHCOM Exercise 

Program Homeland Security Exercise and Evaluation Program Guidance 

Final Planning Conference 

Time frame: 3 months prior to start of exercise. 

Purpose: 

• Provide a last check of the status of the planning and 

preparation for exercise and to finalize, publish, and 

distribute key exercise plans and documents (such as 

exercise plan, supporting plans, scenario, after-action 

review collection management plan, exercise manning 

status, control plan, exercise battle rhythm, and modeling 

and simulation status). 

• Finalize the construct of the exercise with conference 

representatives. 

Final Planning Conference 

Time frame: Minimum 6 weeks prior to start of exercise, lasts 

approximately 1 day. 

Purpose: 

• Final forum for reviewing exercise processes and procedures. 

• No major changes to the design or scope of the exercise, or its 

supporting documentation, should take place. 

• Ensures that all logistical requirements have been met, all 

outstanding issues have been identified and resolved, and all 

exercise products are ready for printing. 

Master Scenario Events List Synchronization Conference 

Time frame: Held shortly after the final planning conference 

and at least 3 weeks prior to start of exercise. 

Purpose: All events and implementers are reviewed for 

timing, realism, and completeness. 

See master scenario events list conference above. 

Tabletop Exercises and Senior Leadership Seminars 

Time frame: Prior to start of exercise. 

Purpose: 

Tabletop exercises are discussion-based exercises involving 

key personnel discussing hypothetical scenarios. 

• Facilitating understanding of concepts and processes, 

and identifying strengths and shortfalls. 

• Provide an opportunity to address new or side issues, or 

work out processes prior to the start of the exercise. 

Senior leadership seminars provide senior leaders (flag 

officer, general officer, and senior executive service level) the 

opportunity to discuss lessons learned from previous 

exercises and the upcoming exercise. 

Tabletop Exercises and Seminars 

Time frame: Not specified. 

Purpose: 

Tabletop exercises involve key personnel discussing hypothetical 

scenarios in an informal setting. 

• Used to assess plans, policies, and procedures or to assess the 

systems needed to guide the prevention of, response to, and 

recovery from a defined incident. 

Seminars are informal discussions, unconstrained by real-time 

portrayal of events and led by a presenter. 

• Orient participants to, or provide an overview of, authorities, 

strategies, plans, policies, procedures, protocols, response 

resources, and/or concepts and ideas. 

Event Tests 

Time frame: Prior to start of exercise. 

Purpose: Conducted to ensure the modeling and simulation 

architecture is sufficient to meet exercise requirements. 

Exercise Setup 

Timeframe: Prior to start of exercise. 

• Exercise planning team assigned to setup should visit the 

exercise site at least 1 day prior to the event to arrange the room 

and test audio/video equipment. 

• On the day of the exercise, planning team members should arrive 

several hours before start of exercise to handle any remaining 

logistical or administrative items pertaining to setup, and to 

arrange for registration. 

Page 73 GAO-09-849 Homeland Defense
Appendix II: Comparison of DOD’s Joint 

Training System with DHS’s Homeland 

Security Exercise and Evaluation Program 

DOD Joint Training System and NORTHCOM Exercise 

Program Homeland Security Exercise and Evaluation Program Guidance 

Communications Exercise 

Time frame: Conducted approximately 2 weeks prior to start 

of exercise. 

Purpose: Confirm all exercise NORAD and NORTHCOM staff 

and external battle staff members have access to and are 

proficient with all required communications systems and 

collaborative tools. 

See Exercise Setup above 

Exercise Execution 

Time frame: Start of exercise. 

• Executed per the exercise plan. 

• Depending on the exercise, it may only involve the 

execution of a command post exercise or be greater in 

scope and involve the simultaneous and/or concurrent 

execution of other training events. 

Exercise Conduct 

Time frame: Start of exercise. 

• Includes setup and wrap-up activities. 

• For an operations-based exercise, conduct encompasses all 

operations occurring between the designated start of the exercise 

and the end of the exercise. 

Facilitated After-Action Review 

Time frame: After the end of the exercise. 

Purpose: A facilitated discussion held at the end of the 

exercise convened for the benefit of the training audience to 

examine actions and results during a training event. 

Hot Wash 

Time frame: Maximum 2 hours after end of exercise. 

Purpose: 

• Provides opportunity to ascertain the level of satisfaction with the 

exercise, identify issues or concerns, and propose areas for 

improvement. 

• Players complete and submit their Participant Feedback Forms 

during the hot wash. 

• All evaluators take notes during play and hot washes for later 

compilation with other observations from their functional areas. 

After-Action Review 

Time frame: May occur during and after the exercise. 

Purpose: An informal process designed by a commander or 

director to provide commanders direct feedback on the 

accomplishment of selected joint mission-essential tasks, 

conditions, and standards stated in terms of training 

objectives for the commander to evaluate training proficiency. 

Controller and Evaluator Debrief 

Time frame: Maximum 1 day after end of exercise. 

Purpose: Exercise planning team leader facilitates this debrief, which 

provides each controller and evaluator with an opportunity to provide 

an overview of the functional area he or she observed and to discuss 

both strengths and areas for improvement. 

See facilitated after-action review. After-Action Conference 

Time frame: 5 weeks after end of exercise, lasts approximately 1 day. 

Purpose: 

• Present, discuss, and refine the draft after-action report, and to 

develop an improvement plan. 

• Solicit feedback and make necessary changes to after-action 

report. 

Source: GAO analysis of DOD and DHS documentation. 

Page 74 GAO-09-849 Homeland Defense
Appendix III: Comments from the Department Appendix III: Comments from the of Defense 

Department of Defense 

Page 75 GAO-09-849 Homeland Defense 
Appendix III: Comments from the Department 

of Defense 

Page 76 GAO-09-849 Homeland Defense
Appendix III: Comments from the Department 

of Defense 

Page 77 GAO-09-849 Homeland Defense
Appendix III: Comments from the Department 

of Defense 

Page 78 GAO-09-849 Homeland Defense
Appendix III: Comments from the Department 

of Defense 

Page 79 GAO-09-849 Homeland Defense
Appendix IV: 

A

GAO Contact and Staff 

cknowledgments 

Page 80 GAO-09-849 

Appendix IV: GAO Contact and Staff 

Acknowledgments 

Davi M. D’Agostino, (202) 512-5431 or dagostinod@gao.gov. GAO Contact 

In addition to the contact named above, Joseph Kirschbaum, Assistant 

Director; Gilbert Kim; David Hubbell; Joanne Landesman; Christopher 

Mulkins; Erin Noel; Terry Richardson; and Richard Winsor made key 

contributions to this report. 

Acknowledgments 

Homeland Defense 
Related GAO Products Related GAO Products 

National Preparedness: FEMA Has Made Progress, but Needs to 

Complete and Integrate Planning, Exercise, and Assessment Efforts. 

GAO-09-369. Washington, D.C.: April 30, 2009. 

Emergency Management: Observations on DHS’ Preparedness for 

Catastrophic Disasters. GAO-08-868T. Washington, D.C.: June 11, 2008. 

National Response Framework: FEMA Needs Policies and Procedures to 

Better Integrate Non-Federal Stakeholders in the Revision Process. 

GAO-08-768. Washington, D.C.: June 11, 2008. 

Homeland Defense: Steps Have Been Taken to Improve U.S. Northern 

Command’s Coordination with States and the National Guards Bureau, 

but Gaps Remain. GAO-08-252. Washington, D.C.: April 16, 2008. 

Homeland Defense: U.S. Northern Command Has Made Progress but 

Needs to Address force Allocation, Readiness Tracking Gaps, and Other 

Issues. GAO-08-251. Washington, D.C.: April 16, 2008. 

Continuity of Operations: Selected Agencies Tested Various Capabilities 

during 2006 Governmentwide Exercise. GAO-08-185. Washington, D.C.: 

November 19, 2007. 

Homeland Security: Preliminary Information on Federal Action to 

Address Challenges Faced by State and Local Information Fusion 

Centers. GAO-07-1241T. Washington, D.C.: September 27, 2007. 

Homeland Security: Observations on DHS and FEMA Efforts to Prepare 

for and Respond to Major and Catastrophic Disasters and Address 

Related Recommendations and Legislation. GAO-07-1142T. Washington, 

D.C.: July 31, 2007. 

Influenza Pandemic: DOD Combatant Commands’ Preparedness Efforts 

Could Benefit from More Clearly Defined Roles, Resources, and Risk 

Mitigation. GAO-07-696. Washington, D.C.: June 20, 2007. 

Homeland Security: Preparing for and Responding to Disasters. 

GAO-07-395T. Washington, D.C.: March 9, 2007. 

Catastrophic Disasters: Enhanced Leadership, Capabilities, and 

Accountability Controls Will Improve the Effectiveness of the Nation’s 

Preparedness, Response, and Recovery System. GAO-06-618. Washington, 

D.C.: September 6, 2006. 

Page 81 GAO-09-849 Homeland Defense
Related GAO Products 

Homeland Defense: National Guard Bureau Needs to Clarify Civil 

Support Teams’ M9ission and Address Management Challenges. 

GAO-06-498. Washington, D.C.: May 31, 2006. 

Hurricane Katrina: Better Plans and Exercises Needed to Guide the 

Military’s Response to Catastrophic Natural Disasters. GAO-06-808T. 

Washington, D.C.: May 25, 2006. 

Hurricane Katrina: Better Plans and Exercises Needed to Guide the 

Military’s Response to Catastrophic Natural Disasters. GAO-06-643. 

Washington, D.C.: May 15, 2006. 

Hurricane Katrina: GAO’s Preliminary Observations Regarding 

Preparedness, Response, and Recovery. GAO-06-442T. Washington, D.C.: 

March 8, 2006. 

Emergency Preparedness and Response: Some Issues and Challenges 

Associated with major Emergency Incidents. GAO-06-467T. Washington, 

D.C.: February 23, 2006. 

Statement by Comptroller General David M. Walker on GAO’S 

Preliminary Observations Regarding Preparedness and Response to 

Hurricanes Katrina and Rita. GAO-06-365R. Washington, D.C.: 

February 1, 2006. 

Homeland Security: DHS’ Efforts to Enhance First Responders’ AllHazards Capabilities Continue to Evolve. GAO-05-652. Washington, D.C.: 

July 11, 2005. 

Homeland Security: Process for Reporting Lessons Learned from Seaport 

Exercises Needs Further Attention. GAO-05-170. Washington, D.C.: 

January 14, 2005. 

Homeland Security: Federal Leadership and Intergovernmental 

Cooperation Required to Achieve First Responder Interoperable 

Communications. GAO-04-740. Washington, D.C.: July 20, 2004. 

Homeland Defense: DOD Needs to Assess the Structure of U.S. Forces for 

Domestic Military Missions. GAO-03-670. Washington, D.C.: July 11, 2003. 

Combating Terrorism: Selected Challenges and Related 

Recommendations. GAO-01-822. Washington, D.C.: September 20, 2001. 

Page 82 GAO-09-849 Homeland Defense
Related GAO Products 

Joint Training: Observations on the Chairman, Joint Chiefs of Staff, 

Exercise Program. NSIAD-98-189. Washington, D.C.: July 10, 1998. 

(351178) Page 83 GAO-09-849 Homeland Defense
GAO’s Mission The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday afternoon, GAO 

posts on its Web site newly released reports, testimony, and 

correspondence. To have GAO e-mail you a list of newly posted products, 

go to www.gao.gov and select “E-mail Updates.” 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Phone The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s Web site, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537. 

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information. 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Ralph Dawn, Managing Director, dawnr@gao.gov, (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, DC 20548 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548 

Public Affairs 

Please Print on Recycled Paper