Background

The U.S. homeland continues to face an uncertain, complex security  environment with the potential for terrorist incidents and natural disasters  which can produce devastating consequences. Ensuring an effective  response will require that federal departments and agencies, states, and  local governments conduct integrated disaster response planning and test  these plans by exercising together.

		Training and Exercises

Exercises play an instrumental role in preparing the nation to respond to  an incident by providing opportunities to test emergency response plans,  evaluate response capabilities, assess the clarity of established roles and  responsibilities, and improve proficiency in a simulated, risk-free  environment. Short of performance in actual operations, exercises  provide the best means to assess the effectiveness of organizations in  achieving mission preparedness. Exercises provide an ideal opportunity to  collect, develop, implement, and disseminate lessons learned and to verify  corrective action taken to resolve previously identified issues. Sharing  positive experiences reinforces positive behaviors, doctrine, and tactics,  techniques, and procedures, while disseminating negative experiences  highlights potential challenges in unique situations or environments or  identifies issues that need to be resolved. According to the National  Response Framework, well-designed exercises improve interagency  coordination and communications, highlight capability gaps, and identify  opportunities for improvement. There are various types of exercises  ranging from tabletop exercises that involve key personnel discussing  simulated scenarios in informal settings to a full-scale exercise, including  many agencies, jurisdictions, and disciplines and a “boots on the ground”  response, such as firefighters decontaminating mock victims.

		Military Mission and Organization

DOD established the Office of the Assistant Secretary of Defense for  Homeland Defense and Americas’ Security Affairs to oversee homeland  defense activities for DOD, under the authority of the Under Secretary of  Defense for Policy, and, as appropriate, in coordination with the Chairman  of the Joint Chiefs of Staff. This office develops policies, conducts  analysis, provides advice, and makes recommendations on homeland  defense, defense support of civil authorities, emergency preparedness, and  domestic crises management matters within the department. The assistant  secretary assists the Secretary of Defense in providing policy directions to  NORTHCOM and other applicable combatant commands to guide the  development and execution of homeland defense plans and activities. This  direction is provided through the Chairman of the Joint Chiefs of Staff.
The Chairman of the Joint Chiefs of Staff, as principal military advisor to  the President and Secretary of Defense, has numerous responsibilities  relating to homeland defense and civil support, including providing advice  on operational policies, responsibilities, and programs. Furthermore, the  Chairman of the Joint Chiefs of Staff and the Joint Staff are responsible for  formulating joint training policy and doctrine. The Joint Staff assists the  Chairman by facilitating implementation of the Chairman’s joint training  programs, including the Joint Training System, Chairman’s sponsored  exercise program, and joint exercise program.
NORTHCOM is the military command responsible for planning, organizing,  and executing DOD’s homeland defense and civil support missions within  its area of responsibility—the continental United States (including Alaska)  and territorial waters (see fig. 1). Homeland defense is the protection of  U.S territory, sovereignty, domestic population, and critical defense  infrastructure against external threats and aggression. DOD is the primary  federal agency responsible for homeland defense operations, such as air  defense, and NORTHCOM is the combatant command responsible for  commanding and coordinating a response to a homeland defense  incident. To carry out its homeland defense mission, NORTHCOM is to  conduct operations to deter, prevent, and defeat threats and aggression  aimed at the United States.
NORTHCOM’s second mission is civil support or defense support of civil  authorities. Civil support is DOD support to U.S. civilian authorities, such  as DHS, for domestic emergencies, both natural and man-made, and  includes the use of DOD personnel—federal military forces and DOD’s  career civilian and contractor personnel—and DOD agency and  component resources. Because these missions are complex and  interrelated, they require significant interagency coordination. Civil  support missions include domestic disaster relief operations for incidents  such as fires, hurricanes, floods, and earthquakes. Such support also  includes counterdrug operations and management of the consequences of  a terrorist incident employing a weapon of mass destruction. DOD is not  the primary federal agency for such missions (unless so designated by the  President) and thus provides defense support of civil authorities only  when (1) state, local, and other federal resources are overwhelmed or  unique military capabilities are required; (2) assistance is requested by the  primary federal agency; and (3) NORTHCOM is directed to do so by the  President or the Secretary of Defense.  See fig. 2 for the pathway for  requesting DOD and NORTHCOM assistance during an incident.
NORTHCOM conducts or participates in exercises to improve readiness to  perform its assigned missions. The command annually conducts 2 large- scale exercises—Ardent Sentry and Vigilant Shield—and participates in  over 30 smaller command, regional, state, and local exercises. Each Ardent  Sentry and Vigilant Shield training event emphasizes one of the key  missions while at the same time including elements of the other. Ardent  Sentry emphasizes the civil support missions; Vigilant Shield the homeland  defense missions. The basis for NORTHCOM’s exercises is DOD’s Joint  Training System. NORTHCOM’s Training and Exercise Directorate is  responsible for planning and executing joint training, exercises, and  education programs to ensure NORTHCOM is prepared to accomplish its  assigned missions.

		DOD’s Joint Training System

Due to the need to prepare for and conduct military operations to defend  the United States and fight the nation’s wars, DOD has developed an  established, authoritative, time-tested process for planning, conducting,  and evaluating exercises in order to test and improve preparedness to  meet its wide range of critical missions. NORTHCOM uses DOD’s Joint  Training System as the basis to design, develop, and conduct exercises.  The Joint Training System provides an integrated, requirements-based  method for aligning training programs with assigned missions consistent  with command priorities, capabilities, and available resources. The joint  system consists of four phases beginning with the identification of critical  capabilities required based on assigned missions, proceeding through the  planning and scheduling of training events, the execution and evaluation  of required training, and assessing training proficiency against required  capability (see fig. 3). This process is designed to ensure that an  organization’s training program is linked to the Joint Mission Essential  Task List, the personnel executing the tasks are properly trained, and  shortfalls in training are identified and corrected in order to improve  readiness. The Joint Training Information Management System is an  automated system specifically designed to assist users in managing  elements of each of the four phases of the Joint Training System. During  the execution phase, commanders and directors focus on executing and  evaluating planned training events, which can be accomplished through  academic training, exercises, or a combination of these activities. During  the execution stage of the Joint Training System, the Joint Event Life  Cycle provides a five-stage methodology for joint-event development  design, planning, preparation, execution, and evaluation. For example,  DOD components prepare for the execution of an exercise by conducting  five conferences, such as the Concept Development Conference where  exercise and training objectives are discussed and scenarios developed.  Activities for the Joint Event Life Cycle are managed through the Joint  Training Information Management System.
Evaluating lessons learned and identifying issues for corrective actions are  fundamental components of DOD’s training and exercise process. The  Chairman of the Joint Chiefs of Staff provides policy, direction, and  guidance for DOD’s Joint Lessons Learned Program. The objectives of  this program are to collect and analyze observations from exercises and  real world events; disseminate validated observations and findings to  appropriate officials; identify and implement corrective actions; and track  corrective actions until reobserved in a subsequent exercise or event to  ensure that the issue has been successfully resolved. Combatant  commands, including NORTHCOM, execute lessons discovery, knowledge  development, and implementation activities scaled to meet the command’s  requirements while supporting and feeding into the Chairman’s Joint  Lessons Learned Program by identifying lessons applicable across  combatant commands and the services.

		The National Exercise Program

The NEP was established in April 2007 under the leadership of the  Secretary of Homeland Security to prioritize and coordinate federal, state,  and local exercise activities and serves as the principal mechanism for  examining the preparation of the federal government to respond to an  incident and adopting policy changes to improve such preparation. The  day-to-day staff-level coordination of the NEP is managed by the NEP  Executive Steering Committee—a working group of the White House’s  Domestic Readiness Group Exercise and Evaluation Sub-Policy  Coordination Committee—and is chaired and facilitated by FEMA’s  National Exercise Division. The steering committee is also responsible for  framing issues and recommendations for the full coordination committee  on exercise themes, goals, objectives, scheduling, and corrective actions.  Figure 4 illustrates the major events and milestones of the NEP and  NORTHCOM’s exercise program and table 1 provides information on  related major documents.
The NEP includes a series of national exercises projected on a 5-year  exercise schedule. These exercises are organized into four tiers with each  tier reflecting different requirements for interagency participation (see fig.  5).
FEMA administers the NEP and maintains the Homeland Security  Exercise and Evaluation Program—a capabilities and performance-based  exercise program—to provide standardized policy, methodology, and  terminology for exercise design, development, conduct, evaluation, and  improvement planning. DHS maintains policy and guidance for this  program.
Similar to DOD’s Joint Training System, the Homeland Security Exercise  Evaluation Program uses an exercise life cycle with five phases:  foundation, design and development, conduct, evaluation, and  improvement planning. This program also provides document templates  for exercise planning and evaluation and a collection of interactive, on-line  systems for exercise scheduling, design, development, conduct,  evaluation, and improvement planning, referred to as the Homeland  Security Exercise Evaluation Program Tool Kit (see fig. 6). FEMA also  has additional resources to support exercises. For example, exercise  stakeholders can access FEMA’s Lessons Learned Information Sharing  system, an interagency Web site for posting lessons learned and sharing  best practices, to learn about promising practices that could facilitate  exercise activities.
A national online comprehensive tool that facilitates scheduling and  synchronization of national-level, federal, state, and local exercises.
A project management tool and comprehensive tutorial for the design,  development, conduct, and evaluation of exercises.
Exercise Evaluation Guide Builder (Beta)
An online application that enables users to customize exercise evaluation  guides and templates.
Master Scenario Events List Builder (Beta)
A tool that enables users to create customized master scenario events list  formats by selecting from a list of data fields.
An online application that enables users to prioritize, track, and analyze  improvement plans developed from exercises and real-world events.

	NORTHCOM’s Exercise Program Is Consistent with DOD’s Joint Training System, but Exercise Reporting is Inconsistent

		NORTHCOM’s Exercise Program is Consistent with DOD’s Joint Training System

NORTHCOM’s Commander’s Training Guidance requires that NORTHCOM  establish a training and exercise program consistent with the Joint  Training System and establishes that training efforts and resources will be  focused on two large-scale exercises annually. The Joint Training System  requires, among other things, that an organization’s training objectives be  linked to its Joint Mission Essential Task List and include the use of the  Joint Events Life Cycle for planning, conducting, and assessing exercises.  We found that NORTHCOM has developed a comprehensive exercise  program consistent with DOD’s Joint Training System. For example,  NORTHCOM uses the Joint Training Information Management System to  link training objectives with its Joint Mission Essential Task List.  NORTHCOM officials enter information on task performance of exercise  participants into the Joint Training Information Management System to  evaluate the extent to which the command is trained based on  performance requirements in the Joint Mission Essential Task List.  NORTHCOM also uses the Joint Training Information Management System  to manage the Joint Events Life Cycle for its large-scale exercises,  including planning exercise milestones and developing a time line that  allows exercise planners to see where they are in the event life-cycle  process. For example, NORTHCOM holds five planning conferences for  each exercise, including a concept development conference, where  exercise and training objectives are discussed and scenarios developed.  We also found that NORTHCOM has conducted 13 large-scale exercises  since it was created in 2002, generally including 2 exercises each year (see  table 2). Vigilant Shield is held in the fall and focuses primarily on  NORTHCOM’s homeland defense mission, and Ardent Sentry is generally  conducted in the spring and focuses on defense support of civil  authorities.

		NORTHCOM’s Documentation of Exercises Is Generally Timely but Inconsistent

NORTHCOM guidance outlines the postexercise documentation required  to be completed for each exercise, including quick look, after-action, and  exercise summary reports; provides a time line for the completion of these  documents; and includes general direction that these documents follow  the same focus areas as the collection management plan—the source  document from which exercise analysts identify, examine, and  recommend emerging issues and trends. We found that NORTHCOM has  generally completed exercise summary reports for its exercises; however,  neither NORTHCOM nor Joint Forces Command officials could locate an  exercise summary report for Unified Defense 03.  In addition,  postexercise documentation is not consistently included on NORTHCOM’s  portal or the Joint Training Information Management System. NORTHCOM  guidance issued in June 2008 provides a time line for the completion of  postexercise documents and has been applicable to 2 subsequent  exercises–Ardent Sentry 08 and Vigilant Shield 09. According to the 2008  guidance, the exercise summary report is to be submitted to the  NORTHCOM Commander within 90 days of completing an exercise. The  Ardent Sentry 08 and Vigilant Shield 09 exercise summary reports were  issued 99 days and 92 days, respectively, after the completion of each  exercise. Overall, we reviewed exercise summary reports for 11 of  NORTHCOM’s large-scale exercises that have taken place since 2003.  Seven of the 11 exercise summary reports were issued within 100 days.  Four of the reports were issued later than 100 days, and 1 of  NORTHCOM’s earlier reports was issued in less than 30 days.
NORTHCOM guidance states that exercise summary reports should  provide the official description of the exercise, identify significant lessons  learned, and be targeted toward a national audience. Guidance also  requires that exercise summary reports follow the same focus areas as the  collection management plan—the source document from which exercise  analysts identify, examine, and recommend emerging issues and trends.  We found that NORTHCOM’s exercise summary reports generally included  an executive summary, training objectives, and the exercise’s major  scenarios and events, but did not consistently include lessons learned,  exercise strengths and weaknesses, or clear recommendations. The  exercise summary reports that included a section on lessons learned  lacked details. For example, 6 of the 11 exercise summary reports we  reviewed included an identified lessons learned section, and just 1 of these  6 reports—Unified Defense 04—provided additional information on  lessons learned beyond identifying the title of each observation and the  status of the observation in the lessons learned management system. As  discussed later in this report, access to this system is required in order to  obtain any additional information on the lesson learned.
We also found that NORTHCOM exercise summary reports have not  followed the same focus areas as collection management plans. For  example, none of the seven exercise summary reports for NORTHCOM  exercises conducted since Hurricane Katrina in 2005 reported on the  information identified in the collection management plans’ focus areas.  Inconsistencies in exercise documentation may be occurring because DOD  and NORTHCOM guidance do not require a standard format or specific  content for postexercise documentation. Although NORTHCOM uses other  methods to document exercises, such as the Joint Training Information  Management System, this system does not include a complete record of  each exercise. For example, the Joint Training Information Management  System does not include the lessons learned from an exercise. In addition,  access to this system is generally limited to DOD officials. Recognizing the  need for a complete and consistent record of each exercise, DHS’s  Homeland Security Exercise Evaluation Program provides a template for  exercise documentation, including format and content. NORTHCOM  used this template for National Level Exercise 2-08, but does not use the  template for its own exercises. Despite differences in the requirements  and complexities of NORTHCOM’s and DHS’ exercise programs, the lack  of a complete and consistent record of each exercise lessens the extent to  which NORTHCOM can ensure it has trained to key focus areas. Further, it  deprives the command of a key source of historical information upon  which to base current and future assessments of exercises and a  consistent venue for sharing lessons learned with interagency partners and  states.

	Interagency Partners and States Have Participated In NORTHCOM’s Exercises, but Challenges Remain with Including States in Planning, Conducting, and Assessing Exercises

		Federal Agencies and States Have Participated in NORTHCOM Exercises

NORTHCOM recognizes the importance of exercising with key partners in  all its missions and that, in order to achieve its goal of being trained and  ready to execute joint operations and ensure a seamless operating  environment, NORTHCOM should maximize exercise participation with  federal, state, and local agencies and National Guard units. NORTHCOM  has included interagency partners, such as DHS, FEMA, and the U.S. Coast  Guard, and several states in its large-scale exercises (see table 3). We  found that 17 civilian federal agencies and organizations have participated  to varying degrees in one or more of the seven large-scale NORTHCOM  exercises that have occurred since Hurricane Katrina made landfall in  August 2005. Seventeen states have participated in NORTHCOM exercises  since that time, and 8 of these states—Arizona, California, Connecticut,  Indiana, Michigan, Oregon, Rhode Island, and Washington—played a  major role by having a portion of the exercise conducted in the state and  having various state agencies and officials participate. For example,  Indiana and Rhode Island played major roles in Ardent Sentry 07 for the  detonation of a 10-kiloton improvised nuclear device and category III  hurricane impacting the New England region, respectively. Both states  established emergency operating centers and exercised large numbers of  state emergency management personnel.
State emergency management and National Guard officials told us that  they participated in NORTHCOM exercises because they wanted to better  understand the (1) capabilities that NORTHCOM could bring to the  response to an incident and (2) command and control issues of the troops  in a state when NORTHCOM is involved. We previously reported that  states’ participation in NORTHCOM exercises helps to build relationships  and improve coordination. Officials from all of the states we met with  told us that they derived benefits from their participation in these  exercises. For example, state emergency management officials from three  states told us that first-hand interaction with federal military forces and  the opportunity to observe the federal response to an incident was  beneficial. In addition, two state emergency management and National  Guard officials told us that NORTHCOM officials were professional, well- trained, and helpful. Further, officials from five states told us that  NORTHCOM provided beneficial resources, such as funds for travel to  attend exercise planning conferences and contractor staff to help state  officials prepare exercise scripts and injects. Finally, officials from two  states told us that the benefits of working with NORTHCOM included  gaining an understanding of the resources and capabilities that  NORTHCOM can provide, as well as understanding how NORTHCOM  coordinates its response through FEMA.
NORTHCOM is also attempting to include states in exercises through the  Vigilant Guard Program. The goal of the Vigilant Guard Program is to  enhance National Guard and State emergency management agency  preparedness to perform their homeland defense and Defense Support to  Civil Authorities roles and responsibilities. It focuses on State Guard Joint  Force Headquarters coordination with the state emergency management  agency and Joint Task Force-State operations and involves multiple states  and agencies. The program began in September 2004 and included one  exercise in fiscal year 2005. Now the plan is to conduct four exercises  annually. NORTHCOM was given management responsibility for the  Vigilant Guard exercises in 2007, although the National Guard Bureau  retains responsibility for budgeting for these events. Two of the four  annual Vigilant Guard exercises are to be linked to major combatant  command exercises, usually NORTHCOM’s Ardent Sentry and Vigilant  Shield. States hosting a Vigilant Guard exercise determine the objectives  for these events, and NORTHCOM provides support. Separate planning  begins for these Vigilant Guard exercises prior to the related planning  meetings for any linked NORTHCOM exercise. NORTHCOM’s Ardent  Sentry 09 is linked with a Vigilant Guard exercise in Iowa with scenarios  including a train derailment and a chemical spill, an epidemic outbreak,  and terrorism incident.

		NORTHCOM Faces Challenges Involving Interagency Partners and States in Planning, Conducting, and Assessing Exercises

A key element to developing effective working relationships with all states  is a well-thought out and consistent process for including the states in  planning, conducting, and assessing exercises. Without such a process,  states may be unwilling to participate in future NORTHCOM exercises,  impacting the seamless exercise of all levels of government and potentially  affecting NORTHCOM’s ability to provide support to civil authorities. We  found that challenges remain which have resulted in inconsistencies in the  way that NORTHCOM involves the states in its exercises. One of DOD’s  challenges is adapting its exercise system and practices to accommodate  the coordination and involvement of other federal, state, local, and tribal  agencies that do not have the same kinds of practices or level of planning  effort. Differences in exercise culture stem from differences in missions,  experience, authority, scope, and resources available to DOD, interagency  partners, and states. DOD has an established, authoritative, time-tested  process for planning, conducting, and evaluating exercises in order to test  and improve preparedness to meet its wide range of critical missions.  Within DOD, training and exercises are considered a vital component of its  overall mission of defending the national interests and significant  resources are devoted to these activities. In contrast, DHS, as the lead for  interagency homeland security efforts, is a new agency and has faced  challenges since it was created due to frequent reorganization and not  being fully staffed. DHS and other civilian agencies and state and local  governments have day-to-day missions and responsibilities that may take  priority over exercises and often do not have the resources or experience  to participate in or conduct exercises. For example, DOD exercises often  are conducted 24 hours a day, 7 days a week and may last a week or more  to enhance the realism of the exercise, while civilian agencies generally  participate 8 hours per day, usually—according to NORTHCOM officials— during normal business hours, and do not exercise longer than a few days.  Therefore, DOD exercises are generally longer in duration, more resource  intensive, and involve more participants than other federal and state  exercises. Furthermore, DOD views itself as the last line of defense and  often exercises until resources are exhausted to fully assess capabilities  and identify areas needing improvement. Civilian agencies and states may  prefer not to exhaust resources during an exercise in order to avoid  appearing unprepared for an incident and the associated political  controversy.
Another challenge that NORTHCOM faces is exercising with the various  states and territories within its area of responsibility considering the legal  and historical limits of the constitutional federal-state structure. The states  have a wide range and type of civilian state agencies responsible for  emergency management, some of which are headed by the Adjutant  General of the state, who also heads the military department or National  Guard, and others are completely separate entities (see table 4).
Working with states has been the responsibility of the National Guard and  is relatively new for a federal military command like NORTHCOM.  NORTHCOM officials face challenges in dealing with the various civilian  agencies, differing emergency management structures, capabilities, and  needs of the states. For example, for Ardent Sentry 08 (linked with  National Level Exercise 2-08), NORTHCOM planned a scenario involving a  chemical bomb attack in Seattle, Washington without consulting the state  health department or civil support team—the agencies responsible for  responding to a chemical or biological attack. State officials told us that  NORTHCOM invited the health department to participate once state  officials informed them that they should be involved, but that the scenario  was already locked in without the input of this key participant. DOD  officials told us that they rely on FEMA regional offices to provide  information on state agencies. However, we believe that NORTHCOM  officials should have determined if all relevant agencies were included in  the exercise when directly interacting with state officials during the  scenario development and other planning conferences, before the  scenarios were locked in. Washington emergency management officials  told us that this affected the realism of the exercise.
NORTHCOM also faces challenges in balancing its training objectives with  those of state agencies and organizations. State and local governments  seek to exercise their first responder capabilities before having their  resources overwhelmed and needing to seek federal assistance. On the  other hand, NORTHCOM seeks to exercise its capability to provide  support to civil authorities when local, state, and other federal resources  are exhausted. This necessarily requires scenarios that exceed the states’  capabilities and that stress DOD capabilities. Officials from four of the  seven states we interviewed told us that NORTHCOM’s exercise scenarios  appeared unrealistic, overwhelmed their states too soon during the  exercise, or did not allow states to fully exercise their own training  objectives. For example, the scenario for Ardent Sentry 06 included  multiple improvised explosive devices detonating over a 4-day period in  various sites, such as the City of Detroit, St. Clair and Wayne Counties,  Michigan, and Windsor, Ontario, Canada with over 14,000 fatalities and a  simultaneous pandemic flu outbreak in Michigan. State emergency  management officials told us that such a large number of casualties would  overwhelm state resources almost immediately, and therefore precluded  fully exercising training objectives for state and local responders.  Officials from these states told us that because they did not have the  opportunity to exercise their own training objectives, they believed  NORTHCOM was using them as a training tool.  A NORTHCOM official  told us that NORTHCOM needs the states to participate in exercises and,  therefore, will be flexible to accommodate other organizations’ training  objectives; however, NORTHCOM ultimately has its own objectives to  exercise. Officials from five of seven states noted that, for example, they  face budget and staffing limitations, and playing a major role in a  NORTHCOM exercise often requires establishing a state emergency  operations center with numerous staff and agencies involved. Given the  expansive scenarios NORTHCOM uses to guide its exercises and the  perception of half of the states we visited that this limits the benefits to  them, we believe that the states may be less likely to expend scarce  resources to participate in future NORTHCOM exercises.
Inconsistencies with how NORTHCOM involves states in planning,  conducting, and assessing exercises is occurring in part because  NORTHCOM officials lack experience dealing with the various state  agencies and emergency management structures. Inconsistencies are also  occurring because NORTHCOM has not established an informed,  consistent process for including states in its exercises. One aspect of this  process is the way that NORTHCOM requests state participation in its  exercises. Currently, NORTHCOM has various processes for requesting  that other federal departments and agencies participate in its exercises,  such as making the request through the Joint Staff. FEMA officials told us  that requests for state participation in NORTHCOM exercises should be  made through FEMA’s regional offices. However, because NORTHCOM  does not have an established process for requesting state participation,  officials from the states we visited told us that NORTHCOM officials made  requests informally and in a variety of ways, including through the  National Guard Bureau, the state’s National Guard, or FEMA’s regional  offices.  In some cases, such as when the state emergency management  agency and state National Guard have a close working relationship, this  method has been effective for NORTHCOM. However, in other cases, this  method has led to more limited exercises. For example, emergency  management officials from one state told us that NORTHCOM does not  have full state representation if it only exercises with the state National  Guard. In that case, NORTHCOM therefore misses out on interaction with  other key state emergency management officials and responders and  affects the realism of the exercise.
Another aspect of the lack of a consistent process for requesting state  participation is potentially missing the opportunity to leverage the existing  expertise of the National Guard Bureau and defense coordinating officers  located in each of the 10 FEMA regional offices. As we previously  reported, the National Guard Bureau and defense coordinating officers  have knowledge and experience in dealing with states in their region and  may be a valuable resource for NORTHCOM officials during the planning  and conduct of exercises. The three defense coordinating officers with  whom we met told us that they participate in NORTHCOM exercises, but  currently their role does not involve requesting state participation on  behalf of NORTHCOM or providing state-specific information to  NORTHCOM exercise officials. Without an informed and consistent  process for including the states in planning, conducting, and assessing its  exercises, NORTHCOM increases the risk that its exercises will not  provide benefits for all participants, impacting the seamless exercise of all  levels of government and potentially affecting NORTHCOM’s ability to  provide support to civil authorities.

	NORTHCOM has a Systematic Lessons Learned and Corrective Action Program, but Gaps Remain

		NORTHCOM Has a Systematic Lessons Learned and Corrective Action Process

DOD and NORTHCOM guidance requires that NORTHCOM identify  observations during the course of normal operations, exercises, and real- world events; capture the detail required to fully understand the problem;  and share valid lessons learned and issues as widely as possible.  NORTHCOM has been identifying observations, lessons learned, and  needed corrective actions from its exercises and operations since the  command was created in 2002. NORTHCOM collects and tracks  observations through the Joint Lessons Learned Information System  (JLLIS)—the automated official DOD system for managing and tracking  exercise observations and recording lessons learned. As of April 2009,  DOD exercise participants input 94 observations into JLLIS during  NORTHCOM’s most recent large-scale exercise, Vigilant Shield 09. Table 5  shows the observations entered into JLLIS or its predecessor for  NORTHCOM’s major exercises since 2006.
The philosophy and approach of NORTHCOM’s Lessons Learned Program  have been largely the same since NORTHCOM published its first  instruction for the program in 2003, although the requirement to re- observe corrective actions in a subsequent exercise or operation before  closing them was not established until 2005. We found that NORTHCOM  generally has a systematic lessons learned and corrective action program,  based on clear procedures and a regular process. Observations are  assigned to an office of primary responsibility within NORTHCOM and  categorized as either a lesson learned—a positive finding—-or an issue  which requires corrective action. NORTHCOM’s intent is to manage and  resolve issues requiring corrective action at the lowest organizational level  possible. This responsibility is generally within NORTHCOM’s various  directorates, component commands, or a Joint Task Force. Issues may  be closed at the directorate level without external approval or oversight.
Broader scope or more sensitive issues requiring the involvement of more  than one directorate or subcommand go into the formal corrective action  board process for review, tracking, and approval as necessary. This  formal process includes two boards—the Corrective Action Board and the  Executive Corrective Action Board—to review and resolve issues. Figure  7 illustrates the flow of NORTHCOM’s lessons learned and corrective  action process.

		Gaps Remain with Collecting Observations From and Sharing Lessons Learned with Interagency Partners and States

Joint Staff officials told us that about 10Department of State officials have access to  JLLIS, and DHS and Department of Energy officials have requested access for some of their  staff. Other departments and agencies, such as the Federal Bureau of Investigation and  Department of Health and Human Services, have received briefings about JLLIS, but have  not requested access. The Joint Staff official also told us that his office does not have  enough staff to support a large number of non-ommon access card users requesting JLLIS  access, and granting access would be a lengthy process due to the software and security  requirements that must be addressed.
Directorate officials into JLLIS. However, NORTHCOM’s portal the same DOD-issued card to gain entry. In addition, NORTHCOM’s  lessons learned manager told us no one has submitted observations using  the template since it was put on the portal. This may be because the  command has not actively publicized how to access the template a underscored the value to the command of obtaining observations from  interagency partners and states. In response to our inquiries in May 2009,   in the  NORTHCOM’s lessons learned manager told us that the command isprocess of adding a link to DHS’s Homeland Security Information Network so that interagency partners and states will be able to submit lessons  learned which can subsequently be transferred to JLLIS by NORTHCOM  officials.
In addition to collecting observations using JLLIS, NORTHCOM can obtain lessons learned from interagency partners and states during postexercise  ours  meetings. NORTHCOM conducts a review called a Hotwash within h of completing the exercise so that exercise participants can discuss  observations that significantly impacted their mission and recommend  emergent themes for discussion during a subsequent review known as the  facilitated after-action review. This review, generally held 7 days after the  s to  exercise is completed, provides an opportunity to present major issue senior leaders and obtain the Commander’s guidance for resolution.  However, the extent to which interagency and state officials are atten and participating in NORTHCOM’s postexercise meetings is unclear.  Based on NORTHCOM’s documentation, only two states (out of the last  six major exercises) participated in a Facilitated After Action Review— California in Vigilant Shield 09 and Alaska in Ardent Sentry 07. Officials  from three states in addition to California told us that they participated  the after-action meeting for the exercises they participated in, but they  may have participated in the regional or national-level meeting rather than  old us  NORTHCOM’s. Officials from two of the seven states we met with t that they did not attend NORTHCOM’s postexercise reviews for the  in  exercises in which they participated at least partly due to staffing and  budget limitations.
NORTHCOM has also attempted to share lessons learned with other  federal agencies and states by using FEMA’s lessons learned sharing  system. For example, NORTHCOM has posted six reports onto FEMA’s  lessons learned system, including four recent exercise reports and two  reports from operations in 2008. However, with one exception, the  documents that NORTHCOM has made available on this system (1)  include only lists of observations and, in some cases, record-tracking  numbers from JLLIS and previous lessons learned systems, and (2) lack  detailed information on individual lessons learned and corrective actions.  Joint Staff and NORTHCOM officials told us that they do not post detailed  information on the unclassified Lessons Learned Information Sharing  system Web site, because it is not adequately protected from the potential  for unauthorized access to records. As a result, the security of the  information cannot be assured. According to these officials, if an  adversary nation or terrorist group gained access to this information, it  may be possible for them to identify weaknesses in NORTHCOM’s  operations that can be exploited. In a recent exercise summary report  NORTHCOM stated that it will post lessons learned, best practices, and  reports that may benefit their non-DOD mission partners in FEMA’s  Lessons Learned Information Sharing system, which the report describes  as a secure, restricted-access information system. Because security  concerns are preventing NORTHCOM from openly sharing all its  unclassified lessons learned with its interagency partners and the states,  the information NORTHCOM does provide may be of limited value for  helping its partners improve the nation’s disaster responsiveness. Because  NORTHCOM is not fully involving other federal agencies and states in its  lessons learned process, it is missing opportunities to learn lessons from  an exercise. For example, officials from two states did not provide  NORTHCOM with lessons learned from exercises because they did not  attend the command’s post exercise reviews. As a result, NORTHCOM  risks the reoccurrence of potential problems that were not identified in its  process.

		NORTHCOM Faces Challenges Managing Corrective Actions

DOD and NORTHCOM guidance requires that issues requiring corrective  actions be tracked and remain open until the solutions are completed and  verified as effective—through training, operations, or exercises. We  found that NORTHCOM directorates and subcommands are closing some  issues prematurely, without confirming that corrective actions were made  or verifying in a subsequent exercise or operation that the corrective  action is effective. We reviewed unclassified records in JLLIS from  NORTHCOM’s previous six large-scale exercises and found at least 77 of  the 375 records or about 20 percent required corrective actions but were  either closed prior to completing the corrective action or closed without  verifying the effectiveness of the corrective action. For example, an  observation was made during Ardent Sentry 07 that NORTHCOM did not  have a process for addressing a foreign nation’s offer of military-to-military  assistance in a major disaster. The issue was validated and the corrective  action developed, but the issue was closed by the originating organization  before the corrective action could be verified or reobserved in a  subsequent exercise. The record was closed even though the Executive  Corrective Action Board directed that it remain open until an exercise of  suitable scope to require significant military support was developed.  Another example of a record being closed without verification or re- observation is an observation made during Ardent Sentry 07 raising  concerns that NORTHCOM personnel could arrive to assess a disaster site  without alerting state officials they would be coming. As a result,  NORTHCOM developed a new Command Assessment Element Concept of  Execution in July 2007 to promote better command and control and  situational awareness; however, the issue was closed before the procedure  could be observed in a subsequent exercise or operation to verify effective  resolution.
These issues are likely being closed without verification or re-observation,  because NORTHCOM Training and Exercise Directorate officials do not  have oversight over the disposition of open issues that are resolved within  directorates or are unable to give long-standing issues the sustained  management attention needed to ensure resolution. NORTHCOM’s lessons  learned manager told us that the command does not have the staff  necessary to oversee the actions on records handled within the other  directorates. In addition, while the checkbox format in JLLIS makes it easy  to see whether an issue is open, awaiting verification, or closed, entries  made in JLLIS regarding corrective actions required, implementation date,  and plan for verification are primarily in a narrative format, which may  make the review and oversight process more time consuming. Without  sufficient oversight, NORTHCOM cannot ensure that corrective actions  are verified and reobserved in a subsequent exercise or operation before  the issue is closed, so that the command knows the solution is effective.  We recognize that such oversight should be addressed without  significantly stressing NORTHCOM’s staff. However, if NORTHCOM does  not ensure that corrective actions are fully resolved, it increases the risk  that these issues may occur again, possibly during crucial, real-world  situations. This lack of oversight, coupled with the lack of a well-thought  out and consistent process for including the states in assessing exercises  as discussed earlier in this report, further limits the knowledge gained and  value of the exercise for all participants.

	NORTHCOM Has Taken Steps to Integrate Its Exercises with the National Exercise Program, but Guidance Is Not Consistently Applied

		NORTHCOM Has Participated in National Exercises and Taken Steps to Integrate with the NEP

Since the NEP Charter was approved in January 2007, NORTHCOM has  participated in the major national exercise held under the NEP and taken  steps to integrate its exercises into the national program. NEP guidance  requires that heads of departments and agencies actively participate in tier  I exercises and recommends participation in tier II exercises either  through the National Exercise Simulation Center or as determined by  agency leadership. Departments or agencies can participate in the NEP  by combining an existing exercise with a NEP exercise, taking part in a  tier II exercise sponsored by a different department or agency, or  requesting to lead a tier II exercise to obtain greater interagency  participation and support. DOD guidance requires that components  participate in or lead planning efforts of NEP exercises as appropriate  given the scenario or as tasked by the Assistant Secretary of Defense,  Homeland Defense and America’s Security Affairs or the Chairman of the  Joint Chiefs of Staff. NORTHCOM’s training guidance specifies the NEP  exercises in which the command plans to participate during the following  2 fiscal years. NORTHCOM combined two of its large-scale exercises— Vigilant Shield 08 and Ardent Sentry 08—with major national exercises  and has taken part in two additional exercises sponsored by other  departments (see table 6). For example, National Level Exercise 1-08, a  tier I exercise, and NORTHCOM’s Vigilant Shield 08 were conducted  October 15 -20, 2007, in parallel with Top Officials 4 and several other  exercises. These exercises were linked together by the use of common  scenarios and objectives intended to test existing plans, policies, and  procedures to identify planning and resource gaps and develop corrective  actions to improve preparedness against a weapons of mass destruction  attack.
NORTHCOM officials told us that they generally would like to participate  in NEP exercises to achieve the benefits of exercising with interagency  partners, but in some cases it is not beneficial to do so. For example, the  officials told us NORTHCOM decided not to combine Ardent Sentry 09  with National Level Exercise 09—a tier 1 exercise scheduled for July  2009—because the objectives and scenarios for the exercises did not meet  their training needs. Although NORTHCOM officials will conduct Ardent  Sentry 09 separately, they are using the National Exercise Simulation  Center—FEMA’s newly established training and exercise facility—to  provide a test run for the center’s use in National Level Exercise 09.
DOD and NORTHCOM have taken steps to integrate exercises with the  National Exercise Program, including posting the command’s exercises on  DHS’s National Exercise Schedule, successfully applying to lead a tier II  exercise, and publishing guidance on integration with the NEP. The NEP  Implementation Plan recommends that federal departments and agencies  post exercises on the NEP’s National Exercise Schedule so that exercises  and planning meetings can be synchronized across the federal  government. NORTHCOM has posted its annual Ardent Sentry and  Vigilant Shield exercises for the first 4 of 5 fiscal years on the national  schedule, while FEMA’s National Exercise Division has posted exercises  for the first 3 fiscal years. As of June 2009, neither the Joint Staff nor any  other combatant commands have posted exercises on the national  schedule. In addition, NORTHCOM recently requested and was granted  approval to lead Vigilant Shield 10 as a tier II exercise scheduled for  November 2009. Vigilant Shield 10 should have greater interagency  participation than it would have received as a tier 3 exercise, since federal  departments and agencies will be required, at a minimum, to participate in  the National Exercise Simulation Center. As of May 2009, the participants  of Vigilant Shield 10 include the DHS and the Departments of Justice,  Energy, Transportation, Health and Human Services, and Veteran Affairs;  the Joint Chiefs of Staff and U.S. Joint Forces Command; and other  government and nongovernment organizations. This exercise will be the  first time that NORTHCOM will share planning responsibilities with  FEMA’s National Exercise Division. This exercise will also be linked to a  Canadian government an exercise to demonstrate its readiness for the  2010 Olympics in Vancouver.

		NORTHCOM Has Not Consistently Applied NEP Guidance

NEP guidance includes policies and tools for the design, planning,  conduct, and evaluation of exercises—known as the Homeland Security  Exercise and Evaluation Program, which creates a common exercise  policy and consistent terminology for exercise planners and serves as the  foundation of NEP exercises. FEMA requires that entities, such as state  and local governments, receiving homeland security grant funding for their  exercises adhere to specific Homeland Security Exercise Evaluation  Program guidance for exercise program management, design, conduct,  evaluation, and improvement planning. We reviewed key program  documents, such as the Implementation Plan, and found that this guidance  is unclear about the extent to which federal agencies should use the  Homeland Security Exercise Evaluation Program. For example, the  Implementation Plan states that the NEP does not displace a preexisting  exercise program, and none of the NEP guidance requires that federal  agencies use the Homeland Security Exercise Evaluation Program.  However, the Implementation Plan states that the Homeland Security  Exercise Evaluation Program will serve as the doctrinal foundation for  NEP exercises.
FEMA officials told us that federal agencies should use this program when  participating in tier I and tier II exercises so that the various exercise  participants have consistency when planning, conducting, and assessing  exercises. We found that NORTHCOM generally has used DOD’s Joint  Training System guidance for planning NEP exercises, defining  capabilities, and reporting exercise results. NORTHCOM officials told us  that the Joint Training System is consistent with the NEP and served, in  part, as the basis for the Homeland Security Exercise and Evaluation  Program. We found that these sets of guidance have similar processes  but use different methods for defining the tasks and capabilities that are  performed and validated in an exercise. The primary differences between  these sets of guidance are that (1) DOD’s task list, which serves as the  basis for its exercises, includes tasks that are specific to military missions,  such as troop movements and sealifts; (2) DHS guidance provides more  detailed criteria for the postexercise documentation, such as content and  format; and (3) DHS’s planning cycle is generally shorter—9 to 15 months  versus 12 to 18 months for DOD. (See table 7.) See app. II for a more  detailed comparison.
According to both the Homeland Security Exercise and Evaluation  Program and Joint Training System guidance, it is important to link tasks  and capabilities with exercise objectives to ensure that participants  exercise or train as they would perform in a real-world event. The  Homeland Security Exercise and Evaluation Program recommends using  DHS’s Target Capabilities List or the Universal Task List to formulate the  tasks and capabilities that underlie the objectives for an exercise. These  lists describe the capabilities government entities need and tasks they are  expected to perform to prevent, protect against, respond to, and recover  from incidents of national significance. In contrast, NORTHCOM derives  its tasks and capabilities from the Universal Joint Task List to formulate  Joint Mission Essential Tasks. According to NORTHCOM guidance, the  command is required to include in its exercises the Joint Mission Essential  Tasks associated with its Joint Training Plan, which is updated annually.  These tasks are identified by joint force commanders as most essential to  their assigned or anticipated missions with priority given to their wartime  missions and describe their priority wartime requirements. We found that  DOD’s operating instruction for participation in the NEP does not provide  guidance on how DOD components should incorporate tasks and  capabilities derived from sources recommended by the Homeland Security  Exercise and Evaluation Program when participating in NEP exercises.  The primary differences between DHS’ and DOD’s lists are that DOD’s task  lists generally incorporate more descriptive metrics and criteria to assess  performance and include tasks that are specific to military missions, such  as troop movements and sealifts. In some cases, state National Guards  officials have had to translate DOD task lists into DHS tasks lists when  working with their civilian partners and vice versa.
We also found that neither DOD’s nor NORTHCOM’s guidance for  developing postexercise reports includes the same degree of specificity  recommended in the Homeland Security Exercise Evaluation Program.  For example, both sets of exercise guidance require postexercise reports;  however, the Homeland Security Exercise Evaluation Program provides  templates and guidance for these documents, including requiring an  improvement plan to clearly outline the corrective actions needed, which  are not included in DOD’s or NORTHCOM’s guidance. In addition,  NORTHCOM’s exercise summary reports for National Level Exercise 1-08  and 2-08 did not contain all information recommended by the Homeland  Security Exercise Evaluation Program. For example, NORTHCOM did not  include the recommended analyses regarding the capabilities and tasks  tied to the exercises’ objectives.
As stated above, we reviewed NEP guidance such as the Implementation  Plan and found it does not clearly state the extent to which federal  agencies are required to follow the Homeland Security Exercise and  Evaluation Program. As a result of this unclear guidance, we found that  agency officials have varying interpretations of the requirements. For  example, a DOD and a Joint Staff official told us that NEP guidance does  not require agencies to use the Homeland Security Exercise Evaluation  Program even for NEP exercises. Therefore, NORTHCOM uses the Joint  Training System rather than the Homeland Security Exercise and  Evaluation Program as the basis for planning, conducting, and assessing  exercises. However, officials from FEMA’s National Exercise Division told  us that all participating agencies should use the Homeland Security  Exercise and Evaluation Program guidance for tier I and tier II NEP  exercises. FEMA officials stated that federal departments and agencies  should be held accountable for meeting key requirements, but that FEMA’s  authority is limited to guiding, supporting, and coordinating with, but not  directing other federal departments and agencies to comply with guidance.  As we have previously reported, we believe that FEMA’s expanded  leadership role under the Post-Katrina Act provides FEMA opportunities  to instill a shared sense of responsibility and accountability on the part of  all agencies.
Neither DOD nor NORTHCOM guidance specifically addresses the extent  to which DHS’s Homeland Security Exercise and Evaluation Program  planning and documentation requirements should be followed. We  recognize that NORTHCOM and DOD must meet their own mission and  exercise requirements and the Joint Training System may be best suited  for NORTHCOM’s exercises; however, all of the states we visited use  Homeland Security Exercise and Evaluation Program guidance. We found  that having differing sets of guidance, such as DOD’s and DHS’ capabilities  task lists and postexercise documentation requirements, makes exercises  more difficult and potentially limits the benefits for participating states.  For example, officials from three states we visited told us that using  NORTHCOM’s exercise planning and reporting requirements rather than  Homeland Security Exercise Evaluation Program guidance has made the  processes more difficult. Further, the Defense Science Board found that  inconsistent approaches to the development and content of postexercise  documentation may affect the ability of organizations to fully learn lessons  identified in exercises. We also reported that when other federal entities  carry out processes that do not specifically follow the Homeland Security  Exercise and Evaluation Program, FEMA managers do not have the  necessary data to measure progress, identify gaps in preparedness, and  track lessons learned—key objectives of the NEP. We believe that  achieving national preparedness requires a whole-of-government approach  and is a shared responsibility among federal, state, local, and tribal  governments and organizations and an integration of their various  standards, policies, and procedures into the national system.

	Conclusions

There is an increasing realization within the federal government that an  effective, seamless national response to an incident requires a strong  partnership among federal, state, and local governments and  organizations, including integrated planning, training, and the exercise of  those plans. For DOD, the effective execution of civil support, especially  amid simultaneous, multijurisdictional disasters, requires ever-closer  working relationships with other departments and agencies and at all  levels of government. NORTHCOM’s use of DOD’s Joint Training System  has provided a robust process for planning and conducting exercises to  improve preparedness to achieve its homeland defense and civil support  missions, and its efforts to involve its interagency partners and the states  in exercises have helped to reduce uncertainty about the process for  responding to an incident. However, without a consistent record of what  has occurred during an exercise that is accessible by all exercise  participants, including those from other federal agencies and states,  NORTHCOM cannot ensure that it has met internal standards, trained to  key focus areas, or compared the goals and results of exercises over time.  Further, a key element to developing effective working relationships with  all states is a consistent process for including states in planning and  executing NORTHCOM’s exercises that incorporates state-specific  knowledge and information. By coordinating consistently with  organizations, like FEMA and NGB, that have knowledge and experience  dealing with states, NORTHCOM can improve the value and effectiveness  of exercises for all of the participants involved.
Exercises provide an opportunity to enhance preparedness by collecting,  developing, implementing, and disseminating lessons learned and verifying  corrective action taken to resolve previously identified issues.  NORTHCOM’s clear procedure for capturing observations in JLLIS and  identifying issues needing corrective action has helped to improve its  capabilities to complete its missions. However, by not providing federal  agencies and states greater access to its lessons learned process,  NORTHCOM will lose opportunities to learn valuable lessons from an  exercise, particularly observations from the states that could enhance  coordination and build more effective interagency relationships. Further,  the risk that issues may reoccur will be increased, particularly when  interagency partners are not aware of key issues or concerns that might  impede the government’s overall responsiveness to a natural or man-made  disaster. In addition, when corrective actions remain open until fully  implemented and verified in a subsequent exercise, NORTHCOM will have  greater assurance that issues raised during exercises are being adequately  addressed and the corrections are in fact solving the problems identified.
NEP policies and tools for the design, planning, conduct, and evaluation of  exercises are intended to create a common exercise policy and consistent  terminology for exercise planners across all levels of government to  improve the federal government’s ability to evaluate national  preparedness. The steps DOD and NORTHCOM have taken to integrate  exercises with the NEP have helped DHS to prioritize and coordinate  federal exercise activities and enhance the federal government’s ability to  respond to an incident. We recognize that NORTHCOM and DOD must  meet their own mission and exercise requirements and the Joint Training  System may be best suited to meet the high standards required for  NORTHCOM’s exercises. However, achieving national preparedness  requires shared responsibility among federal, state, and local governments  and organizations and an integration of their various standards, policies,  and procedures into the national system. We also recognize that the NEP  continues to evolve and become more useful to federal and state partners.  IHowever, in the absence of clear guidance from DHS on the extent to  which agencies should use Homeland Security Exercise Evaluation  Program planning and documentation guidance, DOD should ensure that  its components clearly understand when the use of this guidance is  appropriate so that both DOD and its exercise partners, such as other  federal agencies and states, derive the most benefits from exercises. This,  in turn, contributes to the ultimate success of a whole-of-government  approach to national preparedness.

	Recommendations for Executive Action

To improve NORTHCOM’s consistency with exercise documentation, we  recommend that the Secretary of Defense direct NORTHCOM’s  Commander to develop guidance with specific criteria for postexercise  documentation, particularly the Exercise Summary Report as the official  exercise record, including the content and format to be included in such  reports that will allow the results and lessons learned of exercises to be  easily reviewed and compared.
To improve NORTHCOM’s involvement of interagency partners and states  in its exercises, we recommend that  the Secretary of Defense, in consultation with the Chairman of the  Joint Chiefs of Staff, the Commander, U.S. Northern Command, and  other relevant combatant commanders, coordinate with the  Department of Homeland Security and Federal Emergency  Management Agency to develop guidance and procedures for  consistently involving state officials in planning, executing, and  assessing exercises that incorporate relevant state-specific  information, and  the Secretary of Defense direct NORTHCOM’s Commander to develop  a training plan for NORTHCOM headquarters staff on state emergency  management structures and relevant issues related to working with  civilian state and local emergency management officials.
To improve NORTHCOM’s involvement of interagency partners and states  in its lessons learned and corrective action process and its management of  corrective actions, we recommend that the Secretary of Defense direct:    NORTHCOM’s Commander to establish and publicize valid and easily  accessible procedures for non-DOD exercise participants to submit  observations relevant to NORTHCOM, such as placing a template on  NORTHCOM’s publicly accessible Web site or DHS’s Homeland  Security Information Network, so that NORTHCOM officials have a  clear, secure avenue to obtain observations and assess potential  lessons that originate with its exercise partners;  the Chairman of the Joint Chiefs of Staff, in consultation and  coordination with DHS, to either resolve information assurance issues  so that the combatant commands, including NORTHCOM, can post  Exercise Summary Reports with lessons learned and observations  from NEP exercises on DHS’s Lessons Learned Information Sharing  system to make them easily accessible to interagency partners and  states or establish an alternative method to systematically collect and  share lessons learned; and  the Chairman of the Joint Chiefs of Staff to revise the joint lessons  learned operating instruction to include procedures to ensure that  appropriate corrective actions are implemented and verified in a  subsequent exercise or operation before being closed and that the  reasons for closure are documented. Possible procedures might be  adding a verification checkbox on JLLIS’s issue management page or  requiring that the directorates and subordinate commands within the  combatant commands provide a status report when a correction is  implemented and reobserved or closed for reasons other than re- observation.
To improve NORTHCOM’s ability to work with interagency partners on  major national exercises and further achieve the objectives of the NEP, we  recommend that the Secretary of Defense revise the instruction on DOD  participation in the NEP and/or direct the Chairman of the Joint Chiefs of  Staff to revise the operating instruction regarding DOD participation in the  NEP to provide the general conditions under which the combatant  commands are expected to follow the Homeland Security Exercise and  Evaluation Program planning and documentation requirements or the  DOD’s Joint Training System should be modified for those civil support  exercises.

	Agency Comments and Our Evaluation

In comments on a draft of this report, DOD generally agreed with the  intent of our recommendations and discussed steps it is taking or plans to  take to address these recommendations. DOD also provided technical  comments, which we have incorporated into the report where appropriate.  DHS also reviewed a draft of this report and provided technical comments,  which we have incorporated where appropriate.
In response to our recommendation that NORTHCOM develop guidance  with specific criteria for postexercise documentation to allow the results  and lessons learned of exercises to be reviewed and compared, DOD  agreed that such information should be provided in a standardized format  that can be easily accessed and understood by authorized organizations  which might benefit from such knowledge. DOD cautioned that any  actions in response to this recommendation must accommodate  constraints regarding classified information. We agree that properly  securing classified information is a critical responsibility and believe this  can easily be accomplished without undermining the intent of the  recommendation, which is to improve the consistency and completeness  of formal exercise documentation and thereby its overall value.
In response to our recommendation that DOD coordinate with DHS and  FEMA to develop guidance and procedures for consistently involving state  officials in planning, executing, and assessing exercises that incorporate  relevant state-specific information, DOD agreed that better coordination  for interfacing with state officials can be achieved. DOD also pointed out  that NORTHCOM continues to expand its efforts to work through defense  coordinating officers, existing state National Guard relationships, and  FEMA regional headquarters partners to ensure that states are able to  benefit from participation in DOD-sponsored exercises. However, DOD  also said that while NORTHCOM has continuously engaged and  encouraged state participation in NORTHCOM-sponsored exercises, the  primary audience for such training is and must remain NORTHCOM. DOD  also suggested that our recommendation has applicability to other federal  interagency partners and that the issue should be addressed to the  Exercise and Evaluation Sub-Interagency Planning Committee as a  revision to the National Exercise Program Implementation Plan. As our  report indicates, we agree that NORTHCOM has sought to engage and  involve the states in its comprehensive exercise program. NORTHCOM  plans for and conducts major exercises both inside and outside the  construct of the National Exercise Program. Particularly for NORTHCOM- sponsored exercises focused on the command’s civil support mission, the  effective involvement of and interaction with state and other federal  partners is a critical component of improving and maintaining  NORTHCOM’s preparedness. For NORTHCOM’s participation in national- level exercises, the preparedness goals and objectives of all participants  are equally important. We believe that in developing procedures to  improve coordination with the states, DOD can (1) avoid situations where  exercises meant to improve preparedness are not fully coordinated with  the necessary partners; (2) capitalize on the structures and organizations it  already has in place, such as the defense coordinating officers and  relationships with state National Guard headquarters; and (3) coordinate  with DHS and FEMA to improve the military-civilian interface. With regard  to the latter, the Exercise and Evaluation Sub-Interagency Planning  Committee may indeed be one of the venues at which DOD can effectively  coordinate with its interagency partners.
With respect to our recommendation that NORTHCOM develop a training  plan for NORTHCOM headquarters staff on state emergency management  structures and relevant issues related to working with civilian state and  local emergency management officials, DOD agreed and noted that  headquarters training is required for all newly assigned NORTHCOM staff.  Further, DOD noted that NORTHCOM sponsors three versions of its  defense support of civil authorities seminar that are targeted to staff at  different seniority levels. We agree that NORTHCOM has continued to  improve the level of awareness and training it provides staff on the  complexities of providing defense support to civilian authorities in the  United States. However, this does not fully address our recommendation.  While training on the general procedures of the national response  framework, the nature of state-federal government relations, and DOD’s  proper role in providing military support to civil authorities is invaluable  for NORTHCOM staff, we continue to believe that this should be  supplemented by the kinds of state-specific information that would  provide both exercise officials and all other staff with an understanding of  the key differences between states. These differences are possibly as  numerous as the number of states and play a role in all routine  interactions between the individual states and DOD officials as well as for  effective coordination for exercise planning and coordination during a  natural disaster or some other no-notice incident requiring defense  support to civil authorities.
DOD agreed with our recommendation that NORTHCOM establish and  publicize valid and easily accessible procedures for non-DOD exercise  participants to submit observations relevant to NORTHCOM, such as  placing a template on NORTHCOM’s Web site or DHS’ Homeland Security  Information Network, so that NORTHCOM officials have a clear, secure  avenue to obtain observations and assess potential lessons that originate  with its exercise partners. DOD indicated that collecting exercise  information from all perspectives would provide additional opportunities  to improve NORTHCOM’s ability to accomplish its mission tasks.
DOD also agreed with our recommendation that it work with DHS to  either resolve information assurance issues so that NORTHCOM can post  Exercise Summary Reports with lessons learned on DHS’ Lessons Learned  Information Sharing system or establish an alternative method to  systematically collect and share lessons learned. DOD cautioned that  while wide dissemination of information approved for release would be of  great benefit to homeland security entities it continues to adhere to the  Joint Training System and cannot mandate that DHS alter its Lessons  Learned Information System to make accommodations. DOD also noted  that it has procedures in place to allow specifically cleared individuals  from outside DOD access to information contained in Exercise Summary  Reports. We agree that DOD cannot mandate alterations to the Lessons  Learned Information System. We also agree that the Joint Training System  should remain the chief guidance for the conduct of DOD exercises.  However, we continue to believe that in working with DHS on the proper  level and mode of information sharing, DOD may be able to improve the  dissemination of relevant exercise-related information to all appropriate  officials.
DOD agreed with our recommendation that the Chairman of the Joint  Chiefs of Staff revise the joint lessons learned operating instruction to  include procedures to ensure that appropriate corrective actions are  implemented and verified in a subsequent exercise or operation before  being closed and that the reasons for closure are documented. DOD  indicated that the Chairman of the Joint Chiefs of Staff Instruction  3150.25D could be expanded to provide more guidance and the Joint  Lessons Learned Information System could be updated to provide a  technological solution to address the issue once the process and  procedures are in place. DOD also indicated that the process of verifying  corrective action and closing issues will become more effective with the  modifications it outlined in response to the recommendation.
In response to our recommendation that DOD revise guidance on DOD  participation in the National Exercise Program to provide the general  conditions under which the combatant commands are expected to follow  the Homeland Security Exercise and Evaluation Program planning and  documentation requirements or the DOD’s Joint Training System should  be modified for those civil support exercises, DOD recognized the  importance of ensuring effective interaction with interagency partners for  homeland security-related exercises. However, DOD noted that the  National Exercise Program Implementation Plan contains language placed  there at DOD’s insistence that establishes a process to resolve doctrinal  differences during exercise planning. DOD indicated that together with  provisions in the implementation plan establishing the administration,  scope, and hierarchy of multiagency homeland security exercises and the  5-year National Exercise Program schedule, this should address our  recommendation. DOD further noted that The Joint Training System  remains the Secretary of Defense’s guidance on DOD exercises and that  the National Exercise Program Implementation Plan stipulates that  individual department or agency exercise programs should not be  replaced. We agree that the Joint Training System is and should be DOD’s  primary guidance for ensuring that DOD components train and exercise  according to standards. However, because interagency exercises are  becoming an ever larger part of the national preparedness effort, and to  the extent that effective exercise planning is bolstered by common  procedures, our recommendation is intended to help DOD clarify for its  components the circumstances under which the specific planning and  documentation requirements for the Homeland Security Exercise and  Evaluation Program can be followed without detriment to DOD’s high  training and exercise standards or compromise of the Joint Training  System.
DOD’s written comments are reprinted in appendix III.
We are sending copies of this report to the Secretary of Defense, Secretary  of Homeland Security, and other interested parties. In addition, the report  will be available at no charge on GAO’s Web site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-5431 or dagostinod@gao.gov. Contacts points for our Office of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix IV.

Appendix I: Scope and Methodology

In conducting this review, we generally focused our scope on U.S.  Northern Command’s (NORTHCOM) large-scale exercises conducted  since Hurricane Katrina made landfall in August 2005. To determine the  extent to which NORTHCOM’s exercise program is consistent with  Department of Defense (DOD) training and exercise requirements and  includes relevant exercise partners, we evaluated NORTHCOM’s  compliance with exercise reporting and documentation requirements  established in DOD and command guidance. We reviewed available  guidance to determine requirements for timing, format, and content. We  also compared these requirements with guidance contained in the  Department of Homeland Security’s (DHS) Homeland Security Exercise  and Evaluation Program documentation. We reviewed exercise  documentation for all large-scale exercises the command performed since  it was established in 2002 to determine the extent to which the command  complied with the guidance.
We also performed an assessment of the experiences and level of  participation from some interagency organizations and states in  NORTHCOM’s large-scale exercises. We initially met with Nevada officials  who participated in a NORTHCOM exercise prior to Hurricane Katrina— Determined Promise 03—to provide context to the extent that changes  may have been made to NORTHCOM’s exercise program and help develop  our state selection methodology. We selected a nongeneralizable sample of  six states based on the extent to which they have participated in major  NORTHCOM exercise since Hurricane Katrina and the varying scenarios  of the exercises. The states we selected played a major role in  NORTHCOM exercises by having a portion of the exercise conducted in  their state and having various state agencies and officials participate.  States we selected include Arizona, California, Michigan, Oregon, Rhode  Island, and Washington. We met with representatives from each state’s  emergency management organization and state national guard. Because of  the methodology selected, the resulting data and information from these  state visits could not be projected to make assumptions about the rest of  the states and what they may experience exercising with NORTHCOM. We  also met with officials from three Federal Emergency Management Agency  (FEMA) regional offices that had exercised with NORTHCOM in three of  the last six large-scale exercises. We also interviewed officials from the  Office of the Assistant Secretary of Defense for Homeland Security and  America’s Security Affairs, Joint Staff, and NORTHCOM with knowledge  of and experience with NORTHCOM’s training and exercise program.
To determine the extent to which NORTHCOM is using lessons learned  during exercises to improve mission preparedness, we reviewed DOD,  NORTHCOM, and DHS National Exercise Program (NEP) guidance for  recording, tracking, and managing lessons learned and assessed  NORTHCOM’s management of exercise observations and issues identified  in several of NORTHCOM’s large-scale exercises since Hurricane Katrina  in 2005. We interviewed NORTHCOM, Joint Staff, and FEMA officials  regarding the various lessons learned management systems, and how  interagency and state access to these systems can be accomplished. We  also spoke with an official in the General Services Administration  regarding the types of federal personal identification verification cards  used by DOD and other federal departments and agencies to access  government computer systems. In reviewing the management of  NORTHCOM’s lessons learned program we identified and reviewed all  unclassified exercise observations from its last six large-scale exercises  that had been activated in NORTHCOM’s area of the Joint Lessons  Learned Information System (JLLIS). Our review of the records in JLLIS  entailed determining each record’s status (open or closed), its type (issue  or lesson learned), and each record’s disposition after NORTHCOM staff  have acted on these records to respond to the issues or lessons learned  documented. Based on our review, we generally placed these records into  one of several categories: open; closed, nonconcur; issue closed with  reobservation; issue closed with no reobservation; and lesson learned. In  addition we reviewed several records that had been merged with other  original records because each related to the same issue; however, the  original record for that issue was not part of our universe. Therefore,  without reviewing the lead record the merged records lacked sufficient  information regarding their disposition and that condition became another  category.
Finally, to determine the extent to which NORTHCOM is integrating its  training and exercises with the NEP we reviewed DOD, NORTHCOM, and  Department of Homeland Security guidance to identify any differences in  exercise planning and documentation between DOD’s guidance and that  for the NEP. We used that analysis to determine under what conditions  NORTHCOM should apply standards related to the NEP, and how DOD  and its subordinate commands should participate in the NEP tier I or II  exercises. We reviewed NORTHCOM documentation from two major  national exercises conducted during fiscal year 2008 to determine the  extent to which NORTHCOM employed the guidance from the Homeland  Security Exercise Evaluation Program. We determined that national  exercises that are operations-based exercises in that they involved the  deployment of personnel would be the best candidates for evaluating  NORTHCOM’s participation in such exercises. We also interviewed state  emergency management and National Guard officials from six states that  have exercised with NORTHCOM since 2005, to understand the extent to  which NORTHCOM is integrating its exercise planning and conduct with  the interagency as well as various state governments.
In addressing our objectives, we reviewed plans and related documents,  obtained information, and interviewed officials at the following locations:    NORTHCOM Headquarters, Peterson Air Force Base, Colorado  Springs, Colorado  Joint Forces Command, Joint Warfighting Center, Suffolk, Virginia    The Office of the Secretary of Defense, Washington, D.C.    The Joint Staff, Washington, D.C.    U.S. Army North, Fort Sam Houston, San Antonio, Texas    National Guard Bureau, Arlington, Virginia    Department of Homeland Security, Washington, D.C.    U.S. Coast Guard, Atlantic Area, Portsmouth, VA    FEMA’s National Preparedness Directorate, Washington, D.C.    FEMA Region 1, Boston, Massachusetts    FEMA Region 9, Oakland, California    FEMA Region 10, Bothell, Washington    General Services Administration, Washington, D.C.    Arizona Division of Emergency Management, Phoenix, Arizona    Arizona National Guard, Joint Force Headquarters, Phoenix, Arizona    California Emergency Management Agency, Sacramento, California    California National Guard, Joint Force Headquarters, Sacramento,    Michigan State Police, Emergency Management and Homeland  Security Division, Lansing, Michigan    Michigan National Guard, Lansing, Michigan    Nevada State Division of Emergency Management, Carson City,    Nevada National Guard, Joint Force Headquarters, Carson City,    Rhode Island Emergency Management Agency, Cranston, Rhode Island    Rhode Island National Guard, Joint Force Headquarters, Cranston,    Oregon Military Department, Office of Emergency Management,  Salem, Oregon    Oregon Military Department, National Guard Joint Force  Headquarters, Salem, Oregon    Washington Military Department, Emergency Management Division,    Washington National Guard, Joint Force Headquarters, Camp Murray,  We conducted our review from June 2008 to September 2009 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.

Appendix II: Comparison of DOD’s Joint Training System with DHS’s Homeland Security Exercise and Evaluation Program

We reviewed the time lines and milestones for developing exercises found  in the Department of Defense’s Joint Training System and U.S. Northern  Command’s (NORTHCOM) implementing guidance and compared them  with the Department of Homeland Security’s Homeland Security Exercise  and Evaluation Program guidance to determine the similarities and  differences between them. We used the guidance associated with  operation-based exercises rather than discussion-based exercises to  present the full spectrum of Homeland Security Exercise and Evaluation  Program processes and planning events.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

Davi M. D’Agostino, (202) 512-5431 or dagostinod@gao.gov.

	Acknowledgments

In addition to the contact named above, Joseph Kirschbaum, Assistant  Director; Gilbert Kim; David Hubbell; Joanne Landesman; Christopher  Mulkins; Erin Noel; Terry Richardson; and Richard Winsor made key  contributions to this report.

Related GAO Products

National Preparedness: FEMA Has Made Progress, but Needs to  Complete and Integrate Planning, Exercise, and Assessment Efforts.  GAO-09-369. Washington, D.C.: April 30, 2009.
Emergency Management: Observations on DHS’ Preparedness for  Catastrophic Disasters. GAO-08-868T. Washington, D.C.: June 11, 2008.
National Response Framework: FEMA Needs Policies and Procedures to  Better Integrate Non-Federal Stakeholders in the Revision Process.   GAO-08-768. Washington, D.C.: June 11, 2008.
Homeland Defense: Steps Have Been Taken to Improve U.S. Northern  Command’s Coordination with States and the National Guards Bureau,  but Gaps Remain. GAO-08-252. Washington, D.C.: April 16, 2008.
Homeland Defense: U.S. Northern Command Has Made Progress but  Needs to Address force Allocation, Readiness Tracking Gaps, and Other  Issues. GAO-08-251. Washington, D.C.: April 16, 2008.
Continuity of Operations: Selected Agencies Tested Various Capabilities  during 2006 Governmentwide Exercise. GAO-08-185. Washington, D.C.:  November 19, 2007.
Homeland Security: Preliminary Information on Federal Action to  Address Challenges Faced by State and Local Information Fusion  Centers. GAO-07-1241T. Washington, D.C.: September 27, 2007.
Homeland Security: Observations on DHS and FEMA Efforts to Prepare  for and Respond to Major and Catastrophic Disasters and Address  Related Recommendations and Legislation. GAO-07-1142T. Washington,  D.C.: July 31, 2007.
Influenza Pandemic: DOD Combatant Commands’ Preparedness Efforts  Could Benefit from More Clearly Defined Roles, Resources, and Risk  Mitigation. GAO-07-696. Washington, D.C.: June 20, 2007.
Homeland Security: Preparing for and Responding to Disasters.   GAO-07-395T. Washington, D.C.: March 9, 2007.
Catastrophic Disasters: Enhanced Leadership, Capabilities, and  Accountability Controls Will Improve the Effectiveness of the Nation’s  Preparedness, Response, and Recovery System. GAO-06-618. Washington,  D.C.: September 6, 2006.
Homeland Defense: National Guard Bureau Needs to Clarify Civil  Support Teams’ M9ission and Address Management Challenges.  GAO-06-498. Washington, D.C.: May 31, 2006.
Hurricane Katrina: Better Plans and Exercises Needed to Guide the  Military’s Response to Catastrophic Natural Disasters. GAO-06-808T.  Washington, D.C.: May 25, 2006.
Hurricane Katrina: Better Plans and Exercises Needed to Guide the  Military’s Response to Catastrophic Natural Disasters. GAO-06-643.  Washington, D.C.: May 15, 2006.
Hurricane Katrina: GAO’s Preliminary Observations Regarding  Preparedness, Response, and Recovery. GAO-06-442T. Washington, D.C.:  March 8, 2006.
Emergency Preparedness and Response: Some Issues and Challenges  Associated with major Emergency Incidents. GAO-06-467T. Washington,  D.C.: February 23, 2006.
Statement by Comptroller General David M. Walker on GAO’S  Preliminary Observations Regarding Preparedness and Response to  Hurricanes Katrina and Rita. GAO-06-365R. Washington, D.C.:   February 1, 2006.
Homeland Security: DHS’ Efforts to Enhance First Responders’ All- Hazards Capabilities Continue to Evolve. GAO-05-652. Washington, D.C.:  July 11, 2005.
Homeland Security: Process for Reporting Lessons Learned from Seaport  Exercises Needs Further Attention. GAO-05-170. Washington, D.C.:  January 14, 2005.
Homeland Security: Federal Leadership and Intergovernmental  Cooperation Required to Achieve First Responder Interoperable  Communications. GAO-04-740. Washington, D.C.: July 20, 2004.
Homeland Defense: DOD Needs to Assess the Structure of U.S. Forces for  Domestic Military Missions. GAO-03-670. Washington, D.C.: July 11, 2003.
Combating Terrorism: Selected Challenges and Related  Recommendations. GAO-01-822. Washington, D.C.: September 20, 2001.
Joint Training: Observations on the Chairman, Joint Chiefs of Staff,  Exercise Program. NSIAD-98-189. Washington, D.C.: July 10, 1998.

Summary:
Why GAO Did This Study

U.S. Northern Command (NORTHCOM) exercises to test preparedness to perform its homeland defense and civil support missions. GAO was asked to assess the extent to which NORTHCOM is (1) consistent with Department of Defense (DOD) training and exercise requirements, (2) involving interagency partners and states in its exercises, (3) using lessons learned and corrective actions to improve preparedness, and (4) integrating its exercises with the National Exercise Program (NEP). To do this, GAO reviewed NORTHCOM and NEP guidance and postexercise documentation, assessed NORTHCOM compliance, and compared DOD and NEP exercise requirements.

What GAO Found

NORTHCOM's exercise program is generally consistent with the requirements of DOD's Joint Training System, but its exercise reporting is inconsistent. Since the command was established in 2002, NORTHCOM has conducted 13 large-scale exercises and generally completed exercise summary reports within the required time frame. However, those reports did not consistently include certain information, such as areas needing improvement, because NORTHCOM lacks guidance that specifies exercise reports' content and format, potentially impacting its ability to meet internal standards for planning and execution of joint exercises, and to compare and share exercise results over time with interagency partners and states. Nineteen federal agencies and organizations and 17 states and the District of Columbia have participated in one or more of the seven large-scale exercises that NORTHCOM has conducted since September 2005. However, NORTHCOM faces challenges in involving states in the planning, conduct, and assessment of its exercises, such as adapting its exercise system and practices to involve other federal, state, local, and tribal agencies that do not have the same practices or level of planning resources. Inconsistencies with how NORTHCOM involves states in exercises are occurring in part because NORTHCOM officials lack experience dealing with states and do not have a consistent process for including states in exercises. Without such a process, NORTHCOM increases the risk that its exercises will not provide benefits for all participants, impact the seamless exercise of all levels of government, and potentially affect NORTHCOM's ability to provide civil support capabilities. NORTHCOM has a systematic lessons learned and corrective action program to improve preparedness, but gaps remain with collecting and sharing lessons with agency and state partners and managing corrective actions. Access to the system NORTHCOM uses for managing exercise observations is limited for non-DOD participants, and DOD believes that the Department of Homeland Security's system is not adequately protected from unauthorized users. NORTHCOM's mitigation steps have not resolved the issues. In addition, about 20 percent of the corrective actions tracked by NORTHCOM were being closed prematurely due to gaps in oversight. Closing issues prematurely increases the risk that issues will reoccur and limits the knowledge gained and value of the exercise. NORTHCOM has taken steps to integrate its exercises with the NEP, but guidance is not consistently applied. NORTHCOM has participated in several NEP exercises and is leading its first major NEP exercise in the fall of 2009. However, NORTHCOM has used DOD's Joint Training System planning and documentation requirements rather than DHS's requirements, because NEP guidance is not clear on what exercise planning standard should be used and DOD guidance does not address the issue. The states we visited use NEP guidance. Differences between NEP and DOD guidance could affect the ability of all participants to develop effective working relationships.