Homeland Security: Process for Reporting Lessons Learned from Seaport Exercises Needs Further Attention
Published: 2005-01-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Seaports are a critical vulnerability in the nation's defense against terrorism. They are potential entry points for bombs or other devices smuggled into cargo ships and ports' often-sprawling nature presents many potential targets for attack. To assess the response procedures that would be implemented in an attack or security incident, officials conduct port-specific exercises. Many federal, state, and local agencies may potentially be involved. The Coast Guard has primary responsibility for coordinating these exercises and analyzing the results. GAO examined (1) the emerging framework for coordinating entities involved in security responses, (2) legal and operational issues emerging from exercises conducted to date, and (3) Coast Guard management of reports analyzing exercises. GAO reviewed reports on 82 exercises from fiscal year 2004 and observed 4 exercises as they were being conducted.
section_title: What GAO Found
paragraphs: The framework under which federal agencies would manage a port-terrorism incident is still evolving. The primary guidance for response, the National Response Plan, is in the final stages of approval, and the National Incident Management System, the structure for multiagency coordination, is still being put in place. As a result, it is too early to determine how well the complete framework will function in an actual incident. GAO's review of fiscal year 2004 terrorism-related reports and exercises identified relatively few legal issues, and none of these issues produced recommendations for statutory changes. Most issues have instead been operational in nature and have surfaced in nearly every exercise. They are of four main types: difficulties in sharing or accessing information, inadequate coordination of resources, difficulties in coordinating effectively in a command and control environment, and lack of knowledge about who has jurisdictional or decision-making authority. Reports on the exercises often do not meet the Coast Guard's standards for timeliness or completeness. Sixty-one percent of the reports were not submitted within 60 days of completing the exercise--the Coast Guard standard. The Coast Guard has implemented a new system for tracking the reports, but after a year of use, timeliness remains a concern. The Coast Guard has requirements for what the reports should contain, but 18 percent of the reports did not meet the requirement to assess each objective of the exercise. The Coast Guard has cited several planned actions that may allow for improving completeness. These actions are still in development, and it is too early to determine how much they will help.

section_title: Letter
paragraphs: Seaports have emerged as a critical vulnerability in the nation’s defense  against terrorism. More than 95 percent of the nation’s overseas trade  passes through these ports, and the nation’s economy is highly dependent  on an efficient transfer of goods flowing into and out of these gateways.  Seaports are vulnerable entry points for bombs or other devices smuggled  into cargo ships, and ports’ often-sprawling nature presents many  potential targets for attack. But, the repercussions of such an attack go far  beyond the port immediately affected; an attack at one port could disrupt  the broader flow of goods, creating economic consequences in the billions  of dollars.
In the wake of the terrorist attacks of September 11, 2001, law  enforcement and other agencies have become more aware of the threats to  America’s strategic targets and have taken steps to make ports more  secure. To assess coordination and response procedures that would be  implemented in the event of a terrorist attack, officials in U.S. ports have  conducted exercises that simulate a potential threat, attack, or incident.  These exercises have addressed such scenarios as the explosion of a “dirty  bomb” that releases radioactive materials, threats of an approaching ship  that may have a bomb or other hazardous material aboard, or disruptive  attacks on critical infrastructure or specific facilities within a port. In  fiscal year 2004, the United States Coast Guard, which is the Department  of Homeland Security agency with primary responsibility for port security,  conducted 85 port-based terrorism exercises.
Responding effectively in such exercise scenarios can be difficult.  Depending on the nature of the incident and the particular port involved,  dozens of federal, state, and local agencies may be involved. The incident  may also require close coordination across many jurisdictions, raising  issues about who has authority or how agency personnel can  communicate effectively when they have different chains of command,  operating procedures, and equipment. After these exercises are  conducted, the Coast Guard requires that the unit participating in the  exercise submit an “after-action” report describing the results and  highlighting any lessons learned. These reports are approved for content  and lessons learned by the commanding officer of that unit. Upon  approval, each report is electronically transmitted to Coast Guard  headquarters for inclusion in a master exercise database. Analysis of these  reports presents an opportunity to identify potential barriers to an  effective response during an actual threat or incident. These reports can  also provide valuable input for future exercises conducted by the Coast  Guard or other agencies.
You asked that we examine what has been learned to date in the nation’s  attempts to use such exercises to coordinate effective port security  procedures. Our specific objectives were to (1) describe the emerging  framework under which the federal government coordinates with state  and local entities to address a terrorist incident in a U.S. port; (2) identify  the issues, if any, regarding federal agencies’ legal authority that have  emerged from port security exercises and what statutory actions might  address them; (3) describe the types of operational issues being identified  through these exercises; and (4) identify any management issues related to  Coast Guard-developed after-action reports. On December 8, 2004, we  briefed your offices on the results of our work. Appendix I contains the  briefing slides we presented at that meeting.
Our description of the emerging framework for coordinating an effective  response to a terrorist incident is based on our review of statutes,  regulations, directives, plans, and agency guidance, as well as interviews  with officials from various agencies at the federal, state, and local levels.  Our analysis of statutory and operational issues arising from exercises is  based on a review of 82 Coast Guard after-action reports on terrorism- related exercises conducted at U.S. ports in fiscal year 2004. In addition,  to obtain a more in-depth understanding of exercise issues, we attended  four exercises at various locations around the country. Our evaluation of  management issues related to Coast Guard after-action reports included  interviews with officials at Coast Guard headquarters and in the field,  analysis of Coast Guard and Department of Homeland Security guidance,  and reliance on our prior work related to managing and evaluating  exercises. While our audit work allowed us to review a large number of  exercises for potential legal and operational issues, our analysis is limited  to the degree that, (1) for most exercises, we had to rely on Coast-Guard- developed after-action reports rather than direct observation and (2) the  scope of the exercises themselves was designed around specific plan  elements and subject to resource constraints. Appendix II describes our  scope and methodology in more detail. We conducted our work from June  to December 2004 in accordance with generally accepted government  auditing standards.
subsections: section_title: Background
paragraphs: On November 25, 2002, the President signed into law the Homeland  Security Act, which created the new federal Department of Homeland  Security, and the Maritime Transportation Security Act, which created a  consistent security program specifically for the nation’s seaports. Since  that time, and in keeping with the provisions of these new laws, the federal  government has been developing a variety of new national policies and  procedures for improving the nation’s response to domestic emergencies.  These policies and procedures are designed to work together to provide a  cohesive framework for preparing for, responding to, and recovering from  domestic incidents.
A key element of this new response framework is the use of exercises to  test and evaluate federal agencies’ policies and procedures, response  capabilities, and skill levels. The Coast Guard has primary responsibility  for such testing and evaluation in the nation’s ports and waterways, and as  part of its response, it has added multiagency and multicontingency  terrorism exercises to its training program. These exercises vary in size  and scope and are designed to test specific aspects of the Coast Guard’s  terrorism response plans, such as communicating with state and local  responders, raising maritime security levels, or responding to incidents  within the port. For each exercise the Coast Guard conducts, an after- action report detailing the objectives, participants, and lessons learned  must be produced within 60 days.
section_title: Results
subsections: section_title: Coordination Framework Is Still Evolving
paragraphs: The framework under which federal agencies would coordinate with state  and local entities to manage a port-terrorism incident is still evolving. As  directed by Homeland Security Presidential Directive/HSPD-5, issued in  February 2003, this framework is designed to address all types of  responses to national emergencies, not just port-related events. Key  elements of the framework have been released over the past 2 years. For  example, the Department of Homeland Security released the Interim  National Response Plan in September 2003 and was in the final approval  stage for a more comprehensive National Response Plan in November  2004, as our work was drawing to a close. DHS announced the completion  of the National Response Plan on January 6, 2005, too late for a  substantive review to be included in this report.  However, the finalized  plan is designed to be the primary operational guidance for incident  management and, when fully implemented, will incorporate or supersede  existing federal interagency response plans. According to the updated  implementation schedule in the National Response Plan, federal agencies  will have up to 120 days to bring their existing plans, protocols, and  training into accordance with the new plan. In March 2004, the department  also put in place a system, called the National Incident Management  System, which requires common principles, structures, and terminology  for incident management and multiagency coordination. Although the  framework that will be brought about by the final plan, the management  system, and other actions is still in the implementation phase, some of the  protocols and procedures contained in this framework were already  evident at the port exercises we observed. However, it is still too early to  determine how well the complete framework will function in coordinating  an effective response to a port-related threat or incident.
section_title: Exercises Identified Few Legal Issues
paragraphs: Port security exercises have identified relatively few issues related to  federal agencies’ legal authority, and none of these issues were statutory  problems according to exercise participants and agency officials. Our  review of fiscal year 2004 after-action reports and observation of specific  exercises showed that exercise participants encountered seven legal  issues, but exercise participants and agency officials we interviewed did  not recommend statutory changes to address these issues. In three  instances, exercise participants made nonstatutory recommendations  (such as policy clarifications) to assist agencies in better exercising their  authority, but did not question the adequacy of that authority. In the other  four instances, no recommendations were made either because statutory  authority was deemed sufficient or, in one case, because the issue  involved a constitutional restraint (i.e., under the Fourth Amendment,  police are prohibited from detaining passengers not suspected of  terrorism).
While the exercises were conducted to examine a wide range of issues and  not specifically to identify gaps in agencies’ legal authority, the results of  the exercises are consistent with the information provided by agency  officials we interviewed, who indicated that sufficient statutory authority  exists to respond to a terrorist attack at a seaport. Moreover, when  Department of Homeland Security officials reviewed the issue of statutory  authority, as required by Homeland Security Presidential Directive/HSPD- 5, they concluded that federal agencies had sufficient authority to  implement the National Response Plan and that any implementation issues  could be addressed by nonstatutory means, such as better coordination  mechanisms.
section_title: Exercises Identified Four Main Types of Operational Issues
paragraphs: Most of the issues identified in port security exercises have been  operational rather than legal in nature. Such issues appeared in most after- action reports we reviewed and in all four of the exercises we observed.  While such issues are indications that improvements are needed, it should  be pointed out that the primary purpose of the exercises is to identify  matters that need attention and that surfacing problems is therefore a  desirable outcome, not an undesirable one. The operational issues can be  divided into four main categories, listed in descending order of frequency  with which they were reported:    Communication—59 percent of the exercises raised communication  issues, including problems with interoperable radio communications  among first responders, failure to adequately share information across  agency lines, and difficulties in accessing classified information when  needed.
Adequacy or coordination of resources—54 percent of the exercises  raised concerns with the adequacy or coordination of resources, including  inadequate facilities or equipment, differing response procedures or levels  of acceptable risk exposure, and the need for additional training in joint  agency response.
Ability of participants to coordinate effectively in a command and  control environment—41 percent of the exercises raised concerns  related to command and control, most notably a lack of knowledge or  training in the incident command structure.
Lack of knowledge about who has jurisdictional or decision-making  authority—28 percent of the exercises raised concerns with participants’  knowledge about who has jurisdiction or decision-making authority. For  example, agency personnel were sometimes unclear about who had the  proper authority to raise security levels, board vessels, or detain  passengers.
section_title: After-Action Reports Show Problems Related to Timeliness and Completeness
paragraphs: Our review of the Coast Guard’s fiscal year 2004 after-action reports from  port terrorism exercises identified problems with timeliness in completing  the reports and limitations in the information they contained. Specifically,    Timeliness: Coast Guard guidance states that after-action reports are an  extremely important part of the exercise program, and the guidance  requires that such reports be submitted to the after-action report database  (Contingency Preparedness System) within 60 days of completing the  exercise. However, current practice falls short: 61 percent of the 85 after- action reports were not submitted within this 60-day time frame. Late  reports were submitted, on average, 61 days past the due date. Exercises  with late reports include large full-scale exercises designed to identify  major interagency coordination and response capabilities. Not meeting the  60-day requirement can lessen the usefulness of these reports. Coast  Guard guidance notes, and officials confirm, that exercise planners should  regularly review past after-action reports when planning and designing  future exercises, and to the extent that reports are unavailable, such  review cannot be done. In previous reviews of exercises conducted by the  Coast Guard and others, we found that timely after-action reports were  necessary to help ensure that potential lessons can be learned and applied  after each counterterrorism exercise. The main problem in producing  reports on a more timely basis appeared to be one of competing priorities:  Coast Guard field personnel indicated that other workload priorities were  an impediment to completing reports, but most of them also said 60 days is  a sufficient amount of time to develop and submit an after-action report.
Officials cited the development of the Contingency Preparedness System,  which is the program for managing exercises and after-action reports, as a  step allowing for a renewed emphasis on timeliness. Headquarters  planning staff are able to run reports using this system and regularly notify  key Coast Guard officials of overdue after-action reports. However, this  system was implemented more than 1 year ago, in August 2003, and was,  therefore, in place during the period in which we found a majority of after- action reports were late. We did not compare our results with timeliness  figures for earlier periods, and we, therefore, do not know the extent to  which the system may have helped reduce the number of reports that are  submitted late. Even if the new system has produced improvement,  however, the overall record is still not in keeping with the Coast Guard’s  60-day requirement.
Content and quality: Coast Guard guidance also contains criteria for the  information that should be included in an after-action report. These  criteria, which are consistent with standards identified in our prior work,  include listing each exercise objective and providing an assessment of  how well each objective was met. However, 18 percent of the after-action  reports we reviewed either did not provide such an objective-by-objective  assessment or identified no issues that emerged from the exercise. While  the scope of each exercise may contribute to a limited number of issues  being raised, our past reviews found that after-action reports need to  accurately capture all exercise results and lessons learned; otherwise,  agencies may not be benefiting fully from exercises in which they  participate. Similarly, officials at the Department of Defense, which like  the Coast Guard conducts a variety of exercises as part of its training, said  that if their after-action reports lack sufficient fundamental content, they  cannot be used effectively to plan exercises and make necessary revisions  to programs and protocols. Our review indicated that, in addition to the  pressure of other workload demands, two additional factors may be  contributing to limitations in report content and quality—current review  procedures and a lack of training for planners. Headquarters planning  officials noted that local commands have primary responsibility for  reviewing after-action reports and that limited criteria exist at  headquarters for evaluating the content of reports submitted by these  commands. At the field level, many planners with whom we spoke said  they were unaware of any written documentation or exercise-planning  guidance they could refer to when developing an after-action report.
The Coast Guard has cited several planned actions that may allow for  improved content and quality in after-action reports. These actions include  updating exercise management guidance and promulgating new  instructions related to preparing after-action reports and collecting  lessons learned. While these initiatives may address issues of content and  quality in after-action reports, they are currently still in the development  phase.
section_title: Conclusions
paragraphs: A successful response to a terrorist threat or incident in a seaport  environment clearly requires the effective cooperation and coordination of  numerous federal, state, local, and private entities—issues that exercises  and after-action reports are intended to identify. Complete and timely  analyses of these exercises represent an important opportunity to identify  and correct barriers to a successful response.  The Coast Guard’s inability  to consistently report on these exercises in a timely and complete manner  represents a lost opportunity to share potentially valuable information  across the organization. The Coast Guard’s existing requirements, which  include submitting these reports within 60 days and assessing how well  each objective has been met, appear reasonable but are not being  consistently met. Coast Guard officials cited a new management system as  their main effort to making reports more timely, but this system has been  in place for more than a year, and timeliness remains a problem. It is  important for Coast Guard officials to examine this situation to determine  if more needs to be done to meet the standard. The Coast Guard has  several other steps under development to address issues of report content  and completeness, and it is too early to assess the effect these actions will  have. For this set of actions, it will be important for the Coast Guard to  monitor the situation to help ensure that exercises can achieve their full  and stated purpose.
section_title: Recommendation for Executive Action
paragraphs: To help ensure that reports on terrorism-related exercises are submitted in  a timely manner that complies with all Coast Guard requirements, we are  making one recommendation, that the Commandant of the Coast Guard  review the Coast Guard’s actions for ensuring timeliness and determine if  further actions are needed.
section_title: Agency Comments
paragraphs: We provided DHS, DOJ, and DOD with a draft of this report for review and  comment. The Coast Guard generally concurred with our findings and  recommendation and did not provide any formal comments for inclusion  in the final report. DOJ and DOD also did not have any official comments.
DOD provided two technical clarifications, which we have incorporated to  ensure the accuracy of our report.
As agreed with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days from its  issue date. At that time, we will send copies of this report to the Secretary  of Homeland Security, the Commandant of the Coast Guard, appropriate  congressional committees, and other interested parties. The report will  also be available at no charge on GAO’s Web site at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (415) 904-2200 or by email at wrightsonm@gao.gov, or Steve  Caldwell, Assistant Director, at (202) 512-9610 or by email at  caldwells@gao.gov, or Steve Calvo, Assistant Director, at (206) 287-4839 or  by email at calvos@gao.gov. Other key contributors to this report were  Christine Davis, Wesley Dunn, Michele Fejfar, Lynn Gibson, Dawn Hoff,  David Hudson, Dan Klabunde, Ryan Lambert, and Stan Stenersen.
section_title: Appendix I: Briefing on Legal and Operational Issues Identified from U.S. Port Terrorism Exercises
subsections: section_title: Operational Issues Arising at Exercises
subsections: section_title: Coast Guard Management of AARs Reports Are Submitted Late
subsections: section_title: Coast Guard Management of AARs Report Content and Quality Need Attention
paragraphs: Guidance and experience stress producing AARs that fully assess training objectives and document deficiencies.
Coast Guard guidance: calls for exercises to be designed to expose weaknesses in plans and procedures and  highlight resource and training deficiencies. Minimum requirements for AARs include documentation of each  supporting objective and an assessment of how well each objective was met.
Past GAO work: when AARs do not accurately capture exercise results and lessons learned, agencies may  not be benefiting fully from exercises in which they participate.
DOD perspective: DOD officials said AARs that did not provide fundamental content cannot be used  effectively to plan exercises and make necessary revisions to programs and protocols.  They also noted that  new operational missions may require an additional emphasis on exercise planning and after-action reporting.
Assessment of exercises may not be sufficient: 18 percent of AARs we reviewed identified no issues or did not  provide adequate assessment of training objectives.
Review procedures and training for planners may be insufficient in this area.
Headquarters planning officials noted that the primary review of all AARs resides solely at the local command  level. Although all submitted AARs are reviewed “for general approval” by headquarters officials, they said  that this review uses limited criteria (grounds for rejection include use of inappropriate language or  participants' names).
Many Coast Guard field personnel we interviewed said they were unaware of any written documentation or  exercise planning guidance they could refer to when developing an AAR.
section_title: Coast Guard Management of AARs Internal Coast Guard Efforts to Address Issues
paragraphs: Some efforts to address timeliness are under way, but effects to date are limited.
Coast Guard officials said the Contingency Preparedness System (CPS), the program for managing exercises  and AARs, has allowed for a renewed emphasis on report timeliness. Headquarters planning staff currently  use this system to notify each area of overdue AARs. However, CPS has been in place since August 2003,  and timeliness remains a concern.
Officials have also discussed the possibility of reducing the AAR submission deadline (to as few as 15 days),  but efforts are still ongoing due to “pushback from the field.” They also said that the formal Coast Guard  training courses emphasize that AAR development be incorporated into the planning process and exercise  timeline.
Senior exercise management officials said they are also updating an instruction related to collecting AARs  and lessons learned. They expect it to be promulgated to the field in 1-6 months.
Officials noted the following efforts to improve content and quality of AARs.
Formal training courses that encourage documenting exercise information quickly to capture relevant  information and lessons learned before recall is diminished or competing priorities take over.
Updated instruction on AARs and lessons learned collection (currently in development).
Increased functionality of CPS has been proposed, which may offer additional incentives for planners to utilize  the system.
section_title: Summary of Findings
subsections: section_title: Objective 1
paragraphs: Key elements of the national response framework are evolving.
Release of National Incident Management System and draft National Response Plan.
Transitional period for agencies to revise their plans once the final NRP is released, agencies will  have up to 180 days to revise their plans to align with the NRP.
section_title: Objective 2
paragraphs: Few legal issues surfaced in port exercises or after-action reports.
None of these issues were statutory problems according to exercise participants and agency officials.
section_title: Objective 3
paragraphs: Exercises and after-action reports identified operational issues to varying degrees.
Key issues included: communication, incident command, and resource coordination concerns.
section_title: Objective 4
paragraphs: Many after-action reports are not submitted timely, and content and quality of some does not meet   Actions taken by the Coast Guard to address these problems have had limited effect thus far.
section_title: Appendix II: Objectives, Scope, and Methodology
paragraphs: The objectives of this report were to (1) describe the emerging framework  under which the federal government coordinates with state and local  entities to address a terrorist incident in a U.S. port; (2) identify the issues,  if any, regarding federal agencies’ legal authority that have emerged from  port security exercises and what statutory actions might address them;   (3) describe the types of operational issues being identified through these  exercises; and (4) identify any management issues related to Coast Guard- developed after-action reports. To address these objectives, we reviewed  relevant legislation, regulations, directives and plans, analyzed agency  operational guidance and Coast Guard after-action reports (AARs),  interviewed a variety of federal officials, and observed several port  security exercises.
To identify the emerging framework to address a terrorist incident in a  U.S. port, we reviewed relevant statutes such as the Homeland Security  Act of 2002 and the Maritime Transportation Security Act of 2002 and  implementing maritime regulations at 33 CFR, parts 101 to 106. We also  reviewed Homeland Security Presidential Directive/HSPD- 5 and  Presidential Decision Directive 39. Operational plans that were included in  our analysis included the Initial National Response Plan, the Interagency  Domestic Terrorism Concept of Operations plan (CONPLAN), Interim  Federal Response Plan, and the National Response Plan “Final Draft.” We  also reviewed agency guidance related to exercise planning and evaluation  such as the Coast Guard Exercise Planning Manual and Contingency  Preparedness Planning Manual, as well as the Department of Homeland  Security/ Office of Disaster Preparedness’ Exercise and Evaluation  Program. Findings were supplemented with interviews of key officials in  federal agencies, including the Coast Guard (CG), the Department of  Homeland Security (DHS), Department of Defense (DOD), Department of  Justice (DOJ), and related federal maritime entities such as Project  Seahawk.
To provide a framework for evaluating agencies’ legal authority in  responding to a terrorist incident in a U.S. port, we adopted a case study  methodology because it afforded a factual context for the emergence of  legal issues that could confront agencies in the exercise of their authority.  Our efforts included attending four U.S. port based terrorism exercises  (Los Angeles, Calif.; Hampton Roads, Va.; Charleston, S.C.; Philadelphia,  Pa.), reviewing CG AARs for fiscal year 2004, and conducting in-person  and telephone interviews with DHS, CG, DOJ, DOD, and Project Seahawk.  The port exercises we selected to visit were geographically diverse and  each was conducted in either August or September of fiscal year 2004.  Additional criteria for exercise selection included the strategic importance  of the port (as defined by the Maritime Administration), the variety of  terrorism scenarios to be exercised, and the federal, state, and local  players involved. The AARs we reviewed were based on a list of all fiscal  year 2004 exercises provided to us by the CG. We focused on any  contingency that included terrorism and then requested AARs for those  completed exercises from the CG. According to CG guidance, AARs are  required to be submitted within 60 days of exercise completion. To  ascertain compliance with this guidance, CG personnel provided us with  the dates that AARs for terrorism-related exercises were received at  headquarters. We used this information, in conjunction with the exercise  start and stop dates, to determine which reports were on time, which were  late, and the average time late reports were submitted beyond the 60-day  requirement.
While issues of a legal nature did surface during our observation of  exercises and analysis of AARs, exercise participants and agency officials  did not recommend statutory changes for these issues. We generally relied  upon the agency’s position as to whether legislation was necessary and did  not independently assess the need for legislation by auditing the specific  issues identified in the exercises.
To identify operational issues that occurred during port terrorism  exercises, we relied extensively on perspectives gained through our  observations at the four port terrorism exercises as well as a  comprehensive review of the available AARs for operational issues based  on criteria we developed. In order to determine the frequency of various  operational issues identified in the CG’s AARs, we noted instances that  each subcategory within the major category appeared. These categories  and subcategories were chosen through exercise observation and an initial  review of available AARs by two independent analysts. This allowed us to  identify operational issues that were consistent across the terrorism  exercises. We used the following major categories and subcategories  (which appear in parentheses)
Communication (communication interoperability issues, communication  policy or protocols between or within agencies, information sharing  between agencies),   Command and Control/ Incident Command Structure (NIMS/ICS training,  UC/IC information flow),    Unclear Decision Making/ Jurisdictional Knowledge (unclear decision  making authority, unclear lead authority, unclear authorities/jurisdictions  of other agencies), and   Resource Coordination/ Capabilities (response capabilities, response  coordination/joint tactics).
To analyze the reports, two GAO analysts independently reviewed each  report and coded operational issues based on the above subcategories.  The results of each analysis were then compared and any discrepancies  were resolved. Overall percentages for the major categories were  determined based on whether any of the issues were identified under the  respective subcategories. The maximum number of observations for any  major category was equal to one, regardless of the number of times a  subcategory was recorded.
To identify management concerns regarding the CG’s AARs, we reviewed  our previous studies on this issue as well as CG and DHS issued guidance  on exercise management, such as the Coast Guard’s Exercise Planning  Manual and Contingency Preparedness Planning Manual Volume III. Our  analysis also included in-person interviews with CG exercise management  officials from headquarters and CG planners in the field to gain additional  information on how terrorism exercises are planned and evaluated as well  as how lessons learned are cataloged and disseminated. To ascertain the  effect of untimely CG AARs (CG AARs are required to be completed within  60 days of exercise completion), we also interviewed exercise  management experts from DOD. We conducted a content analysis of the  available AARs to determine the weaknesses in the reports and where  deviations from CG protocol were taking place.
We conducted our work from June to December 2004 in accordance with  generally accepted government auditing standards.