Maritime Security: Better Planning Needed to Help Ensure an Effective Port Security Assessment Program
Published: 2004-09-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Created in the wake of the September 11, 2001, terrorist attacks, the Port Security Assessment Program was designed to evaluate security at the nation's 55 most economically and militarily strategic ports. Implemented by the U.S. Coast Guard, an agency of the Department of Homeland Security, the program focuses on identifying vulnerabilities, suggesting approaches to minimize them, and making the information available to those responsible for developing and implementing portwide security plans. The program has been under way for more than 2 years and has undergone several sets of changes, including the addition of a geographic information system (GIS). GAO was asked to discuss why and how the program changed and assess the Coast Guard's approach for implementing the program in its current form.
section_title: What GAO Found
paragraphs: Changes in the Port Security Assessment Program reflect attempts to deal with two main developments since the program's inception: evolving assessment needs at the ports and missteps in how the initial assessments were carried out. The program was designed as a comprehensive assessment of each port and its critical assets, such as passenger terminals, factories, cargo facilities, and bridges. However, the need for comprehensive assessments was diminished when many owners and operators of these critical assets began conducting their own assessments to comply with new regulatory requirements or apply for security grants. The program's assessments also proved more expensive than expected, and a GAO review conducted at the time found shortcomings in their quality and usefulness. The current program's assessments are more targeted in scope and nature, including the opportunity for local Coast Guard officials to request reviews of specific assets they do not know enough about. To help local authorities with security planning and response, the Coast Guard decided to incorporate a GIS. A GIS is a computer mapping system designed to have many information "layers" that can be easily updated and retrieved. The Coast Guard expects to complete the assessments at the 55 ports by February 2005, but no timeline exists for making the GIS component operational. Although the revised program holds promise, the implementation approach is at increased risk because the Coast Guard is not taking sufficient steps in the planning process. Contrary to best practices for technology systems development, the GIS is being developed without sufficient up-front work to identify how the system will be expected to perform. Both the GIS component and the program as a whole also lack a project plan detailing tasks, schedules, and costs. In other federal agencies, GAO has identified similar projects that failed when such steps were not followed. The initial response of local Coast Guard officials to the new, targeted assessments is generally positive. However, the assessments could be of greater benefit if functional requirements for the GIS were more clearly defined, so the Coast Guard could use the assessments to address gaps in security knowledge.

section_title: Letter
paragraphs: Three years after the terrorist attacks of September 11, 2001, securing the  nation’s ports continues to be a major concern. Ports and associated  waterways are particularly vulnerable because of their size, accessibility,  and the many sites and facilities that could be targeted. Gathering  information about these vulnerabilities is an essential step for developing  deterrents and responding effectively if an incident occurs. One such  effort is the Port Security Assessment Program, which is designed to  assess port vulnerabilities and security measures in the nation’s 55 most  economically and militarily strategic ports. Since November 2001, nearly  $70 million in appropriated funds has been and continues to be spent on  this project, which is administered by the United States Coast Guard, an  agency of the Department of Homeland Security (DHS).
This program has changed considerably since its inception in the days  immediately following the September 11 attacks. Among these changes,  the Coast Guard has added a new feature—a geographic information  system (GIS). A GIS is a computer mapping system with many information  “layers” that can be quickly retrieved and displayed and easily updated. If,  for example, a port received notice of potential threats to chemical plants  in the area, a well-designed GIS could identify locations of these plants,  provide a variety of information about them, and pinpoint available  surveillance and response resources for Coast Guard personnel and others  involved in port security. This tool is intended to provide up-to-date,  readily accessible information to help develop security plans and respond  to specific threats or incidents. However, the experience of other federal  agencies has shown that developing an information technology system,  such as a GIS, that clearly meets users’ needs can be difficult.
Given the role the program is expected to play in enhancing the Coast  Guard’s ability to provide security at our nation’s ports, this report   (1) discusses why and how the program has changed over time and   (2) assesses the Coast Guard’s approach for implementing the program as  it is currently configured.
To address the first objective, we reviewed Coast Guard documents and  spoke with officials at Coast Guard headquarters responsible for  implementing the program. We also visited ports that had been assessed.  At the ports, we interviewed local Coast Guard personnel as well as  numerous stakeholders to determine how the assessment process was  carried out. For part of the history of the program, we also relied on our  previous work. To address our second objective, we interviewed Coast  Guard officials, including the GIS Program Manager to assess progress on  the GIS development effort. We also reviewed Coast Guard documents,  including its systems acquisition guidance, and documentation of the  Coast Guard’s efforts to modify its port security GIS. Finally, we reviewed  information and documentation related to GIS applications and identified  standards and best practices for information systems acquisition and  development to determine best practices for managing such a project. Our  work, which was conducted from June 2003 through August 2004, was  done in accordance with generally accepted government auditing  standards.
subsections: section_title: Background
paragraphs: Creating effective security in the nation’s ports in the post-September 11  world is a challenging task. Ports present attractive targets for terrorists:  they are sprawling, easily accessible by water and land, close to crowded  metropolitan areas, and interwoven with complex transportation  networks. Besides terminals where goods bound for import or export are  unloaded or loaded onto vessels, ports also contain other facilities critical  to the nation’s economy, such as refineries, factories, and power plants.  These many facilities, along with the ships and barges that ply port  waterways, can be vulnerable on many fronts. For example, container  terminals, where containers are transferred between ships and railroad  cars or trucks, need ways to screen vehicles and routinely check cargo for  evidence of tampering. At factories and other facilities where hazardous  materials are present, safeguards must be in place to prevent unauthorized  persons from gaining access. Similarly, vessels ranging from oil tankers to  tugboats need effective access control over critical operating areas, such  as engine and control rooms.
The framework for the nation’s collective response to this challenge is  now found in the Maritime Transportation Security Act (MTSA), passed by  the Congress in November 2002. MTSA’s implementing regulations require  owners and operators of facilities and vessels to conduct assessments that  will identify their security vulnerabilities and to develop security plans to  mitigate these vulnerabilities. Under these regulations, these plans are to  include such items as measures for access control, responses to security  threats, and drills and exercises to train staff and test the plan. MTSA was  enacted after the Coast Guard initially began developing the Port Security  Assessment Program in the wake of the September 11 attacks.
Some basic information about geographic information systems, or GIS,  may be helpful in understanding this component of the Port Security  Assessment Program. A GIS can be thought of as a sort of electronic map,  but with many more capabilities than traditional paper mapping. For  example, paper maps can provide only a static snapshot of selected  entities and their locations and cannot be easily updated or changed. By  contrast, information in a GIS can be easily and continually updated. In  addition, because a GIS stores information on separate “layers” related to  such things as roads or buildings, users can combine data layers at will,  providing the capability to quickly create and view maps for specific  purposes any time they are needed. Data layers in a GIS can be extremely  varied. Typical types include the following:    Layers describing location, ownership, and other information about real  property (called cadastral data).
Layers that have the characteristics of a map and image qualities of a  photograph (called digital orthoimagery).
Layers describing water features such as lakes, ponds, streams and rivers,  canals, oceans, and coastlines (called hydrographical data).
For the Coast Guard, potential GIS layers could include transportation— describing anchorages, bridges, and roadways; utilities—including power  plants, power lines, and substations; and emergency response—including  police and fire stations, and hospitals. Figure 1 illustrates, in a simplified  way, this concept of layers and how they can be integrated.
The database capabilities of a GIS allow many other kinds of information  to be embedded on these data layers as well, so that the information is  easily available. For example, a GIS allows the user to know not only the  location of a building relative to other buildings or roads, but can also  provide information such as the building’s owner, when the building was  built, the building’s contents, and its dimensions and height. This ability to  create maps on demand for specific purposes, with additional information  at the ready, surpasses what can be done with traditional mapping  approaches.
One illustration of a GIS’s usefulness came in connection with efforts to  recover debris from the space shuttle Columbia when it was lost in re- entering the earth’s atmosphere on February 1, 2003. Debris from the  shuttle was spread over at least 41 counties in Texas and Louisiana. In  Texas, a state-operated GIS provided authorities with precise maps and  search grids to guide reconnaissance and collection crews in the field.  Officials in charge of the effort used maps of debris fields, combined with  GIS data about the physical terrain, to carefully track the pieces of debris  found. Figure 2 is a map, created from debris data entered into the GIS,  showing the general west-to-east track of debris data across several east  Texas counties and the outer boundaries of the area in which debris was  found.
section_title: Assessment Program Has Been Extensively Revised
paragraphs: The Coast Guard has made significant revisions to adapt the Port Security  Assessment Program to the increasing amount of security evaluations  performed by port stakeholders and to address shortcomings in the  program’s initial implementation. The Coast Guard initially set out to use  the program as an assessment of security conditions at 55 ports. The Coast  Guard and the contractor it hired to develop the assessment approach and  conduct the assessments started the first assessments in August 2002,  when other assessment efforts were also under way. Port stakeholders  around the country had begun or already completed their own  assessments of their facilities or vessels in order to identify security  vulnerabilities of their assets or obtain federal assistance in strengthening  their security. Even more security information was to become available as  new regulatory requirements were implemented in 2003 requiring security  assessments to be performed by the owners or operators of facilities and  vessels operating in the nation’s ports. This changing security environment  and higher-than-expected costs to complete the contractor’s initial  assessments prompted the Coast Guard to revise the scope of the  contractor’s assessments. Our examination of the contractor’s initial  assessments identified additional shortcomings in the quality of the work  and the assessment approach. In response, the Coast Guard temporarily  postponed all assessment work to make further revisions, both to take  advantage of the other sources of assessment information and make the  assessments more useful in port security planning efforts. The revised  program (1) added a GIS as a new feature and (2) tailored security  assessments for particular purposes, such as synthesizing existing  assessments or assessing certain infrastructure at the direction of local  Coast Guard personnel. The assessments are to be completed by February  2005—but the Coast Guard is still developing its GIS and is uncertain as to  when the GIS will be ready for use.
subsections: section_title: Program Initially Focused on Assessing Post-9/11 Vulnerabilities at Key Ports
paragraphs: The Coast Guard began the Port Security Assessment Program to assess  the vulnerabilities of the nation’s most strategic commercial and military  ports in the aftermath of the September 11, 2001, terrorist attacks. (See fig.  3 for a timeline of the program.) To identify which ports were most  strategic, the Coast Guard considered such factors as cargo volume,  import/export cargo value, volume of passenger traffic on ferries or cruise  ships, population density around the port, the presence of critical  infrastructure or key assets, the presence of military forces or bases, and  whether the port was designated to support major military deployments.  From this analysis, 55 ports out of 361 ports were chosen to be the first to  receive port security assessments.
In April 2002, the Coast Guard selected a contractor to perform the  assessments. Under this arrangement, the contractor was responsible for  developing an approach (which the Coast Guard calls “Version 1”) to  assess vulnerabilities of port assets and systems such as cargo facilities,  manufacturing facilities, passenger terminals, power generation and  fuelling facilities, as well as other infrastructure such as public access  areas and bridges. The assessment was to identify the relationships  between selected assets to port systems, identify the vulnerabilities of  those assets to terrorist attacks, and recommend actions to mitigate the  vulnerabilities. With oversight from the Coast Guard, the contractor had  primary responsibility for conducting key activities of each assessment,  such as identifying which assets should be assessed, collecting data from  stakeholders, making on-site visits, and analyzing the data collected. The  final product was to be a comprehensive written report of the findings  identified during the assessment. Primary customers for this work were  the local Coast Guard Captain of the Port and port stakeholders serving on  Area Maritime Security Committees, who could use it in such security  planning efforts as the development of an Area Maritime Security Plan.
The first assessments began in August 2002; the Coast Guard’s goal was to  complete them at all 55 ports by the end of 2004. To further refine the  approach before assessing “megaports” such as New York/New Jersey or  Los Angeles/Long Beach, as well as to give the program a chance to build  additional assessment teams to perform the work, the Coast Guard  decided to try out the approach at medium-sized ports first such as San  Diego and Boston. Under the time frame the Coast Guard adopted,  officials expected to conduct assessments of 8 ports in 2002, 18 in 2003,  and 24 in 2004.
section_title: Need to Incorporate Work Done by Others and Correct Shortcomings in Contractor’s Assessments Led to Revisions in the Approach
paragraphs: Several actions taken by port stakeholders led to substantial changes in  the approach. One of these developments was that many port stakeholders  were starting or completing assessments on their own. Stakeholders, such  as port authorities, and owners and operators of facilities and vessels  began conducting assessments in order to identify security vulnerabilities  of their assets or to meet application requirements for federal grants. In  some cases, initial assessments were performed shortly after the  September 11, 2001, terrorist attacks and were followed by more  comprehensive assessments conducted either on their own or by  contractors. For example, port stakeholders such as chemical producers  that were members of certain industry or trade organizations were  required to complete assessments of their facilities using approved  assessment methodologies as a condition of their membership in the  organization. Beginning in September 2002, the Coast Guard also issued a  series of suggested guidelines for port stakeholders to use in conducting  security assessments and developing security plans to address any  identified vulnerabilities.
In addition to the assessment activities that many stakeholders voluntarily  undertook after the terrorist attacks, more maritime security information  became increasingly available as the Maritime Transportation Security Act  began to be implemented. Enacted in November 2002, MTSA mandated  major changes in the nation’s approach to maritime security and called for  a comprehensive framework that includes planning, personnel security,  and careful monitoring of vessels and cargo. The regulations implementing  MTSA required owners or operators of specific facilities and vessels in the  nation’s ports to conduct assessments and develop plans to address  vulnerabilities. These security assessments and plans were to be reviewed  and approved by the Coast Guard prior to July 1, 2004. As a result,  facilities and vessels that had not already completed a security assessment  were now required to do so, thereby increasing the amount of assessment  information available from port stakeholders at the 55 ports as July 1,  2004, drew nearer.
Coupled with the changes in the amount of information to be generated by  others, high costs for the first set of assessments prompted the Coast  Guard to begin reassessing the Version 1 approach for conducting the  assessments. According to the Coast Guard, assessments for the first   8 ports cost nearly three times more than was originally expected,  exceeding $1 million per port. To address this issue, the Coast Guard made  changes in the assessment approach, including greater emphasis on  discussions early on in the assessment process with local Coast Guard  Captains of the Port in order to better focus on the facilities and  infrastructure needing to be assessed and the adoption of a standardized  report outline and format to reduce redundancy. The Coast Guard  decided to pilot-test this new approach, which the Coast Guard now calls  “Version 2,” at two ports in the summer of 2003.
As this new approach was being readied, our own review of the  contractor’s assessments disclosed additional shortcomings. In a  September 2003 testimony before the Senate Committee on Commerce,  Science, and Transportation, we expressed concern about how the  assessment program was being implemented. In talking with some port  stakeholders who participated in the assessment, we found that many of  them saw little usefulness in the assessments beyond what they already  knew about their vulnerabilities from previously completed assessments.  Some key port stakeholders declined to participate in the assessment after  receiving lengthy questionnaires from the contractor asking for  information stakeholders considered proprietary. Port stakeholders also  said they had not been given the opportunity to review or comment on the  draft assessment report, which contained errors and inaccuracies. Finally,  the contractor was moving to use the Version 2 approach in the next set of  assessments before the lessons learned from the pilot tests could be  identified and incorporated into the assessment approach.
We shared our findings with Coast Guard officials and suggested that the  assessment approach be further revised. In addition to giving the Captains  of the Port and Coast Guard personnel a larger role in identifying the  critical assets to be assessed, we suggested that the Coast Guard reduce  duplication and lessen the burden on stakeholders by doing more to take  into account already-completed assessments of facilities and assets. The  Coast Guard agreed and postponed conducting more assessments until  additional changes to address these deficiencies were made.
section_title: Subsequent Revisions Incorporated the Use of a GIS and More Focused Assessments
paragraphs: While considering what changes needed to be made to the assessment  program, the Coast Guard also determined that it was essential to provide  local Coast Guard officials and certain members of the local Area Maritime  Security Committee a means to retrieve maritime security information and  display it for planning and response purposes at the ports. Although a  significant amount of security information was now available, it was kept  in disparate locations and was not readily available. With the regulations  implementing MTSA requiring Captains of the Port and Area Maritime  Security Committees to develop portwide Area Maritime Security Plans,  access to the available security information became increasingly important  in order for them to carry out this responsibility and improve the  protection of the marine transportation system.
To provide local Coast Guard officials and certain members of the local  Area Maritime Security Committee access to this information, the Coast  Guard decided to incorporate a GIS as a new feature in the assessment  program. At the local port level, the GIS would integrate the security  information into a single electronic database that would allow the  information to be retrieved and displayed within the context of a  particular port area. Whereas previous assessment results were compiled  into a published report that would characterize the port’s security posture  at a single point in time, GIS has the capability of being updated as new  information becomes available. GIS also provides a tool for visually  depicting the port and for retrieving security or assessment information as  needed in the development or revision of Area Maritime Security Plans.  The Coast Guard believes this will benefit the Captains of the Port and the  Area Maritime Security Committees to better visualize the port and  enhance their ability to develop security plans as well as respond to a  security incident, should one occur.
In addition to the GIS component, the revised program has three other  components, all related to assessments. The Coast Guard revised the  assessment approach so that it would provide more specialized  information about port security. The approach, known as Version 3, has  three different types of assessments that collectively are aimed at  providing both a synthesis of what is already known about security at a  port and studies of specific topics or infrastructure that have not been  fully assessed. When completed, these assessments will provide the core  security information to populate the GIS. These assessment components  are as follows:    Assessment of Assessments—An identification and inventory of  completed security assessments of port assets and critical infrastructure  within a port. This inventory is designed to help the assessment team  minimize the possibility of needlessly duplicating previously completed  assessments as well as to provide the Captain of the Port and the Area  Maritime Security Committee with greater awareness of existing security  information.
Terrorist Operations Assessment—An assessment utilizing the expertise  of contractors comprised of former Navy Special Operations personnel to  provide an outsider perspective on the ports’ vulnerabilities to a terrorist  attack. This assessment is to evaluate potential terrorist targets within the  ports and identify likely attack scenarios for the Captain of the Port and  Area Maritime Security Committee to consider addressing in the Area  Maritime Security Plan.
Special Assessment—Assessment of specific port assets, infrastructure, or  operations that are critical to the port but have not been previously  assessed from a maritime perspective. Performed at the request of the  Captain of the Port and the Area Maritime Security Committee, this  assessment is to provide vulnerability, impact, and countermeasure  information on those assets, infrastructure, or operations. Examples  include blast impact assessments of commercial vessels, plume dispersion  assessments of an attack on vessel or facility with hazardous materials,  and security assessments of underwater tunnels.
The Coast Guard has a more definite schedule for completing the  assessments than for completing the GIS. The Coast Guard resumed  assessments in March 2004 using the Version 3 approach and plans to  complete assessments at the remaining ports by February 2005. For the  GIS component, the Coast Guard plans to use its own GIS. Until this  system is operational for port security, the Coast Guard plans to lease a  commercial GIS that will enable Coast Guard staff to familiarize  themselves with how a GIS works and identify their specific system needs  or requirements so the Coast Guard’s GIS can be customized accordingly.  Project officials chose a commercial-off-the-shelf software application,  iMap, that provides the Coast Guard access to over 800 layers of data  containing information related to the nation’s ports. Because the Coast  Guard’s GIS is still in development for port security, when the GIS  component will be made operational and available to all assessed ports is  yet to be determined.
section_title: Absence of Key Management Elements Places Program’s Potential to Enhance Port Security at Higher Risk
paragraphs: The Coast Guard’s revised approach appears to provide a useful planning  and response tool for port security, but the implementation of the  assessment program is at higher risk because of two major problems.  First, the centerpiece of the new approach, the GIS component, is being  developed without several key project management steps that are critical  to success in such projects. Not following these steps increases the risk  that the data collected will not provide port security officials with the  information they need to adequately assess, identify, and mitigate security  risks. Second, for the GIS component and the program as a whole, the  Coast Guard lacks a strategy that clearly defines how the program will be  managed, how much it will cost, or what activities will continue over the  longer term. Lack of a strategy increases the risk of cost overruns, missed  deadlines, and a less-than-effective program. At the same time the Coast  Guard is facing these problems, it is also conducting security assessments  at individual ports using the revised approach, and for this part of the  program, the results to date appear more favorable. Early indications from  local Coast Guard officials at the ports where the new assessments have  been performed are that these assessments are of some usefulness in  current security planning activities. However, not resolving the broader  planning and management issues could also affect the potential value of  these assessments to fill in any remaining gaps in the Coast Guard’s  awareness of the security posture in the ports.
subsections: section_title: GIS Component Seen as Having Great Potential, but Coast Guard Is Not Following Established Project Management Practices
paragraphs: The Captains of the Port and other Coast Guard officials we talked with  were in agreement in their belief that a GIS with security assessment  information would greatly facilitate their security planning and response  efforts. They provided such examples as the following, based on their  understanding of the tools that would be available with the GIS:    For planning efforts, the visual nature of the GIS would greatly enhance  the Captain of the Port’s and Area Maritime Security Committee’s  understanding of the connections between port facilities, assets, and  infrastructure that would otherwise not be possible through paper reports.
For incident response efforts, the capability of GIS to store and retrieve  security information such as plans and assessments of particular assets  within the port would quicken response times as the information can be  immediately located and viewed.
A useful, well-designed system does appear to carry great promise. For  example, if Coast Guard personnel were alerted that a particular port may  be targeted and that warehouses containing shipping containers were at  risk, officials could quickly create a map showing the location and  contents of the warehouses, ingress points located near the warehouses,  and depth of the nearby waterways throughout the port. Using this  information, security officials could assess the relative risk to each  warehouse, prioritize actions based on the risk level, and act almost  immediately to secure the most vulnerable locations.
However, developing a useful GIS is a significant and complex challenge.  One reason is that every port has its own unique mix of geographic  characteristics and operations that must be accurately captured. For  example, one port may be located along a stretch of river while another  may sit next to the open ocean, one port may have a high volume of cruise  ship traffic while another may have a high concentration of chemical and  petroleum facilities. These different characteristics will require a GIS that  is flexible enough to be of use in a variety of settings. Another reason the  GIS can be challenging is that some security-related situations, such as  potential terrorist activities, involve a great deal of unpredictability and  the kinds of information and analyses needed to address such uncertainty  are difficult to anticipate.
This complexity places a premium on proper planning. Over the years, we  have analyzed information technology systems across a broad range of  federal programs and agencies, and these analyses have repeatedly shown  that without adequate planning, the risks increase for cost overruns,  schedule slippages, and systems that are not effective or usable. For  example, the Bureau of Land Management (BLM) spent more than   $67 million on a system that was never deployed. When the system was  tested prior to deployment, it was found not to meet users’ requirements  because it did not support BLM’s business activities, was too complex, and  significantly impeded worker productivity. We found this system failed  because it was developed without a clear understanding of requirements  and without a credible project schedule with reliable milestones. In  another example, the Centers for Medicare and Medicaid Services had  similar problems that led to its planned Medicare Transaction System  being cancelled—the project did not have fully defined and agreed-to  requirements and had a flawed project schedule.
These types of problems make it prudent to ensure that planning of GIS  applications is adequate. Coast Guard officials indicated that they viewed  the development of the port security GIS database as an add-on to existing  Coast Guard information systems, not as a new database or information  system. Within this context, however, it is still important to ensure that the  steps being taken are likely to produce a satisfactory result. In that light,  we assessed the Coast Guard’s development efforts using established best  practices in the industry for developing information technology systems,  including those created by the Institute of Electrical and Electronics  Engineers/Electronic Industries Alliance. The Coast Guard’s current  efforts do not apply these criteria in two key ways—defining what the GIS  should do and establishing sufficient plans to ensure that the requirements  can be successfully realized. That is, successful implementation of the  Coast Guard’s port security GIS is at higher risk because the Coast Guard  has not used established project management practices, including defining  requirements and developing a project schedule, to oversee and guide the  program.
subsections: section_title: Actions to Develop GIS Requirements Do Not Meet Standards
paragraphs: One aspect of developing any information technology system such as a  GIS involves establishing and maintaining a common and unambiguous  definition of functional requirements among the project team, system  users, and software developer. These requirements define what the system  will be expected to do for its users once it is developed and implemented.  For example, one requirement could be to ensure that the system can link  together specified types of geospatial data to provide the user with  sufficient information. Another requirement could be to ensure that the  users would be provided the capability of printing paper maps and other  information found in the GIS. A third could be that the GIS be available to  its users 24 hours a day, 7 days a week. Requirements such as these could  be important in ensuring that the system will deliver what users need. It is  critical that functional requirements are carefully defined and that they  flow directly from how the organization’s day-to-day operations are or will  be carried out to meet mission needs. Improperly defined or incomplete  requirements have been commonly identified as a root cause for why  systems fail or do not meet their cost, schedule, or performance goals.  Without adequately defined requirements, significant risk exists that a  system will need extensive and costly changes before it will meet the  organization’s needs.
The Coast Guard’s actions to develop GIS requirements are not being  carried out using established practices. The Coast Guard’s approach for  addressing these requirements takes three main forms:    First, the Coast Guard is using the assessments being conducted at the   55 ports to identify requirements for the GIS it is developing.
Second, the Coast Guard is using feedback from the experiences of local  officials with the commercial-off-the-shelf software application currently  in use to help determine what requirements should be included.
Finally, the contractor supporting the interim GIS has been tasked to  identify the GIS data layers most frequently used by the Coast Guard.
However, these actions fall short of meeting best practices. First, there are  indications that requirements identified during the assessment visits did  not necessarily include functional requirements. Second, although tasks to  identify the data layers accessed by the Coast Guard using its interim GIS  solution could be used to identify requirements for the port security GIS,  these tasks have not yet been completed and there is no estimate as to  when the information will be available.
According to Coast Guard officials, the Coast Guard intends to use an  existing information system instead of building a new GIS database or  information system that is exclusive to port security. However, while the  Coast Guard is not developing a new system, greater planning efforts  appear paramount. To the extent that the Coast Guard and other users  believe they need to add new kinds of data that do not currently exist in  the system, both system users and developers need to agree on how to  define and capture this information so that it can be of maximum use. In  addition, if the Coast Guard decides to take a more limited approach,  adding few, if any, new functional requirements, it runs the risk that the  system will be of only partial use. Rather than taking advantage of the  powerful planning and analysis capabilities that a robust geographic  information system could make available, the more limited version could  only be used to develop static maps of ports and their assets. Without  effectively identifying and documenting the requirements for the new  potential functions and data associated with the port security portion of its  GIS, the Coast Guard faces the risk that the GIS will not provide port  security officials with the functionality and information they need to  adequately assess, identify, and mitigate security risks.
section_title: Coast Guard GIS Planning Does Not Meet Established Best Practices
paragraphs: Information technology project management principles and industry best  practices emphasize that a project management plan is needed to define  the technical and managerial processes necessary to satisfy project  requirements. The plan should include, among other activities,    developing a work breakdown structure with a schedule for all of the  tasks to be performed;  identifying and addressing project risks, and implementing a security policy.
The planning document identified by the Coast Guard does not meet these  standards. According to the Port Security Assessment program manager,  the Coast Guard considers the project’s Concept of Operations to be its  project plan. However, the Concept of Operations, does not include  important elements required in a project plan. For example:    Tasks and schedules: The Concept of Operations identifies seven Port  Security Assessment Program objectives, one of which is the use of a GIS,  but does not identify any of the tasks or a schedule for carrying them out.  It also provides a list of eight high-level activities that need to be  completed during the project, but again it lists no associated  implementation tasks and schedule, although it estimates that port  security assessments will be completed by December 2004. Since the  document was written in February 2004, the assessment completion date  has already been postponed by 2 months, and the project manager is  unsure if the interim GIS contract will need to be renewed next spring  because he is not sure when the Coast Guard’s own port security GIS will  be completed and ready for implementation.
Project risk: The Concept of Operations does not address project risks.  As a result of not identifying potential risks, the project has encountered  unexpected problems. For example, two of the eight high-level activities  identified in the Concept of Operations, scheduled to be completed in  April and July 2004, encountered unexpected problems that caused delays  and could hinder their eventual completion.
Security Policy and Project costs: The Concept of Operations does not  address security policy and provides no plan for estimating project costs.  For example, we asked program officials to provide documented cost  information associated with the GIS component, and while we received  some information, it was not sufficient to provide a clear indication of how  much the GIS component would likely cost.
Creating a plan that meets these requirements is essential to ensuring that  the port security assessment GIS project can be successfully completed in  the estimated timeframes with the resources that are available. The Coast  Guard has already encountered problems caused by lack of a reliable  project schedule and risk assessment. According to Coast Guard Officials,  the Coast Guard is adding to an existing system rather than building a new  one. Adding to an existing system, however, does not obviate the need for  careful planning. Until the Coast Guard develops a project management  plan that includes a schedule and milestones, it is at increased risk that the  GIS component of its port security assessment program could be  inadequately managed, resulting in schedule slippages and inaccurate  costs estimates. In addition, without identifying and mitigating risks and  security concerns, the project could encounter unexpected issues that  would need to be addressed, resulting in additional schedule and cost  problems.
section_title: Incomplete GIS Planning Places Usefulness of New Assessments at Greater Risk
paragraphs: The Coast Guard has proceeded to carry out the revised assessments of  individual ports with generally favorable results. The Coast Guard  resumed its assessment program using the Version 3 assessment  components in March 2004 and as of August 1, 2004, had completed on-site  assessments of 12 additional ports in 6 Captain of the Port zones. To  provide an indication of the usefulness of these assessments, we spoke  with the local Captain of the Port or other Coast Guard officials that  participated in the assessment process at each of these zones. In general,  all agreed that the assessments were of some usefulness. Two said that the  assessments provided substantially new information that they did not  previously have or consider. The other four found the completed  assessment results useful by bringing an outside perspective to look at the  port. They said the assessments were helpful in validating their previously  completed assessments or the current awareness of the security posture  within their ports.
The value of these assessments could be enhanced, we believe, if the  Coast Guard addressed the key management practices we have already  discussed in its approach for developing its GIS. By themselves, the  current assessments have value to local Coast Guard officials mostly in  supplementing or validating their knowledge.  However, when used with  the GIS, these assessments also have potential value in helping the  officials “close the loop” on information they may lack. The three  assessment components involve mainly data gathering and analysis, the  results of which are to be fed into the GIS. Without the GIS to integrate  and organize information gathered from these and other sources, those  responsible for planning security cannot as easily identify the  vulnerabilities in their ports and gaps in their awareness of the security  posture within their ports that need to be addressed. At all of the six ports,  the Captain of the Ports or other Coast Guard officials said the value of the  three assessment components would be enhanced if used in conjunction  with a GIS that would be better able to visually display the entire security  posture of the port rather than having to review individual hard copy  assessment reports as they are now published. However, the functional  requirements need to be first defined in order to effectively integrate these  assessment components into the GIS. Until this planning step is taken, the  value of these assessments could fail to reach their full potential.
section_title: Planning Concerns for GIS Component Extends to Overall Management and Direction of the Program
paragraphs: Finally, the uncertainty brought on by the lack of planning for the GIS  component is reflected in a similar uncertainty for the Port Security  Assessment Program as a whole. For the assessment components, future  plans are unclear beyond fiscal year 2005. Once all assessment reports of  the 55 strategic ports are completed—a task the Coast Guard expects to be  done by February 2005— the Coast Guard currently expects the  assessment of assessments component to be an ongoing effort that will be  updated by Coast Guard personnel as new assessment information  becomes available. It expects the special assessments and terrorist  operations assessments to continue through fiscal year 2005 as ports  previously assessed under earlier assessment approaches are revisited, but  it has made no decision about continuing them beyond that time.
Beyond fiscal year 2005, the Coast Guard is currently considering two  options for what to do with the special assessment and terrorist  operations assessment components of the program. The options are   (1) continuing the program at other ports beyond the initial 55 or   (2) conducting some recurring assessment at the 55 ports. Our discussions  with Captains of the Port and Coast Guard officials surfaced mixed views  of the future need for the three assessment components. One Captain  whose port had been assessed under the Version 3 approach said he would  like the assessment team to return to his port within 2 years, in order to  assess the security measures put in place after the completion of the last  assessment. By contrast, Captains for two other ports said they did not  think that the team needed to return unless the critical infrastructure in  their ports changed dramatically. The Coast Guard official responsible for  the program said that as of July 2004, discussions were underway between  program officials, other Coast Guard teams, and DHS officials as to how  the program should proceed in the future to best augment port security  efforts. The outcome of these discussions and future funding provided to  the program will largely determine the extent to which the three  assessment components continue to be implemented as part of the  program.
Although the GIS component will continue to be enhanced, its schedule  for completion and implementation is uncertain. Thus, when the various  program components—GIS and port assessments—are taken together, it is  not clear what activities will be conducted over the longer term, who will  do them, or how much they will cost.
section_title: Conclusions
paragraphs: As the Coast Guard attempts to determine the future of the Port Security  Assessment Program, it needs to ensure that the program provides  maximum effectiveness to its main customers, Captains of the Port and  Area Maritime Security Committees. The initial program had shortcomings  that created a product of marginal value. The revised program has  potential to be more useful because it intends to integrate all of the  assessment information collected by the Coast Guard and other relevant  security authorities and place this information in a GIS. However, the  Coast Guard risks producing a system that is not as useful as it could be,  because its approach lacks a defined management strategy, specific cost  estimates, and a clear implementation schedule. Developing the program’s  GIS component in this way is of particular concern, given the problems  that have resulted when other agencies used the same approach in  attempting to develop their information technology systems. And without  a clear development strategy for GIS, the usefulness of the three  assessment components may also be limited, because local Coast Guard  officials and Area Maritime Security Committees will be less able to use  them to fill the remaining gaps in their awareness of the security posture  within their ports. Getting this project right is important, because the  prospect of a well functioning GIS has great appeal to many Coast Guard  and other port stakeholders, who believe such a tool will be of  considerable help in providing effective port security.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that the revised Port Security Assessment Program  provides the most effective tool possible for security planning and  response, we recommend that the Secretary of Homeland Security direct  the Commandant of the Coast Guard to (1) define and document the GIS  functional requirements and (2) develop a long-term project plan for the  GIS and the Port Security Assessment Program as a whole (including cost  estimates, schedule, and management responsibilities).
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Department of Homeland Security  and the Coast Guard for their review and comment.  The Coast Guard’s  Marine Safety, Security And Environmental Protection Directorate  generally agreed with our recommendations, including the need to finalize  data types and develop a detailed work plan for adding map layers.  Coast  Guard officials provided a number of technical clarifications, which we  incorporated where appropriate to ensure the accuracy of our report.
The Coast Guard commented in detail on two aspects of our report:     The Coast Guard said our report tended to overlook many of the  program’s significant achievements, particularly the value of the three  assessment components. The Coast Guard emphasized the progress that it  had made on tailoring assessments, completing them on schedule, and  reducing their cost from more than $1 million per port to about $200,000  per port.
The Coast Guard also said our characterization of its GIS made it appear  that the Coast Guard was developing an entirely new information  technology system. The Coast Guard emphasized that its GIS was part of  an existing information technology system.
Regarding these concerns, we would make the following points:    First, the amount of emphasis the report places on GIS reflects our review  of Coast Guard documents and interviews with numerous local Coast  Guard officials, which showed that when compared with the three  assessment components, the GIS had the potential to provide substantially  more value.  The program’s Concept of Operations contains multiple  references to the critical and central role the GIS component will hold in  providing a dynamic tool to its users (Captains of the Port and Area  Maritime Security Committees) for port security planning and response.   Further, the end users we talked with expressed near unanimous need for  a dynamic GIS planning and response tool to increase maritime domain  awareness.
Second, we acknowledge that the Coast Guard’s GIS is part of a pre- existing information technology system.  In our view, however, this is not  the key point. The point is the need for GIS planning and functional  requirements. When we assessed the Coast Guard’s development efforts  against established industry best practices for developing information  technology systems, we found the Coast Guard’s current efforts do not  apply two key practices: defining what the GIS system should do and  establishing plans sufficient to ensure that the functional requirements can  be successfully realized.  Our past work has shown that when other  agencies tried to develop systems without these practices, problems  resulted. In short, without adequate planning, we believe that the GIS— and with it, the Port Security Assessment Program—is at risk of  foundering. Hence, the aim of our recommendation is to produce a more  effective GIS tool for port security officials.  If the Coast Guard does  establish functional requirements and a clear strategy for its GIS, the  system will more likely meet its potential, and port security officials will  be more likely to use it effectively.
We are sending copies of this report to relevant congressional committees  and subcommittees, the Secretary of Homeland Security, the Commandant  of the Coast Guard, and other interested parties.
If you or your staffs have any questions about this report, please contact  me at (415) 904-2200 or at wrightsonm@gao.gov or Steve Calvo, Assistant  Director, at (206) 287-4800 or at calvos@gao.gov. Key contributors to this  report are listed in appendix II. This report will also be available at no  charge on the GAO Web site at http://www.gao.gov.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our two objectives for this report were to (1) discuss why and how the  Port Security Assessment Program has changed over time and (2) assess  the Coast Guard’s approach for implementing the Port Security  Assessment Program as it is currently configured  To address why and how the assessment program changed, we reviewed  Coast Guard documents, interviewed officials at Coast Guard  headquarters responsible for implementing the program, and visited three  ports that had been assessed under the previous program assessment  approach. At these ports, we interviewed local Coast Guard personnel as  well numerous stakeholders to determine their views about how the  assessment process was carried out. These stakeholders included, for  example, operators of container terminals, power plants, cruise ship  terminals, port authorities, and chemical facilities. We also relied on our  previous work related to the program. For background information on the  role of the geographic information system (GIS) as a tool for planning and  response, we identified city and state government agencies that have GIS’s  in place and talked with GIS managers and experts from these agencies.  We also met with federal government GIS experts who had experience  with implementing GIS within the federal environment. They included  experts from the Federal Emergency Management Agency, Bureau of  Customs and Border Patrol, and United States Geological Survey. Finally,  we met with GIS experts at universities and elsewhere to further our  understanding.
To assess the Coast Guard’s approach for implementing the Port Security  Assessment Program in its current form, we interviewed a variety of Coast  Guard and other officials. For GIS, we interviewed the Coast Guard’s GIS  Program Manager and others to determine the progress made to date. For  the assessment portion of the program, we interviewed Coast Guard  officials from the six Captain of the Port zones that are responsible for the  security of the 12 ports assessed under the most recent program approach.  To establish criteria for assessing the program’s current approach, we  reviewed Coast Guard documents. We also reviewed information and  documentation related to GIS applications and identified industry best  practices for information systems acquisition and development to  determine criteria for managing such a project. We reviewed  documentation of the Coast Guard’s efforts to modify its port security GIS  to determine whether the progress made met the criteria we established.  In conducting our assessment, we also relied upon our work on the  development of major information technology systems throughout the  federal government.
Our work, which was conducted from June 2003 through August 2004, was  done in accordance with generally accepted government auditing  standards.
section_title: Appendix II: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to those named above, Chuck Bausell, Jason Berman,  Christopher Hatscher, Nicholas Larson, Elizabeth Roach, and   Stan Stenersen made key contributions to this report.