Proliferation Security Initiative: Agencies Have Adopted Policies and Procedures but Steps Needed to Meet Reporting Requirement and to Measure Results
Released: 2012-03-27T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: In 2003, the Bush Administration announced the Proliferation Security Initiative (PSI) to enhance U.S. efforts to prevent the spread of weapons of mass destruction (WMD). PSI is not a program housed in only one agency, but instead is a set of activities with participation by multiple U.S. agencies and other countries. Congress recommended that the Department of Defense (DOD) and Department of State (State) establish policies, procedures, and indicators to measure results and required that they submit annual reports. It also mandated that GAO report on PSI effectiveness. In 2008, GAO likewise recommended that law enforcement agencies also establish policies, procedures, and performance indicators.

This report assesses (1) the progress relevant agencies have made since 2008 in establishing recommended PSI policies and procedures and issuing required annual reports; and (2) the extent to which PSI activities have enhanced and expanded U.S. counterproliferation efforts.

GAO reviewed and analyzed agency documents and interviewed officials from State, DOD, and other agencies with PSI responsibilities.

section_title: What GAO Found
paragraphs: U.S. agencies have adopted interagency guidance documents that establish PSI policies and procedures and have submitted annual reports; however, these reports do not contain expenditure data for all agencies as required by law. The agencies produced documents that contain general PSI policies and procedures. In addition, DOD and the Department of Homeland Security’s Customs and Border Protection (CBP) developed policies and procedures specifically to guide their agencies’ PSI activities. The annual reports submitted in 2009, 2010, and 2011 met requirements to describe PSI-related activities planned for future years and those that took place in the preceding year. Although the reports included an account of DOD’s PSI expenditures, they did not contain all expenditures for other agencies for PSI activities as required by law.

U.S. officials participated in a range of PSI activities since 2008 to meet their objective of expanding and enhancing counterproliferation efforts, but it is unclear to what extent these activities have achieved the objective because agencies lack measures of results. The agencies either led or participated in 22 PSI activities from fiscal year 2009 through fiscal year 2011 including multilateral meetings and exercises. Officials stated that their outreach efforts contributed to increased support for PSI since GAO’s 2008 report, such as the increase from 93 to 98 countries endorsing PSI. In addition, they have extended access to PSI activities to more countries that are not part of the group of 21 PSI Operational Experts Group countries, for example by holding regional planning meetings. Despite recommendations of Congress and GAO that agencies develop PSI performance indicators, DOD, State, CBP, and the Federal Bureau of Investigation have not developed indicators that can be used to systematically measure progress toward the stated PSI objective. Further, the agencies have not systematically evaluated PSI activity results. Although some officials indicated plans to develop PSI performance indicators, officials from DOD and State also cited several challenges to developing indicators to measure PSI activities’ results including difficulty quantifying how PSI activities improved capacity. However, GAO has previously reported that, despite such challenges, developing measures that help link activities to results is possible. PSI agencies could develop a framework that links performance measures to outcomes. For example, such a framework could link the number of participants trained to changes in national policies that strengthen participant countries’ authority to interdict the shipment of WMD, their delivery systems, and related materials.

section_title: What GAO Recommends
paragraphs: GAO recommends that State and DOD provide all required expenditure information in PSI annual reports and develop a framework for measuring PSI’s results. DOD partially concurred with both recommendations and State partially concurred with the reporting recommendation. State disagreed with the framework recommendation, but noted its support for analysis consistent with it.

section_title: Letter
paragraphs: For almost 10 years, the United States has been actively participating in  the Proliferation Security Initiative (PSI) to assist in U.S. efforts to break  up black markets, detect and intercept weapons of mass destruction  (WMD) materials in transit, and use financial tools to disrupt this  dangerous trade. The U.S. government issued the National Strategy to  Combat Weapons of Mass Destruction in 2002, and the proliferation of  WMD, such as nuclear- or missile-related goods or technology, remains  one of today’s key challenges to international security. In May 2003,  President Bush announced PSI, a multinational effort, including the  United States, to focus on building WMD interdiction capabilities by  endorsing the Statement of Interdiction Principles. According to the  Department of State (State), PSI’s objective is to enhance and expand  U.S. efforts to prevent the flow of WMD, their delivery systems, and  related materials on the ground, in the air, and at sea, to and from states  and nonstate actors of proliferation concern. The current administration  has continued to support PSI, and agencies with PSI responsibilities have  reaffirmed that objective. Furthermore, the administration has declared  PSI an important global tool for countering the spread of WMD-related  materials, while President Obama has called for it to be turned into a  “durable international institution.”

In our September 2006 report on PSI, we found that U.S. agencies did  not have the policies and procedures in place to plan and manage their  PSI activities or performance indicators to measure their results. We  recommended that they correct these deficiencies. Following our report,  Congress passed the Implementing Recommendations of the 9/11  Commission Act of 2007 (the 9/11 Act), in which it expresses its sense  that relevant agencies and departments take actions to expand and  strengthen PSI, including establishing clear PSI policies and procedures  and performance indicators to measure the results of PSI activities. The  9/11 Act also required DOD and State to submit annual PSI reports and  mandated GAO to assess and report periodically on PSI’s effectiveness,  including progress made in implementing the law’s provisions. In 2008,  we reported that agencies had taken some steps but needed to do more  to strengthen and expand PSI, such as taking actions on some of the  law’s provisions.agencies’ actions and the progress evaluating PSI since our 2008 report.  Specifically, this report assesses (1) the progress relevant agencies have  made since 2008 in establishing recommended PSI policies and  procedures and issuing required annual reports; and (2) the extent to  which PSI activities have enhanced and expanded U.S.  counterproliferation efforts.

This report updates and provides information on the  To assess what agencies have done since 2008 to establish  recommended PSI policies and procedures and issue required annual  reports, we reviewed the findings and recommendations in GAO’s 2006  and 2008 reports on PSI and the congressional mandate. We reviewed  documentation of policies and procedures developed since 2008. We also  requested and reviewed annual reports submitted to Congress by State  and the Department of Defense (DOD) and analyzed them for compliance  with the requirements in the 9/11 Act. We reported expenditure data as it  appeared in the annual reports to Congress and determined the data  were sufficiently reliable for the purposes of our report. To assess how  PSI activities have enhanced and expanded U.S. efforts to prevent the  flow of WMD materials and to determine what actions, if any, agencies  have taken to develop indicators of PSI’s success, we reviewed and  analyzed documents from DOD, State, the Department of Homeland  Security (DHS), and the Department of Justice (DOJ). For both  objectives, we also interviewed relevant officials from those agencies.  (See app. I for a detailed discussion of our scope and methodology.)

We conducted this performance audit from June 2011 to March 2012 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: PSI is a multinational effort to prevent the trafficking of WMD, their  delivery systems, and related materials to and from states and nonstate  actors of proliferation concern. PSI has no formal organization or  bureaucracy. In the United States, PSI is not a program housed in a  single agency, but instead is a set of activities with participation by seven  agencies and the intelligence community. PSI encourages partnership  among states to work together to develop a broad range of legal,  diplomatic, economic, military, law enforcement, and other capabilities to  prevent WMD-related air, land, or sea transfers to states and nonstate  actors of proliferation concern. International participation is voluntary, and  there are no binding treaties on those who choose to participate.  Countries supporting PSI are expected to endorse PSI principles,  embodied in four broad goals in the Statement of Interdiction Principles of  September 2003, by a voluntary, nonbinding “political” commitment to  those principles. See appendix II for the full text of the Statement of  Interdiction Principles. They also voluntarily participate in PSI activities  according to their own capabilities. According to the principles, PSI  participants use existing national and international authorities to put an  end to WMD-related trafficking and take steps to strengthen those  authorities, as necessary.

The U.S. government’s PSI efforts involve participation in three broad  activities: multilateral PSI planning meetings called Operational Experts  Group (OEG) meetings, PSI exercises, and other efforts to encourage  support and capacity for interdictions, such as workshops and  conferences.military, law enforcement, intelligence, legal, and diplomatic experts from  the United States and other OEG countries meet to consider ways to  enhance the WMD interdiction capabilities of PSI participants, build  support for the initiative, develop operational concepts, organize PSI  exercises, and share information about national legal authorities. The  policy office in the Office of the Secretary of Defense heads the U.S.  delegation to these multilateral meetings.

According to State, at multilateral PSI planning meetings,  PSI exercises vary in size and complexity, and some involve military  personnel and assets from participating PSI countries. Other exercises  examine the use of law enforcement or customs authorities to stop WMD  proliferation. There are also “tabletop” exercises or simulations, which  explore scenarios and determine solutions for hypothetical land, air, or  sea interdictions. Among the most visible PSI exercises are those that  combine a tabletop and a live interdiction exercise using military assets  from multiple PSI countries, such as practicing the tracking and boarding  of a target ship.

Other activities include both outreach to countries that have not endorsed  PSI principles and cooperation and collaboration with countries that have  endorsed PSI and are seeking assistance to increase their capacity to act  in accordance with the Statement of Interdiction Principles. These efforts  include workshops, training, conferences, and bilateral discussions with  foreign government officials. U.S. officials said they engage in bilateral  discussions, for example, to conclude PSI shipboarding agreements or to  seek ways to overcome obstacles to support for PSI principles. State  takes the lead in diplomatic outreach efforts.

section_title: Agencies Have Adopted Policies and Procedures, but Annual Reports Lack Some Required Information
paragraphs: U.S. agencies have adopted interagency guidance documents that  establish policies and procedures for all agencies participating in PSI  activities. In addition, agencies have submitted annual reports, though  they lacked required estimated and actual expenditure information from  most participating agencies.
subsections: section_title: Interagency PSI Policies and Procedures Established
paragraphs: The 9/11 Act expresses Congress’ sense that DOD and State should  establish, among other things, clear policies and procedures and roles  and responsibilities for PSI. As we noted in our 2008 report, while DOD  and Customs and Border Protection (CBP) had established some policies  and procedures for their PSI activities, State and some law enforcement  agencies had not. Since then, an interagency group, including  representatives from all agencies participating in PSI, produced two  companion guidance documents, Guidance for U.S. Government  Activities in Support of the Proliferation Security Initiative and Insuring the  Durability of PSI: An Action Plan, that the interagency group adopted in  2010 and 2011, respectively. According to National Security Staff  officials, these documents are a primary source of policies and  procedures for all relevant agencies—defining PSI activities, providing  guidance on interagency communication, and addressing objectives and  responsibilities. For example, they establish that State has primary  responsibility for diplomatic outreach activities and that DOD leads the  U.S. delegation at OEG meetings. According to officials, such  interagency policies and procedures also allow all U.S. agencies involved  in PSI to plan activities without duplicating efforts and to properly and  coherently articulate the U.S. government’s vision and strategy on PSI.

In addition to interagency guidance, DOD and CBP also have their own  PSI-specific policy documents. As we reported in our 2008 review of PSI,  DOD had PSI-specific policies and procedures in place, specifically those  encouraging combatant command participation in PSI exercises. DOD  Joint Staff provided guidance directing combatant commands to leverage  the staff, assets, and resources of the existing DOD exercise program in  support of PSI exercises. This Joint Staff guidance provided procedures,  including roles and responsibilities, for the planning and execution of U.S.  military support to PSI. Among other things, the guidance encouraged  combatant commands to change existing DOD exercises by adding a PSI  component. Our 2008 review also reported that CBP, a component of  DHS, produced a PSI-specific directive that provides roles and  responsibilities, policies and procedures, and PSI-relevant definitions.  Approved in 2006, CBP is currently revising this document to reflect  updated roles and responsibilities.

section_title: Agencies Submitted Reports, but They Lacked Some Required Information
paragraphs: The 9/11 Act required DOD and State to submit to Congress in February  an annual comprehensive joint report, beginning in 2008. The report is  to consist of a 3-year plan describing PSI-related activities and identifying  estimated expenditures for these activities, and a description of the PSI- related activities and associated expenditures carried out during the fiscal  year preceding the year of the report.report, although responsibility for leading the annual effort alternates   DOD and State co-author each  between the two. The lead agency solicits input from all U.S. agencies  participating in PSI and consolidates the information provided to produce  the final report for submission to Congress. State and DOD have  submitted annual PSI reports in 2009, 2010, and 2011 including  descriptions of PSI activities, but these reports do not include all required  estimated expenditures for PSI activities over the next 3 fiscal years and  the amount expended in the prior year.

DOD is the only agency performing PSI activities that has provided all  required planned and prior year expenditure information for these reports.  For example, the 2011 report includes information from DOD stating that  it plans to conduct a regional PSI exercise called Leading Edge in Central  Asia in fiscal year 2012 and in the Middle East in fiscal year 2014 and that  each exercise is estimated to cost $600,000 for staff travel to support  planning events and exercise execution. Table 1 shows annual planned  expenditures from that report for PSI-related exercises in fiscal years  2012 to 2014 for DOD combatant commands and Joint Staff only,  including many of the major PSI-related activities expected in that period.  The report indicates that these expenditures include travel expenses,  conference hosting fees, contracting support, training expenses, and  other uniquely PSI-attributable expenses and lists all agencies consulted  in the preparation of the report.

In addition, DOD has fully reported its prior year expenditures and, in  many cases, includes expenditure amounts by activity. For example, the  2011 report states that DOD’s total PSI expenditures for that year were  approximately $519,000. Further, the report includes activity-specific  expenditure information, such as the $31,400 DOD spent for training aids  and staff travel in support of the Phoenix Express PSI exercise.

The annual reports do not uniformly include expenditure data for any  other agencies participating in PSI. For example, the 2010 report states  that the United States hosted an OEG meeting in Miami, Florida, for  which the United States had a delegation of 34 representatives from  DOD, State, DHS, the Federal Bureau of Investigation (FBI), the  Department of the Treasury (Treasury), and the Department of Energy  (DOE), among others. The report states that DOD contributed about  $145,000 but does not include expenditure data from other agencies for  the event. Table 2 shows the extent to which PSI annual reports include  expenditure information for U.S. agencies participating in PSI activities.

Although reports contain some expenditure data from agencies other than  DOD, none of the other agencies provide this data for all of the annual  reports. For example, for fiscal years 2008 and 2009, DOE reported total  PSI expenditures of $69,000 and $55,000 respectively. Also, FBI reported  that it spent a total of about $750,000 to organize and sponsor PSI events  in Hungary and Australia in fiscal year 2009. However, no expenditure  data were provided for activities in which DOE participated for fiscal year  2010 and for FBI’s activities in fiscal years 2008 and 2010. From DHS,  CBP officials could document submitting PSI-specific expenditures of  about $35,000 for the 2010 report; however, the report did not include this  information, stating that the only costs DHS incurred were associated with  travel to PSI events. According to State officials, State does not identify  and track its PSI-related expenditures because it made the decision that  such a breakout is unnecessary. However, officials said they could  calculate State’s total PSI-related expenditures because these are almost  entirely for travel to OEG meetings or bilateral negotiations. FBI officials  also reported that they do not separately track PSI expenditures.

Of the four agencies we spoke with, only DOD makes a specific budget  request for PSI-related activities. U.S. Strategic Command has a fiscal  year 2011 budget of $800,000 to provide financial assistance for  combatant commanders to plan, participate in, and execute WMD  interdiction exercises. This fund is used routinely but not exclusively to  support PSI activities such as hosting a PSI exercise, embedding PSI  scenarios into broader military exercises, ensuring that appropriate  subject matter experts can participate in PSI exercises, or purchasing  interdiction-related training aids. According to State officials, they prefer  funding PSI expenditures from their general operating accounts because  this practice allows them greater flexibility and the ability to fund PSI- related travel when the need arises. FBI and CBP officials also reported  that they provide PSI funding when necessary through more general  operating accounts.

Agencies that do not track and report their expenditures are not providing  Congress sufficient information for Congress to assess U.S. participation  in PSI. If agencies are unable to provide requested assistance, but are  not tracking or reporting on expenditures, it is difficult for the agencies to  determine and demonstrate to Congress whether they are effectively  prioritizing their use of limited resources. For example, FBI officials stated  they could not provide PSI-related training requested in 2011 by  Colombian national police because they did not have sufficient funding.

Moreover, the lack of required expenditure information limits Congress’  ability to oversee agencies’ commitment to the PSI objective.

section_title: Extent to Which Activities Meet PSI Objective Is Unclear because Agencies Lack Measures of Results
paragraphs: DOD, State, CBP, and FBI officials participated in a range of PSI activities  since 2008 to meet their objective of expanding and enhancing  counterproliferation efforts, but it is unclear to what extent these activities  have achieved the objective because agencies lack measures of results.  The agencies either led or participated in 22 PSI activities from fiscal year  2009 through fiscal year 2011, including multilateral meetings and  exercises. In addition, they have extended access to PSI activities to  more countries that are not part of the 21-country OEG. Despite  recommendations in the 9/11 Act and by GAO that agencies develop PSI  performance indicators, DOD, State, CBP, and FBI have not developed  indicators that can be used to systematically measure progress toward   Further, the agencies have not systematically  the stated PSI objective.evaluated PSI activity results. Although some officials indicated plans to  develop PSI performance indicators, officials from DOD and State also  cited several challenges to developing indicators to measure PSI  activities’ results, including difficulty quantifying how PSI activities  improved capacity. However, GAO has previously reported that, despite  challenges, evaluating results and developing measures are possible.  One approach PSI agency officials could consider is developing a  framework to link performance measures, such as number of participants  trained, to outcomes, such as changes in national policies that strengthen  their authority to perform interdictions.
subsections: section_title: U.S. Agencies Participated in PSI Activities to Support PSI’s Objective
paragraphs: PSI’s objective is to enhance and expand our capacity to prevent the flow  of WMD, their delivery systems, and related materials on the ground, in  the air, and at sea, to and from states and nonstate actors of proliferation  concern. Agency officials provided us with a range of PSI activities they  have participated in since our 2008 report to support the PSI objective.  U.S. agencies led or participated in 22 PSI activities from fiscal year 2009  through fiscal year 2011 and State officials reported numerous informal  bilateral consultations designed to provide tools that increase countries’  capacity to interdict illicit shipments of WMDs and WMD-related materials.  As table 3 shows, these activities fall under the three broad categories of  the U.S. government’s PSI efforts: multilateral OEG meetings, PSI  exercises, and other activities such as workshops and training sessions.  See appendix III for a summary of each of the PSI activities with U.S.  participation from fiscal year 2009 to December 2011. For example, the  U.S. Naval Forces Africa hosted an exercise in May 2010 that included a  PSI maritime interdiction scenario designed to improve regional  cooperation and maritime security in the Mediterranean basin. Also, a  U.S. delegation participated in an Australia-hosted exercise in September  2010 that focused on an aircraft counterproliferation scenario and  associated customs and law enforcement authorities and challenges. A  regional PSI planning meeting was held in conjunction with the exercise.

Officials from DOD, State, DHS, and FBI stated that their outreach efforts,  including State’s diplomatic efforts, have contributed to an increased  number of countries supporting PSI since our 2008 report. For example,  the number of countries that became PSI countries by endorsing the  Statement of Interdiction Principles has increased from 93 to 98 since  2008 and DOD officials said that they have ongoing efforts with several  non-endorsing countries and believe that the number of endorsing  countries will continue to rise. The most recent endorsees are Antigua  and Barbuda, Colombia, the Republic of Korea (South Korea), St. Vincent  and the Grenadines, and Vanuatu. Further, the number of signed bilateral  shipboarding agreements between the United States and individual PSI  countries increased between November 2008 and February 2012 from 9  to 11. Bilateral shipboarding agreements put procedures into place and  identify points of contact to permit the timely inspection by either party of  vessels flying their flags suspected of transporting proliferation-related  cargo. State officials said these agreements are significant because they  have been signed by countries with some of the largest ship registries.

U.S. agencies, in cooperation with the 20 other OEG countries, also have  extended access to PSI activities to additional countries since our last  report. In 2008, we reported that U.S. agencies had not built relationships  with PSI countries that are not OEG countries by involving them in PSI  planning and activities. We recommended that U.S. agencies work in  conjunction with other leading PSI countries to increase cooperation,  coordination, and information exchange with PSI countries that were not   In May 2009, the  invited to multilateral OEG planning meetings. countries of the OEG committed to consider ways to involve more  countries in future PSI planning meetings. According to U.S. officials, they  also agreed to increase the number of PSI activities, including regional  PSI planning meetings, that include PSI-endorsing countries beyond the  OEG countries and, in some cases, non-endorsing countries. In addition,  the 2011 U.S. interagency PSI report to Congress stated that activities  planned for fiscal years 2012 through 2014 are designed to increase  cooperation, coordination, and information exchange with the broad range  of PSI-endorsing countries beyond the countries that normally participate  in the OEG.

GAO-09-43.

Eight of the 17 participating delegations at a PSI activity hosted in  September 2010 by Australia represented countries not normally  invited to multilateral PSI planning meetings. The activity included an  aircraft counterproliferation scenario exercise and a regional PSI  planning meeting.

According to U.S. officials, the Republic of Korea endorsed PSI in  May 2009 as a result of its participation in PSI activities and became  one of the 21 leading PSI countries in November 2010. It organized a  PSI exercise and hosted a regional PSI workshop in 2010.

U.S. officials said they are also currently developing the Critical  Capabilities and Practices effort, a clearinghouse for PSI-related  information and tools, designed to make PSI lessons learned and best  practices available both to PSI activity participants and to endorsing  countries that were unable to participate in the activities. The OEG  countries discussed and confirmed support for the development of the  Critical Capabilities and Practices effort. Officials said they have not set a  target date for the effort to begin making PSI-related information and tools  available.

section_title: U.S. Agencies Have Not Established a Framework to Measure PSI’s Results
paragraphs: While U.S. agencies have undertaken a range of PSI efforts since 2008,  they have not established a framework to measure PSI activities’ results  and, therefore, it is unclear to what extent these activities have enhanced  and expanded capacity to prevent the flow of WMD, their delivery  systems, and related materials on the ground, in the air, and at sea, to  and from states and nonstate actors of proliferation concern. The 9/11 Act  recommended that DOD and State establish indicators to measure results  of their PSI activities. It is the responsibility of the implementing agencies  to measure the results of PSI activities. Linking the activities to the  initiative’s overall objective could help the agencies better organize and  prioritize future activities. In 2008, we reported that agencies had not  established PSI performance indicators. We also recommended that law  enforcement agencies, such as DHS and FBI, which we found to have  become increasingly involved in PSI activities, establish PSI performance  indicators and they concurred.

The four agencies we reviewed have not established formal performance  indicators that can be used to systematically measure progress toward  the stated PSI objective. DOD officials said they have general  counterproliferation goals but have not developed PSI indicators. FBI  officials said that they have performance measures applying to activities  of their Weapons of Mass Destruction Directorate, and that their PSI- specific goal is the commitment to participate in PSI activities as  appropriate, including at least one PSI exercise per year. Although FBI  officials consider this to be a performance indicator, participation alone  does not provide information to measure the results of PSI activities.  State has identified as its PSI indicator “An Effective Global Network  Countering WMD Proliferation-Related Trafficking.” However, the  measures used to track the indicator’s progress change from year to year,  making it difficult for State to assess systematically the results of its PSI  activities. For example, although State did not meet its fiscal year 2009  target to have 100 PSI-endorsing countries, it did not set targets for the  number of PSI endorsees for fiscal years 2010 or 2011. We reported in  2008 that CBP developed a PSI Implementation Plan with expected goals  and targets, but that it had not been updated since June 2006. As of  February 2012, CBP officials stated that many of the goals are outdated  and that the plan has not been updated because they wanted to wait and  align their plan with the interagency PSI policy documents that were  completed in 2010 and 2011. The officials acknowledged that  performance measurement is important and said they are working to  include indicators in their updated plan.

In addition, although some agencies have made efforts to assess  individual PSI activities, the four agencies we reviewed have not  performed a systematic evaluation of the results of PSI activities. Some  officials draft summary reports following individual activities to identify  issues to address in planning for future activities. For example, FBI  officials said they conducted a formal evaluation of the two workshops  they sponsored in 2009. The report from the September 2009 workshop  in Australia included some reporting of feedback from a written survey  conducted at the end of the activity. Similarly, a CBP official provided  examples of post-activity reports that include recommendations for future  PSI efforts. However, these reports do not represent a systematic  evaluation that links the activity results to the PSI objective. The officials  with whom we met said they consider past activities when planning for the  future, but they have no systematic evaluation of the activity results or  impact to aid their planning. Further, the annual PSI reports to Congress  do not generally document results of U.S. agencies’ activities beyond the  names of the participating countries.

State and DOD officials cited a number of challenges to developing  indicators to measure the impact or results of the U.S. government’s PSI  activities. According to the officials, results that can be quantified can also  be misleading. For example, the officials stated the following:

Tracking the overall number of new countries endorsing PSI ignores  the fact that U.S. agencies have strategic reasons for focusing efforts  on a certain country or subset of countries, even if a larger number of  other countries might be persuaded more quickly to endorse.

Tracking the number of interdictions does not necessarily link PSI  activities to the initiative’s objective. According to the officials, the  initiative’s activities are focused on building capacity to perform  interdiction, but actual interdictions are not performed as part of PSI.  They emphasized that they cannot credit successful interdictions to  PSI activities, in part because there are many efforts in addition to PSI  that are focused on counterproliferation. Also, a change in the number  of interdictions over the previous year could be attributed to a range of  positive and negative factors, including better capacity to interdict,  increased transfers of illicit material, or failure to deter transfers.

Tracking measures of U.S. efforts, such as the number of activities  hosted by the United States, ignores the fact that PSI is a  multinational effort.

Further, State officials said some of the results of PSI activities are  difficult to quantify. For example, although the officials said tools and  expertise provided to participants at activities are used by participants to  enhance their ability to interdict illicit cargo, the extent of the improvement  is often difficult to measure and track.

Despite the challenges U.S. agencies face in developing performance  indicators, it is possible for agencies to measure performance. One  possible approach agencies could consider is to develop a framework  that links PSI activities to the initiative’s objective. GAO has previously  identified such frameworks, called logic models, that agencies could  adopt even if they face performance measurement challenges similar to  some of the ones identified by PSI implementing agencies.GAO reported that using a logic model could allow the agencies to  consider indirect outcomes and unquantifiable benefits when linking  activities and outputs to the overall objective. By specifying the program’s  theory of what is expected, a logic model can help evaluators define  measures of the program’s progress toward its ultimate goals. In  particular, GAO found that logic models were used or could be used to  measure results in programs with challenges similar to some of PSI’s,  including, for example, difficulty in observing changes in behavior  occurring after activity completion and difficulty in attributing outcomes to  activities because of external factors’ influence on the outcomes. Figure 1  provides generic guidance on how a logic model framework could be  used to link agencies’ program inputs and outputs to outcomes or impact.  A logic model can help identify pertinent variables and how, when, and in  whom they should be measured, as well as other factors that might affect  program results.

section_title: Conclusions
paragraphs: PSI has the potential to increase global capacity to recognize and  interdict the flow of WMD, their delivery systems, and related materials on  the ground, air, or sea, to and from states and nonstate actors of  proliferation concern. The previous and current administrations have  committed to implementing PSI and President Obama has called for it to  be turned into a durable international institution. Given this commitment, it  is important that agencies ensure that PSI is fully implemented according  to legislative recommendations and requirements. Since our 2008 report,  U.S. agencies have developed interagency PSI policies and procedures  that satisfy a recommendation of the 9/11 Act. They have also submitted  annual reports covering fiscal years 2008 through 2010 documenting their  PSI activities, but these reports have not included the associated funding  information for all PSI implementing agencies, as required by law. While  all agencies we spoke with except DOD stated that they fund their PSI  activities from their general funds, rather than request an annual PSI  budget, it does not eliminate the requirement that they provide  expenditure information for their PSI activities under the 9/11 Act. Without  reports that include the required information for all agencies’ PSI  activities, Congress will not know how much the U.S. government is  spending on PSI and how such funds are being allocated. Also, while  U.S. agencies have provided years of documentation of a range of  activities they have performed or plan to perform under PSI, they have  been unable to demonstrate if or how these activities are linked to the PSI  objective. Without performance indicators that can be used to  systematically measure progress toward the stated PSI objective and a  framework for measuring the results of PSI activities, Congress and the  public do not have a sufficient basis to judge whether PSI activities are  successful.

section_title: Recommendations for Executive Action
paragraphs: To ensure that Congress has information to assess U.S. participation in  PSI, we recommend that the Secretaries of Defense and State take the  following two actions:  1.  Include in the annual PSI report to Congress the required expenditure  information for all U.S. agencies participating in PSI activities; and 2.  Develop a framework for measuring PSI activities’ results, including  performance measures where possible that help link the results to  PSI’s objective.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD, State, FBI, and DHS. DOD and  State provided written comments on a draft of this report, which are  reprinted in appendixes IV and V along with our responses to specific  points. FBI and DHS, along with DOD and State provided technical  comments that we have incorporated into this report, as appropriate.

In commenting on the draft report, DOD partially concurred with both of  our recommendations. Consistent with our recommendation, DOD agreed  that the annual PSI report should include expenditure information for all  U.S. agencies. DOD stated that the reports have included information on  expenditures by DOD and other agencies that are unique to PSI, while  excluding items that are accounted for in agency general operating  budgets. However, we found that some expenditure amounts not reported  in the annual reports were unique to PSI and, therefore, should have  been included. In a 2009 e-mail to CBP, a DOD official stated that travel  expenses could be included in its submission to the annual report, as long  as they were specifically for a PSI event. However, after CBP submitted  such expenditures to DOD, they were excluded from the 2010 annual  report even though they were for PSI-specific travel. DOD also partially  concurred with our recommendation to develop a results framework. The  department cited challenges in establishing objective and quantifiable  measures of success but committed to make an effort to implement the  recommendation by using the Critical Capabilities and Practices (CCP)  concept as a results framework and to identify meaningful performance  measures, where appropriate.

State partially concurred with the recommendation to provide expenditure  information for all U.S. agencies participating in PSI activities. The  department explained that it is difficult to define some expenditures as  unique to PSI, for example, because travel in support of PSI events often  coincides with travel in support of other department operational activities.  However, the department said it would closely examine travel-related and  other expenses unique to PSI in order to include them in future reports to  Congress. State did not concur with our recommendation that it should  develop a results framework. State said it had some indicators, such as  the numbers of PSI-endorsing countries, for use in measuring PSI  progress, but cautioned that PSI does not lend itself to collective data that  would provide reasonable approximation of results. Nonetheless, State  cited the CCP effort as one tool it intends to use, in coordination with  other participating U.S. agencies, “which could contribute to an effective  future analysis of the outcomes of coordinated PSI activity,” which is  consistent with our recommendation. Although we made a written request  for documentation of State’s PSI performance indicators in July 2011,  State did not provide documentation of its PSI indicator and targets until  March 2012, after it provided its response to our draft report. We have  revised our report findings to include this documentation and our  assessment. Upon reviewing the documentation provided, we found that  the metrics State identified were not consistently listed as annual metrics  in the strategic plan to which they refer. For example, neither the number  of endorsing states nor the conclusion of shipboarding agreements were  listed as metrics for fiscal years 2010 or 2011. In addition, State set no  numeric targets for its participation in PSI activities for fiscal years 2010  or 2011. Without an overall results framework including, where possible,  consistent indicators and targets that can be tracked over time, State  cannot systematically evaluate its PSI activities.

We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, the Secretary of State, the  Secretary of Homeland Security, the Attorney General, and other  interested parties. In addition, this report is available at no charge on the  GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please  contact me at (202) 512-9601 or melitot@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. Key contributors are listed in appendix VI.

section_title: Appendix I: Scope and Methodology
paragraphs: To assess the progress agencies have made since 2008 in establishing  recommended Proliferation Security Initiative (PSI) policies and  procedures, and issuing required annual reports, we reviewed the  findings and recommendations in GAO’s 2006 and 2008 reports on PSI  and the Implementing Recommendations of the 9/11 Commission Act of  2007 (the 9/11 Act). We requested and reviewed documentation of  policies and procedures developed since 2008. Because the National  Security Staff is the custodian of the interagency policy documents, we  discussed the contents of those documents with National Security Staff  officials and reviewed the documents to confirm that they contained PSI  policies and procedures. We also reviewed PSI annual reports submitted  to Congress by the Departments of State (State) and Defense (DOD) and  analyzed them for compliance with the requirements in the 9/11 Act. We  reported expenditure data as they appeared in the annual reports to  Congress and discussed the reliability of the data with agency officials.  We determined the data were sufficiently reliable for the purposes of our  report. Further, we interviewed relevant agency officials to better  understand the extent to which they implemented the requirements and  recommendations in the 9/11 Act and any compliance challenges they  faced.

To assess the extent to which PSI activities have enhanced and  expanded U.S. efforts to prevent the flow of weapons of mass destruction  (WMD) materials, we reviewed and analyzed documents from DOD,  State, the Department of Homeland Security (DHS), and the Department  of Justice (DOJ). We analyzed the information on each PSI activity in the  annual reports covering fiscal years 2009 and 2010. We reported the  number of PSI activities with U.S. participation based on agency  documents. Because the agencies have not yet submitted the annual  report for PSI activities in fiscal year 2011, we analyzed information on  fiscal year 2011 activities provided by agency officials. We also  interviewed relevant agency officials and solicited responses from seven  DOD combatant commands to a list of questions about the PSI. In  particular, we requested and analyzed documentation of actions relevant  agencies have taken, if any, to develop indicators of PSI’s success. In  addition, we compared our findings with those of the 2008 GAO PSI  review and reviewed GAO reports assessing agencies’ evaluation  frameworks and performance measurement.

For both objectives, we focused on DOD and State because the  applicable recommendations and requirements in the mandate were  addressed to them. In addition, we assessed DHS and DOJ progress  because GAO made recommendations to those agencies in 2008 to  develop policies, procedures, and performance indicators because of the  increased involvement of law enforcement agencies in U.S. PSI efforts.

section_title: Appendix II: Full Text of the Statement of Interdiction Principles
paragraphs: The PSI is a response to the growing challenges posed by the  proliferation of WMD, their delivery systems, and related materials  worldwide. The PSI builds on efforts by the international community to  prevent proliferation of such items, including existing treaties and  regimes. It is consistent with, and a step in the implementation of the UN  Security Council Presidential Statement of January 1992, which states  that the proliferation of all WMD constitutes a threat to international peace  and security, and underlines the need for member states of the UN to  prevent proliferation. The PSI is also consistent with recent statements of  the G8 and the European Union, establishing that more coherent and  concerted efforts are needed to prevent the proliferation of WMD, their  delivery systems, and related materials. PSI participants are deeply  concerned about this threat and of the danger that these items could fall  into the hands of terrorists and are committed to working together to stop  the flow of these items to and from states and nonstate actors of  proliferation concern.

The PSI seeks to involve, in some capacity, all states that have a stake in  nonproliferation and the ability and willingness to take steps to stop the  flow of such items at sea, in the air, or on land. The PSI also seeks  cooperation from any state whose vessels, flags, ports, territorial waters,  airspace, or land might be used for proliferation purposes by states and  nonstate actors of proliferation concern. The increasingly aggressive  efforts by proliferators to stand outside or to circumvent existing  nonproliferation norms, and to profit from such trade, requires new and  stronger actions by the international community. We look forward to  working with all concerned states on measures they are able and willing  to take in support of the PSI, as outlined in the following set of  “Interdiction Principles.”
subsections: section_title: Interdiction Principles for the Proliferation Security Initiative
paragraphs: PSI participants are committed to the following interdiction principles to  establish a more coordinated and effective basis through which to impede  and stop shipments of WMD, delivery systems, and related materials  flowing to and from states and nonstate actors of proliferation concern,  consistent with national legal authorities and relevant international law  and frameworks, including the UN Security Council. They call on all states  concerned with this threat to international peace and security to join in  similarly committing to:  1.  Undertake effective measures, either alone or in concert with other  states, for interdicting the transfer or transport of WMD, their delivery  systems, and related materials to and from states and nonstate actors  of proliferation concern. “States or nonstate actors of proliferation  concern” generally refers to those countries or entities that the PSI  participants involved establish should be subject to interdiction  activities because they are engaged in proliferation through: (1) efforts  to develop or acquire chemical, biological, or nuclear weapons and  associated delivery systems; or (2) transfers (either selling, receiving,  or facilitating) of WMD, their delivery systems, or related materials. 2.  Adopt streamlined procedures for rapid exchange of relevant  information concerning suspected proliferation activity, protecting the  confidential character of classified information provided by other  states as part of this initiative, dedicate appropriate resources and  efforts to interdiction operations and capabilities, and maximize  coordination among participants in interdiction efforts. 3.  Review and work to strengthen their relevant national legal authorities  where necessary to accomplish these objectives, and work to  strengthen when necessary relevant international law and frameworks  in appropriate ways to support these commitments. 4.  Take specific actions in support of interdiction efforts regarding  cargoes of WMD, their delivery systems, or related materials, to the  extent their national legal authorities permit and consistent with their  obligations under international law and frameworks, to include:  a.  Not to transport or assist in the transport of any such cargoes to or  from states or nonstate actors of proliferation concern and not to  allow any persons subject to their jurisdiction to do so. b.  At their own initiative, or at the request and good cause shown by  another state, to take action to board and search any vessel flying  their flag in their internal waters or territorial seas, or areas beyond  the territorial seas of any other state, that is reasonably suspected  of transporting such cargoes to or from states or nonstate actors  of proliferation concern, and to seize such cargoes that are  identified. c.  To seriously consider providing consent under the appropriate  circumstances to the boarding and searching of its own flag  vessels by other states, and to the seizure of such WMD-related  cargoes in such vessels that may be identified by such states. d.  To take appropriate actions to (1) stop and/or search in their  internal waters, territorial seas, or contiguous zones (when  declared) vessels that are reasonably suspected of carrying such  cargoes to or from states or nonstate actors of proliferation  concern and to seize such cargoes that are identified and (2) to  enforce conditions on vessels entering or leaving their ports,  internal waters, or territorial seas that are reasonably suspected of  carrying such cargoes, such as requiring that such vessels be  subject to boarding, search, and seizure of such cargoes prior to  entry. e.  At their own initiative or upon the request and good cause shown  by another state, to (a) require aircraft that are reasonably  suspected of carrying such cargoes to or from states or nonstate  actors of proliferation concern and that are transiting their airspace  to land for inspection and seize any such cargoes that are  identified and/or (b) deny aircraft reasonably suspected of carrying  such cargoes transit rights through their airspace in advance of  such flights. f.

If their ports, airfields, or other facilities are used as transshipment  points for shipment of such cargoes to or from states or nonstate  actors of proliferation concern, to inspect vessels, aircraft, or other  modes of transport reasonably suspected of carrying such  cargoes, and to seize such cargoes that are identified.

section_title: Appendix III: Proliferation Security Initiative Activities from Fiscal Year 2009 to December 2011
subsections: section_title: Month Fiscal year 2012 November

section_title: Fiscal year 2011 September

section_title: Fiscal year 2010 September

section_title: Month Fiscal year 2009 September
paragraphs: Source- GAO analysis based on DOD and State data.

section_title: Appendix IV: Comments from the Department of State
subsections: section_title: GAO Comments
paragraphs: 1.  Our recommendation focuses on developing a framework that links  PSI activities to the initiative’s objective, and not just on indicators,  which alone do not link the activities to the desired outcomes.  Although we made a written request for documentation of State’s PSI  performance indicators in July 2011, State did not provide  documentation of its PSI indicator and targets until March 2012, after  it provided its response to our draft report. We have revised our report  findings to include this documentation and our assessment. Upon  reviewing the documentation provided, we found that the metrics  State identified were not consistently listed in its bureau strategic plan  as annual metrics. For example, neither the number of endorsing  states nor the conclusion of shipboarding agreements were listed as  metrics for fiscal years 2010 and 2011. In addition, State set no  numeric targets for its participation in PSI activities for fiscal years  2010 and 2011. Without an overall results framework including, where  possible, consistent indicators and targets that can be tracked over  time, State cannot systematically evaluate its PSI activities. 2.  State’s decision to consider including PSI-specific expenditures in  future reports to Congress is consistent with our recommendation. 3.  If the Critical Capabilities and Practices (CCP) concept is developed  as a framework that links PSI activities to outcomes and the objective  of PSI, it will be consistent with our recommendation. (See also  comment 1.)  4.  Based on State’s response, we have revised the report to reflect  State’s justification for not breaking out PSI expenditures. However,  State’s decision not to track PSI expenditures is inconsistent with the  requirement to report such information annually.  5.  The description of OEG meetings as planning meetings is consistent  with our 2008 report on PSI. State concurred with this decision. In  addition, we believe that the activities currently listed by State are  consistent with PSI planning. We have added a footnote in the  background more fully explaining OEG meetings.

section_title: Appendix V: Comments from the Department of Defense
subsections: section_title: GAO’s Comments
paragraphs: 1.  The 9/11 Act requires that DOD, in conjunction with State, report to  Congress annually on the amount expended for the prior year’s PSI  activities. Because DOD is the only agency we spoke with that makes  a specific budget request for PSI-related activities, its decision not to  report expenditure amounts accounted for in agency general  operating budgets limits Congress’ knowledge of the amount the U.S.  government is spending on PSI and how those funds are being  allocated. In addition, we found that some expenditure amounts not  included in the annual reports were unique to PSI and, therefore,  should have been included. In a 2009 email to CBP, a DOD official  stated that travel expenses could be included in its submission to the  annual report, as long as they were specifically for a PSI event.  However, after CBP submitted such expenditures to DOD, they were  excluded from the 2010 annual report even though they were for PSI- specific travel. 2.  DOD’s willingness to work with interagency partners toward  developing and using a framework to assess PSI activities and toward  identifying meaningful performance measures is consistent with our  recommendation. If the CCP is developed as a framework that links  PSI activities to outcomes and the objective of PSI, it will be  consistent with our recommendation.

section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Godwin Agbara (Assistant  Director), Jeffrey Baldwin-Bott, Lynn Cothern, and Mattias Fenton made  key contributions to this report. Martin de Alteriis, Mark Dowling, and Mary  Moutsos also provided technical assistance.


Question-summary annotations: questions: question: How have U.S. agencies established PSI policies and procedures?
answer: U.S. agencies have adopted interagency guidance documents that establish PSI policies and procedures and have submitted annual reports; however, these reports do not contain expenditure data for all agencies as required by law.
child_questions: question: What documentation have the agencies produced?
answer: The agencies produced documents that contain general PSI policies and procedures.

question: What PSI guidance have DOD and CBP developed?
answer: In addition, DOD and the Department of Homeland Security’s Customs and Border Protection (CBP) developed policies and procedures specifically to guide their agencies’ PSI activities.

question: To what extent did the 2009-2011 reports met PSI requirments?
answer: The annual reports submitted in 2009, 2010, and 2011 met requirements to describe PSI-related activities planned for future years and those that took place in the preceding year. Although the reports included an account of DOD’s PSI expenditures, they did not contain all expenditures for other agencies for PSI activities as required by law.

questions: question: To what extent have PSI activities achieved their objectives regarding counterproliferation?
answer: U.S. officials participated in a range of PSI activities since 2008 to meet their objective of expanding and enhancing counterproliferation efforts, but it is unclear to what extent these activities have achieved the objective because agencies lack measures of results. The agencies either led or participated in 22 PSI activities from fiscal year 2009 through fiscal year 2011 including multilateral meetings and exercises.

question: What were the results of the outreach efforts?
answer: Officials stated that their outreach efforts contributed to increased support for PSI since GAO’s 2008 report, such as the increase from 93 to 98 countries endorsing PSI.
child_questions: question: How has the range of PSI activities been expanded?
answer: In addition, they have extended access to PSI activities to more countries that are not part of the group of 21 PSI Operational Experts Group countries, for example by holding regional planning meetings.

question: To what extent can the progress of the PSI program be measured?
answer: Despite recommendations of Congress and GAO that agencies develop PSI performance indicators, DOD, State, CBP, and the Federal Bureau of Investigation have not developed indicators that can be used to systematically measure progress toward the stated PSI objective. Further, the agencies have not systematically evaluated PSI activity results.
child_questions: question: What efforts have been made to develop PSI performance indicators?
answer: Although some officials indicated plans to develop PSI performance indicators, officials from DOD and State also cited several challenges to developing indicators to measure PSI activities’ results including difficulty quantifying how PSI activities improved capacity.

question: What has GAO recommended regarding assessment?
answer: However, GAO has previously reported that, despite such challenges, developing measures that help link activities to results is possible. PSI agencies could develop a framework that links performance measures to outcomes. For example, such a framework could link the number of participants trained to changes in national policies that strengthen participant countries’ authority to interdict the shipment of WMD, their delivery systems, and related materials.

questions: question: Why was the PSI program created?
answer: In 2003, the Bush Administration announced the Proliferation Security Initiative (PSI) to enhance U.S. efforts to prevent the spread of weapons of mass destruction (WMD).
child_questions: question: How is the PSI program organized?
answer: PSI is not a program housed in only one agency, but instead is a set of activities with participation by multiple U.S. agencies and other countries.

question: What has Congress recommended regarding PSI?
answer: Congress recommended that the Department of Defense (DOD) and Department of State (State) establish policies, procedures, and indicators to measure results and required that they submit annual reports.

question: What has GAO recommended regarding PSI?
answer: It also mandated that GAO report on PSI effectiveness. In 2008, GAO likewise recommended that law enforcement agencies also establish policies, procedures, and performance indicators.