Department of State: Status of Achieving Key Outcomes and Addressing Major Management Challenges
Released: 2002-01-07T00:00:00

Letter

Background

GPRA is intended to shift the focus of government decisionmaking, management, and accountability from activities and processes to results and outcomes achieved by federal programs. New and valuable information on the plans, goals, and strategies of federal agencies has been provided since federal agencies began implementing GPRA. Under GPRA, annual performance plans are to clearly inform the Congress and the public of (1) the annual performance goals for agencies’ major programs and activities, (2) the measures that will be used to gauge performance, (3) the strategies and resources required to achieve the performance goals, and (4) the procedures that will be used to verify and validate performance information. These annual plans, issued after transmittal of the President’s budget, provide a direct linkage between an agency’s longer-term goals and mission and day-to-day activities. Annual performance reports are to subsequently describe the degree to which performance goals were met. The issuance of the agencies’ performance reports, due by March 31 of each year, represents a new and potentially more substantive phase in the implementation of GPRA—the opportunity to assess federal agencies’ actual performance for the prior fiscal year and to consider what steps are needed to improve performance and reduce costs in the future.

The Department of State is the lead government agency representing U.S. interests overseas. As such, it is vested with a wide range of responsibilities, including formulating U.S. policy on diverse international issues, coordinating and implementing U.S. government programs and activities overseas, and influencing other countries to adopt policies and practices consistent with U.S. interests. The department, among other things, monitors and reports political, economic, and other developments in foreign countries; assists U.S. business overseas; provides services to U.S. citizens abroad; issues passports and visas; participates in international drug control programs; and manages facilities and provides security for U.S. personnel stationed at embassies and consulates.

Assessment of State’s Progress and Strategies in Accomplishing Selected Key Outcomes

This section discusses our analysis of State’s performance in achieving its selected key outcomes and the strategies the department has in place, particularly regarding human capital and information technology, for accomplishing these outcomes. In discussing these outcomes, we have also provided information drawn from our prior work on the extent to which State verified and validated its performance data.

It is difficult to determine the level of progress toward achieving the five key outcomes that are discussed in the performance report for fiscal year 2000. For some key outcomes, it is unclear what State was trying to accomplish, due to a lack of linkages between activity-based performance measures and outcomes and a failure to report on many of the performance indicators outlined in the plan for 2000. The department provided little discussion on why progress in meeting certain performance targets was not reported, why other performance targets were not met, and what strategies would be used to achieve the unmet (and unreported) performance targets. State’s performance plan for 2002 discusses strategies to meet performance goals and more clearly links the key outcomes to strategies and performance indicators. In addition, the department developed new performance indicators for some key outcomes. However, as we reported in June 2000, many indicators continue to be output rather than outcome oriented, and some indicators are not readily measurable.

The performance report for 2000 and the performance plan for 2002 discussed issues of data verification and validation for four of the five outcomes. In the performance report, State discussed data sources and the unit responsible for verifying data quality for four outcomes, but it did not discuss the frequency of data collection and the efforts taken to validate the data for three of those four outcomes (weapons of mass destruction, Americans traveling and living abroad, and counterterrorism). The 2002 plan discussed (1) the sources and management of performance data for four outcomes (weapons of mass destruction, Americans traveling and living abroad, international crime, and counterterrorism); (2) the frequency of data collection for two outcomes (international crime and counterterrorism); and (3) the validation of data for two outcomes (weapons of mass destruction and Americans traveling and living abroad). The department did not discuss how it will collect and validate performance data for the outcome of expanding foreign markets for U.S. goods and services.

The performance report and performance plan provided detailed discussions of departmentwide strategic human capital and information technology efforts under the Diplomatic Readiness initiatives. However, other than for a few instances—such as State’s efforts to use the Internet for disseminating information on consular services and travel safety—the report generally did not discuss these issues within the context of achieving the key outcomes. The performance plan for 2002 listed the number of staff positions the department requested for all five key outcomes, but it did not explain how changes in the number of these positions would affect the achievement of the outcomes. The performance plan included a detailed discussion of information technology issues related to providing services to American citizens traveling and living abroad, but it did not do so for the remaining four outcomes.

The Threat From Weapons of Mass Destruction

State reported it achieved one of its four planned performance targets for this outcome, made progress toward achieving a second, did not achieve a third target, and made no progress toward its fourth target. However, for the two indicators for which State reported progress, one performance target was not achieved and progress toward the other was unclear because State did not provide a full description of its actual accomplishment.

State’s strategic goal for this key outcome was to eliminate the threat to the United States and its allies from weapons of mass destruction and destabilizing conventional arms. State’s fiscal year 2000 performance plan identified one performance goal regarding the negotiation and implementation of nonproliferation treaties, agreements, and controls with Russia and the other Newly Independent States. To measure achievement of this goal, State focused on four performance indicators: (1) establishment of nonproliferation export controls in Russia and the other Newly Independent States, (2) progress on the Strategic Arms Reduction Treaty III (START III) with Russia, (3) accession of Newly Independent States to the Chemical Weapons Convention, and (4) agreement between the United States and Russia on antiballistic missile defense and theater missile defense demarcation. Under nonproliferation activities, State’s target was to have export controls in place in all 12 Newly Independent States, with a majority of these countries enforcing them. Actual performance showed that five countries (Belarus, Georgia, Moldova, Kazakhstan, and Ukraine) have the controls in place, and one (Ukraine) enforces them. State did not explain why the targets under this outcome were not met. For the second performance indicator, State reported that the United States and Russia held preliminary discussions about arms reductions under START III during 2000. While important, the result does not meet the target for 2000 of having a treaty signed by the United States and Russia. Under the third indicator, the department stated that 12 additional countries ratified or acceded to the Chemical Weapons Convention, but it is unclear whether this achievement meets the planned target of 2 additional Newly Independent States joining the convention because the department did not report the countries that joined. Finally, State reported it made no progress related to antiballistic missile defense and theater missile defense demarcation because Russia was not willing to engage in formal discussions.

The performance report discussed strategies to achieve the unmet performance goal of having all 12 Newly Independent States institute nonproliferation export controls and having the majority enforce them. The department stated that it has expanded into two distinct export control offices—one focused on policy and the other on sanctions and assistance. However, State did not report why the performance targets were not met; thus it is difficult to determine how this strategy will help the department achieve the unmet target in the future. The report did not discuss why the START III negotiations were delayed or a likely time frame for when the negotiations would be completed and the treaty signed, nor did the report discuss the actions State will take to meet its targets for theater and antiballistic missile defense demarcation.

Overall, the fiscal year 2002 plan provides a more clear definition of State’s intended strategies and performance regarding weapons of mass destruction by organizing the performance goals and indicators in a more logical fashion. State’s unified approach eliminated many goals and indicators that resulted from State’s prior year approach that was devolved to the regional level. By focusing on agencywide, rather than regional, objectives and limiting its planning to nonmandated activities, State reduced the number of performance goals for this key outcome from 75 for fiscal year 2001 to 7 for 2002 and the number of performance indicators from 64 to 15. In addition, State will continue to use two categories for judging 2002 performance for this outcome—successful and minimally successful—each of which has specific associated levels of performance. However, some of the performance indicators and targets identified continue to lack valid measures of progress. For example, one new indicator measuring performance toward missile defense defines “successful” performance as “undertak appropriate actions to implement U.S. Government decisions on missile defense.” As in prior plans, State identified the agencies with which coordination is needed for each performance goal, yet failed to discuss the nature and extent of the coordination necessary or the new strategies that may be needed.

In the performance report, State addressed data verification by stating the sources of data. The performance plan for 2002 provided the same information on data verification, in addition to the units responsible for data collection and management. However, the department did not indicate how it would verify the data. In the 2002 plan, State requested an increase of 20 full-time staff for this outcome, but it did not say how the additional positions would be allocated.

Foreign Markets for U.S. Products and Services

State reported it achieved performance targets for all seven indicators outlined in the performance plan for 2000. However, for three indicators that State reported as having been achieved, we found the department only partially met its targets. Finally, the linkage between one of the performance indicators and the outcome was unclear. State has deleted that indicator from its subsequent performance plan.

State has two strategic goals in the trade area that we treated as one key outcome: (1) opening foreign markets to increase trade and free flow of goods, services, and capital and (2) expanding U.S. exports to $1.2 trillion by early in the 21st century. Performance goals and indicators were generally objective and measurable. Under the first of the selected outcomes—opening markets—four of five indicators were results oriented. However, it was difficult to see the relationship between one performance indicator—number of countries maintaining a 90-day stock of petroleum—and the outcome of opening markets to U.S. goods and services. Performance indicators for the second selected outcome, expanding U.S. exports, were also objective and measurable. Neither the performance report nor the performance plan discussed data verification and validation for this outcome.

The performance plan indicated the department will reduce the number of staff positions by about 27 percent (256 positions), but it did not state which functions within the 2 strategic goals would be affected.

Open Foreign Markets to Increase Trade and the Free Flow of Goods, Services, and Capital

State reported that it had fully achieved its targets for four of its five performance indicators and partially met the fifth indicator. However, from our assessment, State had successfully achieved only three targets— the number of countries signing more liberal aviation agreements; the total sales of U.S. telecommunication and information technology equipment; and the total foreign operations revenues of U.S. telecommunications, including satellite services. For another indicator, the number of Newly Independent States and former Communist countries acceding to the World Trade Organization, State targeted four countries—Georgia, Croatia, Lithuania, and Moldova. State said it successfully met the target because three countries joined the organization in 2000. However, only one of the targeted countries, Georgia, joined the organization during fiscal year 2000, while another joining country, Albania, was not on State’s initial list. Croatia completed its entry requirement during fiscal year 2000, but did not formally join until November 30, 2000. Lithuania and Moldova did not join until well into fiscal year 2001. On the final indicator, the number of countries maintaining a 90-day petroleum stock reserve, we agree that State successfully met the target of having two target countries—the Czech Republic and South Korea—maintain 90-day petroleum reserves. However, the fact that seven other countries previously maintaining the desired level of oil stocks fell out of compliance with the standard negates this achievement. Regardless of whether State achieved the performance target, as we reported in June 2000, it is difficult to see a linkage between the number of countries maintaining a 90-day petroleum stock and the goal of expanding foreign markets for U.S. products and services. As such, State has dropped this indicator for both 2001 and 2002 performance.

Although State did not establish clear fiscal year 2002 performance goals for this part of the key outcome, it did discuss the strategies to achieve the outcome and the indicators and performance targets it will use to judge progress. State will continue to work toward strengthening international frameworks for open markets for goods, services, and investment; integrating social policy goals into the international agenda for reforming economic policies and rules; ensuring openness of international markets for e-commerce and biotechnology; and integrating developing and transition economies into the world economy. To monitor performance, State will focus on 11 indicators. These indicators are generally objective and measurable and linked to the outcome, but performance targets for three indicators are not clearly defined: (1) needs met in Integrated Framework for Trade-Related Capacity Building Roundtables, (2) status of bilateral investment dialogues, and (3) status of multilateral policy dialogues.

Expand U.S. Exports to $1.2 Trillion by Early in the 21st Century

State reported success in achieving both performance indicators outlined for this outcome; however, the department did not report on one of the performance targets prescribed by the plan. State’s one performance goal and two reported indicators were objective and measurable. The performance goal focused on U.S. government export promotion to the “best international market prospects,” and State reported it successfully achieved the targets for two performance indicators—the total level of U.S. exports (goods only) and the attainment of bilateral investment treaties. In developing the 2000 performance plan in 1998, State set 3 targets related to bilateral investment treaties to be achieved in fiscal year 2000: (1) sign 4 additional treaties with undetermined countries, (2) conduct 8 ongoing treaty negotiations, and (3) engage in preparatory discussions with 50 countries. The target for the number of treaties signed was actually achieved during 1999, and State concentrated its efforts in 2000 “on obtaining advice and consent from the Senate on 10 previously signed treaties and engag in negotiations with 8 countries.” The department did not indicate whether it was preparing discussions with additional countries. For the other indicator, State reported sales of $755 billion in total U.S. exports of goods, surpassing the 2000 target of $720 billion.

State’s performance goals, strategies, indicators, and performance targets outlined in the performance plan for 2002 are more clearly linked to the key outcome of expanding U.S. exports to $1.2 trillion than they were in the 2000 performance report. State’s two performance goals are to (1) provide support to U.S. businesses competing worldwide and (2) increase exports of oil and gas field equipment and services, telecommunications equipment and services, and agricultural produce. Strategies include promoting business and exports, supporting U.S. foreign direct investments, and promoting corporate responsibility and best practices. State will assess progress toward achieving the outcome using eight performance indicators that are generally objective and measurable. Although all of the indicators have specific baselines for 2000 and performance targets for 2002, most, such as the number of presentations to business audiences, are activity based. However, two indicators—(1) telecommunications and information technology equipment sales and (2) agricultural exports—are outcome oriented. A third indicator—oil and gas sector exports—is also outcome oriented; however, State provided no baseline for this indicator, and its performance target is vague and not measurable.

The Ability of American Citizens to Travel and Live Abroad Securely

State reported that it had fully met two of its performance targets for this key outcome and explained that it did not report on the two other indicators outlined by the 2000 performance plan because these indicators had become obsolete. Based on our assessment, although State made great strides toward achieving planned performance for one of the two indicators discussed, it did not fully achieve the target prescribed by the 2000 plan.

State’s strategic goal for this key outcome was to protect the safety and security of American citizens who travel and live abroad, and its performance goal for 2000 was to deliver information and services needed to travel and live abroad knowledgeably, efficiently, and courteously. This performance goal was clearly defined, results oriented, and measurable. Furthermore, State established four measurable and objective performance indicators. However, one of the key services that State provides American citizens traveling abroad—timely, effective passport issuance—is not an indicator under this key outcome. Neither the performance plan for fiscal year 2000 nor the fiscal year 2000 performance report provides performance information on this prominent activity, although the 2000 report provides a narrative description of what State considers improved passport service.

State dropped two of the four indicators prescribed by the performance plan for fiscal year 2000. The department reported that one of the indicators, status of the American Citizen Services system’s Year 2000 compliance, is no longer applicable. For the other indicator, memoranda of understanding between State and commercial airlines serving U.S. citizen passengers in the event of an air disaster, State reported that specific memoranda are no longer necessary because new legislation and regulations serve the same purpose and that the target has been successfully achieved. In the report, State did not specifically cite the relevant legislative and regulatory changes.

For the performance indicator of Internet usage to disseminate information on consular services and travel safety, State surpassed its target of receiving 150,000 hits per day on its Web page, averaging 240,000 hits per day. State also said it achieved its customer satisfaction targets for this indicator. However, it is difficult to determine whether the 2000 target for percent of users who found the information helpful was met. State reported that user satisfaction with consular Web site information was 90 percent, which was 5-percentage points below the target for 2000. State cited three primary sources of user dissatisfaction: (1) inability to get questions answered by telephone, (2) unwillingness or inability to call the passport information 900 (toll) telephone number, and (3) no listing of the winners of the Diversity Visa lottery. However, two of the reasons cited reflected dissatisfaction with telephone-based customer service rather than Web site content. Nonetheless, State said it met the target and therefore all aspects of the indicator. Further complicating the issue, State did not provide data on the relative influence the three sources of dissatisfaction had on the overall user satisfaction rate. Thus, the explanations seem incomplete.

State also reported that it successfully met the final performance target, reducing the case workload per officer to left-behind parents in child abduction cases from 140 cases to 80, despite an actual reduction to 90 cases per worker. State officials told us they declared success for this performance indicator because the reduction was significant. However, State provided no explanation of why it was not successful in reducing the workload to 80 cases per officer. The department stated that progress toward achieving this target was due in large part to increasing the number of full-time staff from 11 to 26.

In addition, State’s strategy for enhancing American citizen services under the performance plan covering fiscal year 2000 indicated that the department would work closely with foreign governments to improve treatment of U.S. citizens incarcerated abroad, promote greater participation in multilateral treaties to which the United States is a party, and help other agencies provide efficient distribution of benefits overseas. Yet State provided no indicators or targets to measure progress in these areas. The 2000 performance report does not include these activities as strategies State used in fulfilling this goal. The plan also states that ratification of the Hague Convention on International Adoptions was expected and that State must make preparations for dealing with the increased workload the convention would entail. But State did not report on what preparations, if any, had been made for the convention’s ratification or a resulting increase in workload.

Although one of the key services State provides American citizens traveling abroad is the timely and effective issuance of passports, neither the 2000 performance plan nor the 2000 performance report provides performance information on this service. However, the 2000 report said that a governmentwide satisfaction survey showed that the service passport customers received exceeded their expectations and that the majority of customers were more satisfied than they were in 1998.

Under its performance plan for 2002, State replaced its past indicator for child abductions—reducing the workload level of caseworkers—by focusing on a new, automated case-tracking system. Performance targets for 2002 include pilot testing and deploying systems that track international parental child abduction cases and international adoption cases. In addition, the system will track U.S. citizens who have been arrested overseas and instances of their mistreatment. State also will report on two passport service indicators: the number issued and the status of efforts to introduce digital passport photos. Finally, State will continue to track consular information activities through the use of the Consular Activities Web site. State will continue to use two alternative levels of measure—successful and minimally effective—for the indicators associated with this outcome. For each performance indicator, the alternative levels of measure have specific results-oriented targets.

In the performance report, State addressed data verification by stating the sources of performance data and the units responsible for data collection and management. The performance plan for 2002 also provided a brief description of how and by whom the data will be verified and validated.

International Crime and the Availability and/or Use of Illegal Drugs

State reported success in achieving performance targets associated with three indicators presented in the 2000 performance plan. However, overall progress under this outcome is unclear because the department did not discuss progress toward achieving the planned targets for three additional indicators prescribed by the 2000 plan. In addition, the linkages between the outcome and some of the strategies and indicators are unclear, and the indicators on which the department reported were output rather than outcome oriented.

State’s 2000 performance plan and prior GPRA products treated international crime and drug trafficking as one strategic goal: to minimize the impact of international crime on the United States and its citizens and reduce significantly, from 1997 levels, the entry of illegal drugs into the United States. State’s performance report for 2000 presented one performance goal covering both key outcomes and addressed only the indicators related to international law enforcement training programs outlined in the performance plan for fiscal year 2000. The report did not discuss performance results for its remaining three indicators outlined in the 2000 plan: (1) foreign spending on and commitment to crime control efforts, (2) agreements to counter international organized crime, and (3) antinarcotics awareness campaigns. The report also did not present indicators related to the flow of illegal drugs into the United States. No explanation was provided on why the other indicators and measures were excluded.

For the international training program indicators, the performance plan for fiscal year 2000 identified three performance targets; however, the performance report for fiscal year 2000 addressed only two—the number of academies established and the number of foreign officials trained. State gave no explanation on why “help countries establish their own national-level law enforcement training institutions” was not discussed. Furthermore, the report did not demonstrate how establishing training academies and training foreign officials help State combat international crime and reduce the availability of illegal drugs.

State reported that targets for the two performance indicators related to law enforcement training were successfully achieved. The department stated that it completed development of two international law enforcement academies but only cited one partnered country—Botswana.

State also worked with the U.S. Agency for International Development and the Department of Justice’s Office of Overseas Prosecutorial Development, Assistance, and Training, as we reported in October 2000 and April 2001, in addition to other U.S. agencies and host countries, to achieve its target of training 10,500 foreign officials in law-related areas. Both of these indicators were output oriented, and State did not demonstrate tangible effects of training on reducing international crime.

The 2002 performance plan treats international crime and reducing entry of illegal drugs into the United States as two separate strategic goals. This approach provides more clarity and better linkages between State’s performance goals and the outcomes than previous performance plans. To minimize the impact of international crime on the United States and its citizens, State has two general goals—strengthening international cooperation against transnational crime and improving law enforcement and criminal justice institutions in targeted countries. To achieve these goals, State plans to continue using training-related items as strategies. In addition, State plans to work with other countries to develop new anticrime and antismuggling/trafficking operations, begin negotiating an international anticorruption agreement, and work with multilateral agencies to implement the Transnational Organized Crime Convention and related protocols. While some of the associated indicators are not easily quantifiable, performance requirements are generally results oriented.

To achieve the objective related to reducing the entry of illegal drugs into the United States, State plans to employ numerous strategies to reduce the cultivation of illicit coca and opium poppy and foreign cultivation of marijuana. These strategies are generally results oriented, and the indicators tracking progress are objective, measurable, and directly related to the performance goal and key outcome. Strategies related to a second goal—criminal justice sectors of foreign governments break up major drug trafficking organizations and effectively investigate, prosecute, and convict major narcotics criminals—are aimed at diplomatic initiatives to get foreign countries to approve and implement the 1988 United Nations Drug Convention, as well as assist others with legislative, training, and law enforcement activities. The two performance indicators—the number of countries adhering to the convention and the number of foreign officers trained—are objective and measurable and clearly linked to the performance goals and key outcome; however, they apply to only two of State’s many planned activities.

In the performance report, State addressed data verification by discussing the sources of performance data, the frequency of data collection, the units responsible for data collection and management, and how the data were validated. Although the performance plan for 2002 addressed the first three items for data verification, it did not discuss how the data would be validated.

The performance plan stated that the department will increase staffing for its anticrime efforts by 18 percent (76 positions), but it did not provide specifics on how these positions will be allocated. The plan also stated that the number of positions for combating the illegal flow of drugs would be reduced by about 22 percent (28 positions), but it did not state how this would affect activities associated with this strategic goal.

International Terrorist Attacks

The events of September 11, 2001, heightened the awareness and importance of this outcome and could affect the strategies the department utilizes to combat terrorism. While the events have no bearing on fiscal year 2000 results that the department presented in its performance report, they could lead to changes to State’s fiscal year 2002 counterterrorism performance goals, strategies, and indicators. The department’s 2002 performance goals, strategies, and indicators were generally results oriented; however, performance targets for the some of the indicators were not well defined. Although the goals, strategies, and indicators remain relevant even after the September 11 events, State officials said they plan to revise the performance targets and may consider altering, replacing, and adding some strategies and indicators.

State reported that it successfully achieved its performance targets on all indicators prescribed by the performance plan for fiscal year 2000. However, progress toward achieving the outcome is difficult to determine because linkages between the outcome and its strategies and indicators were not clear. In addition, indicators were output rather than outcome oriented. Moreover, the department did not report on two basic measures associated with the outcome—the number and severity of terrorist attacks.

State’s strategic goal for this key outcome was to reduce the incidence and severity of international terrorist attacks, particularly against American citizens and interests. For fiscal year 2000, State reduced the number and type of performance goals (from two in 1999 to one in 2000) and performance indicators (from seven in 1999 to five in 2000), as outlined in the department’s performance plan for 2000. The outcome, performance goal, and indicators were objective and measurable; however, the performance goal and performance indicators were output rather than outcome oriented. The department’s objective was to reduce the severity and incidence of international terrorist attacks against American citizens and interests by minimizing human casualties, physical destruction of property, and the political effects of terrorism. To accomplish this, State focused predominantly on training—altering the performance goal to “train foreign governments in methods and techniques of combating terrorism.” All five of the indicators and performance targets involved either holding conferences, coordinating interagency terrorism response exercises and deployments, or training personnel. State reported it achieved each of the performance targets. Although these measures are outputs, some do operate as proxies for outcomes. For example, the 2000 performance report’s number of exercises and deployments are outputs that serve as proxies to outcomes—in this case, high readiness to respond to international terrorist incidents.

The fiscal year 2000 report lacks the single most important and easily quantifiable performance indicator for this performance goal: the number of terrorist attacks, especially against American citizens. These figures are readily available and published annually by the Department of State. The fiscal year 1999 report included “level of terrorist attacks” as a performance indicator, but it only discussed the trends in such attacks, not the actual figures. The fiscal year 2000 report dropped the level or number of attacks as a performance indicator.

For 2002, State’s performance goals are to reduce the number of attacks, bring terrorists to justice, reduce or eliminate state sponsored terrorist acts, delegitimize the use of terror as a political tool, enhance international response, and strengthen international cooperation and operational capabilities to counter terrorism. To achieve these goals, State has developed strategies that strengthen international conventions and resolutions and U.S. laws; isolate sources of support, including freezing financial assets; and train foreign officials in counterterrorism through bilateral exercises. State will measure progress in 2002 using seven new indicators. State said it will report on the number of terrorist attacks and the rate of deaths and casualties against baseline data for fiscal year 2000, with performance targets to reduce the levels of each. The department also established an indicator and baseline for the number of countries signing and implementing international treaties on counterterrorism; however, part of the performance target for this indicator is unclear, focusing on “increased effective recourse to treaties for extraditions and renditions.” Two indicators—the level of cooperation and coordination among friendly governments and the level of professionalism, training, and quality of key equipment in key friendly countries—are not easily measurable. State established baselines for each, stating the current level is fair or good, but it does not explain what is meant by these categories, nor how one measures “increased professionalism” or “improved coordination.” State used a similar baseline for another indicator, the successful collection of evidence and obtaining witness cooperation for prosecutions under U.S. laws, but the performance target—increased use of foreign collected evidence and witnesses in U.S. prosecutions—seems quantifiable and, we believe, should have a consistent baseline.

Due to the sensitive nature of the information for the remaining two indicators—(1) identification of terrorist threats and prevention of terrorist attacks and (2) the level of fund-raising activity on behalf of terrorist groups—State reported no baseline data. Performance for the former indicator will be measured by assessing the number and lethality of attacks. The target for the latter indicator is a reduction in funds raised through front organizations.

In the performance report, State addressed data verification by discussing the sources of performance data and the units responsible for data collection and management. The performance plan for 2002 also included a statement of the frequency of data collection, but it did not discuss how the data would be validated. The performance plan also stated that the number of staff positions for this outcome would be reduced by 23 percent (58 positions), but it did not state which areas would be affected by the cuts.

Comparison of State’s Fiscal Year 2000 Performance Report and Fiscal Year 2002 Performance Plan With the Prior Year Report and Plan for Selected Key Outcomes

For the selected key outcomes, this section describes major improvements or remaining weaknesses in State’s (1) fiscal year 2000 performance report compared with its fiscal year 1999 report and (2) fiscal year 2002 performance plan compared with its fiscal year 2001 plan. It also discusses the degree to which State’s fiscal year 2000 report and fiscal year 2002 plan address concerns and recommendations by the Congress, GAO, and State’s Inspector General.

Comparison of Performance Reports for Fiscal Years 1999 and 2000

Despite efforts to improve the clarity and readability of its performance report, State’s report for 2000 suffered from many of the same weaknesses of the 1999 report. Two major weaknesses are that (1) many indicators continue to be activity based rather than outcome oriented and (2) relationships between some key outcomes and strategies and indicators remain unclear. In addition, State changed some indicators, or the methodologies for measuring them, with little or no discussion on why it did so; and like 1999, the department failed to report on many of the performance indicators defined by its plan for 2000. Moreover, State did not discuss strategies for addressing unmet (and unreported) performance targets and did not provide assessments of how performance in fiscal year 2000 could affect estimated performance levels for 2001.

Little Connection Between State’s Actions and the Achievement of Key Outcomes

The 2000 performance report continues to rely on activity- or output-based indicators, rather than outcome-oriented indicators; and State did not establish connections between its actions and the success or failure of key outcomes. We recognize that the nature of State’s mission and operating environment makes it difficult for the department to avoid either having very broad key outcomes that are then not addressed sufficiently by the indicators or broad indicators that make it difficult to assess State’s role in the outcome. Given this difficulty, we find it commendable that most reported indicators are quantifiable and that all reported indicators have baselines and targets. Nevertheless, as we suggested in 1999, there are opportunities to better describe State’s outputs and clarify how they affect outcomes. For example, assessing whether terrorist attacks were reduced requires either a comparison of the relative number of terrorist attacks or a count of instances when specific attacks were prevented. In the 1999 report, one of the indicators for this issue was the “level of terrorist attacks against American citizens and interests.” This indicator provided a direct measure of whether the goal was met (outcome), but it is unclear on what action the United States took (output) to influence the outcome. Rather than linking an action or strategy to the indicator, as noted above, in its 2000 report State did not report the number or severity of terrorist attacks.

Unclear Relationships Among Key Outcomes, Strategies, and Indicators

The relationships between some strategies and key outcomes were not always apparent. For example, State does not clearly explain how some strategies, such as establishing law enforcement academies in southern Africa and New Mexico, related to the key outcome of minimizing the impact of international crime and stemming the flow of illegal drugs. State also reported on some indicators that were not clearly linked to key outcomes. For example, it is not clear how a nation maintaining a 90-day stock of petroleum reserves becomes a more open market for U.S. goods and services. In addition, reducing the caseload of overseas child- abduction investigators appears more closely related to reducing the impact of international crime on American citizens than providing efficient, knowledgeable, and courteous services to Americans traveling and living abroad.

Furthermore, the relationship between performance indicators and strategies were not always clear. In particular, it is unclear whether each strategy is designed to pursue specific indicators under a performance goal or whether the strategies are part of a more general effort. For example, three of the five indicators for the key outcome, “opening foreign markets” to U.S. goods and services, related to a single strategy— ”advancing civil aviation, transport, telecommunications, and energy initiatives”—while most of the remaining nine strategies had no readily associated indicators.

Little or No Explanation for Changes to Performance Indicators

Some indicators to measure performance in 1999 were changed in 2000 without explanation, which makes it difficult to track performance from one year to the next. For some indicators used in 1999, measurement methodologies were altered with no discussion in either the performance plan or performance report for 2000, resulting in orders of magnitude differences. For the indicator “sales of U.S. telecommunications and information technology equipment,” State’s 1999 performance report showed a 1998 baseline of $18.9 billion in sales and a 1999 target of $20.6 billion, but it did not provide actual performance for 1999. For the same indicator in 2000, State reported a 1998 baseline of $111 billion in sales, a 1999 actual level of $123 billion, a 2000 target of $120 billion, and a 2000 actual level of $140 billion. It is apparent that State changed the methodology for assessing this indicator, but the department did not explain why it did so.

In addition, State changed past performance results for some indicators. For example, regarding expansion of markets for U.S. products and services, the actual level for the total exports of goods for 1999 cited in State’s 1999 performance report was $693 billion. However, the 2000 performance report adjusted the 1999 actual level for the total exports of goods to $673 billion. State provided no explanation for this change. If the fiscal year 2000 report’s actual level for 1999 is correct, then State did not reach its 1999 performance target of $685 billion that it reported it had successfully met in its 1999 performance report.

Comparison of Performance Plans for Fiscal Years 2001 and 2002

In contrast to the performance report, State’s 2002 plan represents a major improvement over 2001 by (1) focusing more on the overall agency level, rather than upon geographic regions; (2) eliminating redundant performance goals and indicators, and goals and indicators that were not clearly linked to outcomes; and (3) associating performance with necessary resources. However, the plan still has certain weaknesses. Many performance goals and indicators continue to be activity based rather than outcome oriented, and some performance indicators remain vague and unmeasurable. Furthermore, although State listed its partner agencies, offices, and bureaus; it did not explain its roles and responsibilities in these partnerships.

State’s performance plan for 2002 uses a more unified, agencywide approach than the regional focus used in the 2001 plan. This approach resulted in more clarity and the elimination of redundant material. State agreed with our 2000 assessment that the goals and objectives in the 2001 performance plan were overly scattered. State formed key outcome teams that identified performance goals for the department, rather than have the various bureaus set their own goals. Using this approach, State achieved a drastic reduction in the number of performance goals and indicators across all outcomes. For example, in 2000, we reported that State’s plan for 2001 had 75 performance goals and 64 performance measures over 5 regions for its efforts to reduce threats from weapons of mass destruction. In the 2002 plan, State reduced the number of performance goals and measures to 7 and 15, respectively. By deleting redundancies, combining related or subsidiary goals and indicators, and eliminating goals and indicators not clearly linked to key outcomes, State devised a performance plan for 2002 that is more focused and is more useful for understanding its intentions and for monitoring progress against baselines.

Another improvement over the plan for fiscal year 2001 is the inclusion of a resources summary for the selected outcomes. The performance plan for 2002 presented a summary chart showing the actual funding and number of positions for fiscal year 2000 and the estimated funding and number of positions for fiscal years 2001 and 2002 for each key outcome. However, the plan did not indicate how these resources would be applied to the various goals and strategies within the outcomes.

Nonetheless, some deficiencies remain with State’s 2002 plan. Many indicators continue to be activity based rather than outcome oriented. For example, regarding expansion of U.S. exports by $1.2 trillion, five of eight indicators describe outputs of activities, four of which tallied the number of meetings, conferences, and training opportunities, while the fifth indicator measured the status of the launching of an intranet Web site. In addition, a number of performance targets are vague and not measurable. For example, the target for “status of bilateral investment policy dialogues” for the opening of foreign markets outcome is undefined (“measurable progress on improvement of investment climates”). Similarly, it is unclear how progress will be measured for the counterterrorism indicator “level of coordination among friendly governments” (“improved coordination in countries, including close allies, countries with major terrorist presence, and those receiving [antiterrorism and Foreign Emergency Support Team] training”). In addition, it will be difficult to measure a “eepening of new energy investment. . . resulting in increased flows of energy equipment and service exports from U.S. firms” for oil and gas exports, even assuming a baseline for judging was provided. A better performance target would be the total value of oil and gas sector exports.

Finally, although State provides a list of the agencies, offices, and bureaus working together to achieve the outcomes, it does not explain how the organizations interact or their respective roles. State’s role in establishing international law enforcement academies and increasing the technical capabilities of law enforcement and judicial institutions, for example, is not as apparent as the roles played by some of its partnered agencies, such as the Department of Justice. Although State may be involved in negotiating agreements for establishing law enforcement academies and other training institutions, other organizations, such as the Department of Justice’s Office of Overseas Prosecutorial Development and Assistance Training, actually provide the training services.

State’s Efforts to Address Its Major Management Challenges Identified by GAO

We have identified two governmentwide high-risk areas: strategic human capital management and information security. Regarding human capital, we found that State’s 2002 performance plan has performance goals and measures related to human capital, and its 2000 performance report explained its progress in resolving human capital challenges. The department reported that one of its performance goals for 2002 is to be staffed with a fully skilled workforce. State reported the number of employees in training and related programs and other efforts to better prepare its existing workforce, declaring these efforts to be successful. However, it did not address efforts to recruit new staff, including possible incentives for information technology and other hard-to-attract staff. Regarding information security, we found that State’s performance plan has performance goals and measures related to information security, and the department’s performance report explained its progress in resolving its information security challenges. State said that, primarily based on our recommendations, it has undertaken efforts to improve information technology security to close several material weaknesses. However, we have not determined the effectiveness of these programs. Also, the independent auditor who reviewed State’s 2000 principal financial statements said that implementing our recommendations does not necessarily mean that the related material weaknesses have been corrected.

In addition, we have identified five other major management challenges facing State: (1) enhancing overseas communications, information technology, and knowledge management; (2) rightsizing U.S. overseas presence; (3) improving the security of overseas facilities and enhancing the management of overseas security programs; (4) improving financial management capabilities; and (5) improving visa processing and enhancing border security. We found that the department’s performance report discussed its progress in addressing all of these challenges. Of these major management challenges, State developed performance goals and measures for 2002 that are directly related to four of these challenges. State did not detail performance goals, strategies, or measures to address one of these challenges—rightsizing U.S. overseas presence—in the performance plan. For that challenge, State reported it had been unable to achieve a consensus with other agencies, which is essential since rightsizing is an interagency effort. However, in August 2001, State reported on actions it plans to take to address the recommendations of the Overseas Presence Advisory Panel. Moreover, the President identified rightsizing overseas presence as 1 of 14 initiatives in his management agenda for fiscal year 2002.

Conclusions

Weaknesses in State’s fiscal year 2000 performance report made it difficult to determine the department’s progress toward achieving the key outcomes. These weaknesses are rooted in performance goals and indicators established in its performance plan for 2000, which was prepared in 1998 and which we have criticized in a prior report. State acknowledged flaws in its previous GPRA products and sought to correct many of them in its performance plan for 2002. State has taken a major step toward implementing GPRA requirements by producing a fiscal year 2002 plan that is superior to earlier efforts. Although we identified several areas where improvements can be made, such as developing indicators that are focused on results instead of outputs, we believe that the improved plan lays the groundwork for State to provide a more clear assessment of its performance and progress next year. In its future performance report, State will need to focus its efforts on reporting on all indicators in the plan and, if targets are not achieved, providing clear explanations of the reasons why and what actions State plans to achieve the targets in the future.

Recommendations

We recommend the Secretary of State ensure that performance reports for 2001 and subsequent years (1) address all performance goals and indicators outlined in State’s performance plans, (2) provide clear and specific explanations for why performance goals and performance targets were not met, and (3) outline the actions the department will take to achieve unmet goals.

Scope and Methodology

As agreed, our evaluation was generally based on (1) the requirements of GPRA, (2) the Reports Consolidation Act of 2000, (3) guidance to agencies from OMB for developing performance plans and reports (OMB Circular A-11, part 2), (4) previous reports and evaluations by us and others, (5) our knowledge of State’s operations and programs, (6) our identification of best practices concerning performance planning and reporting, and (7) our observations on State’s other GPRA-related efforts. We discussed our review with officials in the Office of Management Policy and Planning, which is responsible for preparing the performance reports and plans required by GPRA. We also discussed our work with State’s OIG. The department outcomes that were used as the basis for our review were identified by the Ranking Minority Member of the Senate Committee on Governmental Affairs as important mission areas for the department and generally reflect the outcomes for all of State’s programs or activities. The major management challenges confronting State, including the governmentwide high-risk areas of strategic human capital management and information security, were identified by us in our January 2001 performance and accountability series and high-risk update and by State’s Office of the Inspector General in December 2000. We did not independently verify the information contained in the performance report and plan, although we did draw from our other work in assessing the validity, reliability, and timeliness of State’s performance data. We conducted our review from April 2001 through November 2001 in accordance with generally accepted government auditing standards.

Agency Comments

We obtained written comments on a draft of this report from the Department of State (see app. II). The department indicated that it is taking action to improve its GPRA processes and products and would implement our recommendations in its fiscal year 2001 performance report. The department also provided technical comments, which we have appropriately incorporated into this report.

As arranged with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. At that time, we will send copies to appropriate congressional committees; the Secretary of State; and the Director, OMB. Copies will also be made available to others upon request.

If you or your staff have any questions, please call me at (202) 512-4128.

Appendix I: Observations on the Department of State’s Efforts to Address Its Major Management Challenges

The following table identifies the major management challenges confronting the Department of State, including the governmentwide high- risk areas of human capital and information security. The first column lists the challenges identified by our office and State’s Office of the Inspector General (OIG). The second column discusses what progress, as discussed in its fiscal year 2000 performance report, State made in resolving its challenges. The third column discusses the extent to which State’s fiscal year 2002 performance plan includes performance goals and measures to address the challenges that we and OIG identified. We found that State’s performance report discussed State’s progress in resolving most of its challenges, but it did not discuss State’s progress in resolving the following two challenges: (1) enhancing emergency preparedness and conducting crisis management exercises and (2) correcting weaknesses in federal financial assistance management. Of the agency’s 10 major management challenges, its performance plan for fiscal year 2002 included goals and measures applicable to 5 of the challenges and partly applicable to 3 additional challenges. The plan does not address the remaining two challenges and therefore does not include goals and measures for them.

Appendix II: Comments From the Department of State

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to Mr. Brummet, Claude Adrien, David G. Bernet, Stephen Caldwell, Janey Cohen, Albert Huntington III, Cynthia Jackson, Edward Kennedy, Beth Sirois, and Eve Weisberg made key contributions to this report.