Nonproliferation and Disarmament Fund: State Should Better Assure the Effective Use of Program Authorities
Released: 2012-11-30T00:00:00

Letter

Background

Congress authorized the President to establish NDF in 1992 under section 504 of the FREEDOM Support Act. The legislation authorized the President to use NDF to promote a variety of bilateral and multilateral nonproliferation and disarmament activities. In 1994, the President delegated authority for the program to the Secretary of State, who subsequently delegated authority for the program to the Under Secretary of State for Arms Control and International Security. The NDF office, within ISN, is responsible for day-to-day management of the program. The NDF Director leads the office, which has a staff of 16 people, including both State officials and contract employees.

Congress funds NDF annually through the Nonproliferation, Anti- terrorism, Demining, and Related Programs appropriations account, within the Foreign Operations, Export Financing, and Related Programs Appropriations Acts. NDF received $10 million in initial funding for fiscal year 1994. Since fiscal year 1994, NDF has received $597 million in total appropriations. From fiscal years 2007 through 2012, NDF appropriations ranged from a high of $118 million in fiscal year 2009 to a low of $30 million in fiscal year 2012. According to State, NDF is unusual among U.S. foreign assistance programs in that it does not request funding for specific activities as part of its annual Congressional Budget Justification. The NDF Director stated that this helps ensure that NDF has the flexibility to respond to nonproliferation and disarmament opportunities as they arise, rather than tying NDF funds to particular projects or locations in advance.

NDF Authorities

The FREEDOM Support Act provided NDF with a broad mission to fund bilateral and multilateral nonproliferation and disarmament activities and annual appropriations bills have consistently granted NDF other key authorities. NDF has used its authorities under the FREEDOM Support Act to fund a diverse set of projects. Table 1 outlines NDF activities authorized by the FREEDOM Support Act and provides examples of the types of activities NDF has funded. State officials and NDF program documents have characterized NDF’s mission as focused on funding unanticipated or unusually difficult projects of high priority to the U.S. government.

Figure 1 illustrates the dismantling of a Scud missile as part of an NDF- funded project in Ukraine.

In addition to the authorities granted to NDF in the FREEDOM Support Act, annual appropriations bills have also consistently provided NDF with three key authorities that are designed to increase NDF’s flexibility in carrying out nonproliferation and disarmament activities around the globe, as opportunities arise. These include the authority to (1) undertake projects notwithstanding other provisions of law (notwithstanding authority); (2) implement projects anywhere in the world or through international organizations when it is in the national security interest of the United States to do so, notwithstanding provisions of the FREEDOM Support Act that limited certain NDF activities to the independent states of the former Soviet Union (FSU) (geographic authority); and (3) use funding without restriction to fiscal year (no-year budget authority).

NDF Project Review and Approval Process

State uses a multistep process to review NDF project proposals and determine which projects to fund, as shown in figure 2.

According to NDF officials, NDF does not typically develop its own project proposals. Rather, other agencies, such as DOD and DOE, and other State offices, such as the Office of Export Control Cooperation, submit project proposals. NDF’s Review Panel, which is chaired by the Assistant Secretary of State from ISN, reviews these proposals. Two ISN Deputy Assistant Secretaries of State and the Assistant Secretaries of State from the Bureau of Political-Military Affairs and the Bureau of Arms Control, Verification, and Compliance serve as the other voting members on the panel. Officials from other U.S. agencies, including DOD, DOE, OMB, the Department of Commerce, and the Department of Homeland Security, as well as representatives from the National Security Council and U.S. intelligence community, also attend panel meetings. After reviewing the project proposals, the voting members of the NDF Review Panel make recommendations to State’s Under Secretary for Arms Control and International Security to approve, deny, or defer projects. In the Review Panel meetings, members can also propose modifications, such as increasing or decreasing the amount of funding for a project. The Under Secretary has the final authority to approve a project. NDF officials stated that in certain cases—for example, if a project is particularly urgent—the NDF Review Panel may not formally meet to review a proposal before it is submitted to the Under Secretary. In those cases, NDF instead may discuss the proposal with other Review Panel agencies in a different venue, such as at a National Security Council meeting.

After the Under Secretary approves a project, but before work begins, State provides a 15-day advance notification to Congress to inform it of State’s intent to begin work on the project. As part of the notification, NDF informs Congress of its intent to obligate a specified amount of funds on the project. NDF then considers these funds designated for that project and not available for use on other projects, unless a subsequent notification is made.

NDF Projects

From fiscal years 1994 through 2012, NDF notified Congress of its intent to initiate work on 179 projects. NDF subsequently cancelled 19 of these projects after their notification and put an additional project on hold because of congressional concerns. As of the end of fiscal year 2012, NDF had 33 active projects. NDF also reported that, as of the end of fiscal year 2012, it had an additional 42 projects for which work was completed or cancelled, and the financial review of the projects was finished. In accordance with NDF close-out procedures, NDF is in the process of seeking approval from the Under Secretary for Arms Control and International Security before officially closing them.

Since the beginning of fiscal year 2007, NDF has notified Congress of its intent to initiate work on a total of 24 projects, with a high of 15 in fiscal year 2010 and a low of zero in fiscal year 2011. Figure 3 shows the number of congressionally-notified projects from fiscal years 2007 through 2012.

NDF funding amounts for projects vary significantly. NDF has notified Congress of its intent to spend as much as $50 million to as little as $179,000 on individual projects initiated since the beginning of fiscal year 2007. The lengths of projects also vary. For example, since the beginning of fiscal year 2007, NDF has closed out projects that were completed in as little as a few months to more than 9 years. In addition, some projects are follow-up projects that build on projects initiated in earlier fiscal years. For example, beginning in fiscal year 1998, NDF has undertaken five separate projects—the most recent of which was initiated in fiscal year 2010—to assist the government of Kazakhstan in shutting down a nuclear reactor in Aktau.

NDF divides its projects into four categories: (1) destruction and conversion, (2) safeguards and verification, (3) enforcement and interdiction, and (4) education and training. Since the beginning of fiscal year 2007, State has committed the most resources to projects in the destruction/conversion category. In fiscal years 2007 through 2012, 39 percent of NDF funding for new projects went to projects in this category. Figure 4 shows a breakdown of funding for NDF among the four project categories, as well as administrative expenses, for fiscal years 2007 through 2012.

NDF’s Authorities Provide It Significant Operational Flexibility, but under Its No-Year Budget Authority, NDF Has Not Determined Needed Carryover Balances

NDF has several key authorities that provide it significant operational flexibility; however, it has not determined its needed carryover balances and it has taken years to close out many of its projects in the absence of guidance for closing them. Annual appropriations bills have consistently provided NDF with three key authorities that it has used to carry out its activities. First, NDF has used its notwithstanding authority to fund projects in countries where other U.S. programs are barred from operating by U.S. sanctions or other legal restrictions. Second, NDF has used its geographic authority to fund projects in a range of countries around the globe. Third, NDF has used its no-year budget authority to carry over balances not designated for specific projects from one year to the next. However, NDF has not determined appropriate levels for these balances, which have increased significantly in the past several years. Additionally, NDF has taken many years to close some projects where work was never started, or was suspended, and has not established guidance for determining when inactive projects should be closed out and unexpended no-year funds made available for other projects.

NDF Used Its Notwithstanding Authority to Implement Several Projects Where Laws and Regulations Otherwise Restricted U.S. Assistance

Annual appropriations acts have consistently granted NDF notwithstanding authority, which allows NDF to undertake projects “notwithstanding any other provision of law.” As a result, NDF has the ability to fund projects in countries where other U.S. programs are generally barred from operating by U.S. legal restrictions. For example, when North Korea agreed to the disablement of its Yongbyon nuclear reactor in 2007 after progress in diplomatic talks, NDF was able to fund the project because of its notwithstanding authority, while other U.S. agencies, such as DOD and DOE, could not because various U.S. legal restrictions limited the assistance they could provide the country.

According to State officials, NDF’s broad notwithstanding authority is uncommon among U.S. government programs. For example, the 2010 National Defense Authorization Act provided DOD’s Cooperative Threat Reduction (CTR) program notwithstanding authority for the first time in the program’s history and granted only limited use of the authority.cannot use its notwithstanding authority for more than 10 percent of CTR’s appropriation for a given fiscal year and must meet other requirements before exercising the authority, such as obtaining concurrence from the Secretaries of State and Energy.

When seeking to use its notwithstanding authority, NDF requests approval from the Under Secretary for Arms Control and International Security. According to the NDF Director, when NDF was established, State decided that NDF’s notwithstanding authority should, as a matter of policy, be approved at the Under Secretary level, rather than at a lower level. intent to use the authority as part of the 15-day congressional notification process. Although U.S. law does not require that State inform Congress of NDF’s use of its notwithstanding authority, the conference report accompanying the fiscal year 2012 Consolidated Appropriations Act directed the Secretary of State to notify the Committees on Appropriations in writing, within 5 days of exercising NDF’s notwithstanding authority. The conference report also directed that the notification include a justification for the use of the authority.

State noted that legally, notwithstanding authority applies to NDF funds by the terms of the legislation and does not require a formal determination to rely upon this authority.

Secretary for approval given the sensitive nature of the projects. In the final three cases, NDF requested the use of its notwithstanding authority for classified projects whose details cannot be publicly reported.

In those cases where NDF requested the use of its notwithstanding authority to overcome specific laws or regulations, it identified several different legal restrictions it needed to overcome. For example:

NDF requested the use of its notwithstanding authority to initiate work on a project in Libya in fiscal year 2012. Among other things, the authority was required to overcome restrictions on U.S. security assistance to countries that engage in a consistent pattern of gross violations of human rights.

In the case of two projects at the Yongbyon site in North Korea, NDF requested the use of its notwithstanding authority to, among other things, overcome “Glenn Amendment” restrictions within the Arms Export Control Act. The Glenn Amendment triggers U.S. sanctions if the President determines that a nonnuclear country (as defined by the Nuclear Nonproliferation Treaty) has detonated a nuclear explosive device.

NDF also requested the use of its notwithstanding authority to overcome a restriction that the Foreign Assistance Act would have imposed on a project in Iraq. The Act restricts U.S. assistance to countries that have severed diplomatic relations with the United States and which have not entered into a new bilateral assistance agreement once diplomatic relations have resumed. At the time of the project, there were concerns regarding the status of the United States’ bilateral agreement with Iraq.

In addition to using its notwithstanding authority to bypass restrictions on U.S. assistance to particular countries, NDF has in some cases also used its notwithstanding authority to overcome laws and regulations pertaining to contracting and acquisitions. For example, NDF used its notwithstanding authority on some contracts to overcome Federal Acquisition Regulation (FAR) competition requirements, according to a 2004 State Inspector General report. Additionally, a 2009 National Academies of Science report examining options for strengthening and expanding DOD’s CTR program noted that, because of its notwithstanding authority, NDF is not subject to contracting requirements, including the FAR, which CTR must follow. The report noted that this ability may allow NDF to undertake certain projects more quickly and at less expense than CTR. However, according to State officials, NDF has not typically used the program’s notwithstanding authority to bypass federal contracting laws and regulations. State officials said that, while NDF has almost always relied on sole-source bids, rather than a competitive bidding process, it primarily selected contractors to implement projects using existing flexibilities in the law and regulations available to all agencies. For example, State officials stated that NDF has relied on provisions in the FAR that permit sole-source contracts in situations where there is an urgent and compelling need.

In addition to competition requirements, some NDF officials stated that NDF may use its notwithstanding authority to bypass other types of acquisition requirements, such as “Buy America” provisions. For example, one NDF official stated that to expedite work on NDF’s project at the Yongbyon reactor in North Korea, NDF purchased some of the equipment used from China.

NDF’s Geographic Authority Has Allowed It to Fund Projects around the World

Since 1994, annual appropriations acts have provided NDF with broad geographic authority to fund projects worldwide as nonproliferation and disarmament opportunities arise. NDF’s geographic authority allows it to fund projects outside the states of the FSU if the Under Secretary for Arms Control and International Security makes a determination that it is in the national security interest of the United States to do so. NDF’s authority to fund projects globally since the program’s start in 1994 is in contrast to the authorities of some other U.S. nonproliferation programs. For example, DOD’s CTR program was not authorized to fund any projects outside the FSU until the passage of the Fiscal Year 2004 Defense Authorization Bill and continued to face various restrictions on conducting work outside the FSU until 2007. In addition, as we reported in December 2011, many of DOE’s Defense Nuclear Nonproliferation programs, which originated in the early 1990s following the dissolution of the Soviet Union, have focused primarily on improving nuclear security in Russia. The NDF Director stated that, while NDF has been used to supplement projects in the FSU or to fill emergency gaps, its primary emphasis has always been on other parts of the world.

Since 1994, NDF has used its geographic authority to fund projects in Central and South America, North and Sub-Saharan Africa, Eastern and Western Europe, the Balkans, the Middle East, and Asia. As shown in figure 5, NDF has funded projects in several different countries since the beginning of fiscal year 2007, including Afghanistan, Egypt, Kazakhstan, North Korea, and Ukraine, among others. It has also funded a limited number of projects in the United States, including the construction of training facilities at DOE’s Hazardous Materials Management and Emergency Response site in Washington State for the purpose of training foreign nationals.

Over the life of the program, Congress has consistently granted NDF no- year budget authority in annual appropriations bills. This authority makes NDF appropriations available for obligation until expended, rather than requiring them to be obligated within a particular time period, such as a fiscal year. This authority has allowed NDF to carry over balances across multiple fiscal years that it has not designated for specific projects. NDF considers money to be designated for a specific project and no longer available for use on other projects at the point when it notifies Congress of its intent to fund the project, unless NDF renotifies the funds for the purpose of another project. NDF’s carryover balances have increased over time and are at historically high levels. Figure 6 provides an overview of NDF’s various categories of funding and how NDF accumulates carryover balances. In addition, NDF’s no-year budget authority allows it to close projects and apply the unexpended funds to future projects; however, NDF has sometimes delayed in closing out some projects for many years, including projects where no work ever occurred or was suspended. Until projects are closed, any unexpended project funds are not reported as part of NDF’s carryover balances. As a result, NDF’s carryover balance is likely understated.

NDF’s carryover balances have grown significantly in the past few years to historically high levels. NDF’s carryover balance peaked at the end of fiscal year 2009 at $122 million in unnotified funds, which were carried over into fiscal year 2010 as shown in figure 7. Unnotified funds include funds never designated for a project, as well as any unobligated and unexpended project funds that once again become available as unnotified funds when a project is closed. Before the end of fiscal year 2009, NDF’s balance carried over into the next fiscal year had been $10 million or higher three times since the program began in fiscal year 1994 and had never been higher than $22 million. NDF’s carryover balance was $86 million at the end of fiscal year 2012.

NDF has not established a formal means of determining the amount of money it needs to carry over from year to year to respond to unanticipated nonproliferation and disarmament opportunities.According to the Assistant Secretary of State for International Security and Nonproliferation, State management is aware of the growth in NDF’s carryover balances and is committed to spending them down as opportunities consistent with the mission of NDF arise.

In the past several years, increases in NDF’s annual appropriations from levels in earlier fiscal years, as well as the initiation of work on a smaller number of projects, have contributed to NDF’s increased carryover balances. As shown in figure 7 above, NDF’s appropriation was never more than $30 million before fiscal year 2005 and was only higher than $20 million in one fiscal year. However, from fiscal years 2005 through 2012, NDF’s appropriation has been $30 million or more every year. NDF’s appropriation reached a high of $118 million in fiscal year 2009, which included $77 million in a supplemental appropriation. NDF also initiated a limited number of projects in the past 2 years. For example, it initiated only one new project in fiscal year 2012 and no projects in fiscal year 2011. In total, NDF notified Congress of 24 projects from fiscal years 2007 through 2012, compared with 63 projects from fiscal years 2001 through 2006. In part, NDF officials stated that the decline in the number of projects initiated was caused by the creation of other U.S. government programs that are now able to fund various activities from their own budgets that might have previously required NDF funding. For example, NDF officials noted that NDF previously funded certain types of export control assistance activities that State’s Export Control and Related Border Security Assistance program is now able to fund and implement. However, NDF officials noted that while NDF has initiated a smaller number of projects in fiscal years 2007 through 2012, many of the projects it has initiated have involved significantly larger amounts of notified funds than in fiscal years 2001 through 2006. For example, NDF had only one project with over $10 million in notified funds in fiscal years 2001 through 2006, while it had 11 projects with over $10 million in notified funds in fiscal years 2007 through 2012.

NDF’s budget request for fiscal year 2013 was $30 million.

Other U.S. government programs also receive no-year money and have the ability to carry over balances from year to year. We have previously reported on efforts by some of these programs to determine appropriate carryover balance amounts. For example, in contrast with NDF, DOE’s National Nuclear Security Administration has established thresholds for the carryover balances of its Defense Nuclear Nonproliferation programs. These threshold amounts are based upon specified percentages of the total funds available to each of the Defense Nuclear Nonproliferation programs in a given fiscal year. As we reported in December 2011, if programs’ carryover balances exceed these thresholds, they will trigger additional scrutiny by the National Nuclear Security Administration as to whether the carryover balances are appropriate to meet program requirements.

NDF also maintains a significant amount of funds that it notified to Congress in the past for projects, but has not yet obligated. Of the 32 active NDF projects initiated in fiscal year 2010 or earlier, 25 percent of the total notified funds have not yet been obligated. This represents more than $66 million in notified but unobligated funds. As some NDF projects take many years to complete, NDF does not necessarily obligate all funds early in their implementation. However, of the 32 active NDF projects initiated in fiscal year 2010 or earlier, we identified 5 projects for which less than 25 percent of the notified funds had been obligated.

Because NDF’s funding is no-year money, NDF can close projects for which it has never started work, or has suspended work, and apply the unexpended funds to future projects. However, NDF has not established guidance for determining when it should close out inactive projects. As a result, NDF funds may be tied up for years in projects where no work is occurring, precluding the funds’ use for other projects. For example, NDF maintains over $24 million in unobligated funds from a $25 million project in North Korea that it notified to Congress in fiscal year 2008. NDF has not obligated any of the funds for this project since North Korea expelled International Atomic Energy Agency inspectors and U.S. monitors from the country in April 2009 and work on the project was abruptly halted. Additionally, NDF has not yet obligated any of the $750,000 notified in fiscal year 2005 for a project to support Proliferation Security Initiative interdiction activities. According to NDF officials, no funds have been obligated to date because they have not identified any Proliferation Security Initiative activities that warranted the use of the funds.

NDF has not developed guidance that establishes time frames for closing cancelled or completed projects to ensure that they are closed out in a timely manner. NDF data show that in the past, NDF has taken years to cancel and close some projects where little or no work ended up occurring. Of the 61 projects NDF has closed out since the beginning of fiscal year 2007, 16 were cancelled projects for which less than 20 percent of the notified funds were ever obligated and expended. For six of these cancelled projects, NDF took more than 10 years to close them out from the date they were initially notified to Congress, and for an additional 3 projects NDF took more than 5 years to close them out from the date they were notified to Congress. In total, these 9 projects had over $8.3 million in notified funds that were never expended.

In addition to cancelled projects, NDF has taken years to close some completed projects. For example, of the 61 projects NDF has closed out since the beginning of fiscal year 2007, we identified 13 that NDF closed out more than 10 years after work on the project was completed and an additional 18 that NDF closed out more than 5 years after work on the project was completed. These 31 projects had over $3.5 million in notified but unexpended funds. The unexpended funds for these cancelled and completed projects were eventually made available for use on future projects. However, it can take years from the time projects are cancelled or completed to the time they are closed out, which can result in an understatement of the amount of money NDF has available.

NDF officials noted that prior to 2005, NDF took years to close completed and cancelled projects because it lacked the needed staff. However, NDF officials stated that since then, the office has hired additional staff and developed procedures to help ensure that projects are closed out more quickly. Additionally, NDF officials noted that the office has eliminated its backlog of projects needing to be closed. However, NDF still has 42 projects for which it has completed all financial close-out activities, but is in the process of seeking approval from the Under Secretary for Arms Control and International Security before closing them and returning the unexpended funds to the NDF account. These 42 projects have a total of over $19 million in unexpended funds that will be added to NDF’s unnotified balances, once they are closed.

State Has Not Conducted a Program Evaluation of NDF and Lacks Information for Conducting an Evaluation to Inform NDF’s Program Management

State has not conducted a program evaluation of NDF and lacks information that would be useful in doing so. A program evaluation is a systematic study to assess how well a program is working and that can identify lessons learned for future projects. State has developed a new policy requiring bureaus to evaluate programs, projects, and activities. To comply with this policy, State issued guidance requiring bureaus to submit an evaluation plan for fiscal years 2012 through 2014, identifying the programs and projects they plan to evaluate. However, ISN, which oversees NDF, did not include NDF in its fiscal years 2012 through 2014 evaluation plan. Moreover, State currently lacks information, such as the results of some projects and lessons learned, that could be used to conduct a program evaluation of NDF and that would help inform the management of the program.

State Has Not Conducted an NDF Program Evaluation

Since NDF became operational in 1994, State has not conducted a program evaluation of NDF, according to ISN and NDF officials.

Although NDF reported to Congress in its fiscal year 2013 budget submission that all of its projects are evaluated in-house, these documents are project close-out monitoring reports and not evaluations. As State and other organizations have noted, monitoring and evaluations are conceptually and operationally different. GAO defines evaluation as individual, systematic studies that are conducted periodically or on an as- required basis to assess how well a program is working, while project close-out reports consist of formal documentation that indicates completion of the project or phase of the project.

ISN and NDF officials explained that NDF and its projects have never been subject to a program evaluation because of the unique nature of each project. For example, according to NDF officials, to get one country to agree to dismantle its Scud missiles, NDF agreed to pay for that country’s armed forces to use a labor-intensive method to dismantle the missiles. However, NDF officials also noted that there are common features to many projects that can serve as the basis for lessons learned. Our analysis of NDF’s project database shows that since NDF’s first project in 1994, NDF has implemented a number of similar projects that could have been evaluated to determine the lessons learned for use in present and future projects. For example, NDF has implemented 11 destruction and conversion projects involving missiles and rockets, 5 of which involved the destruction of Scud missiles. The first of these missile destruction and conversion projects took place in 1994 and the latest began in 2010. In addition, as noted earlier in this report, NDF has implemented at least five projects involving the shutdown of a nuclear reactor in Kazakhstan.

State Has Developed a New Evaluation Policy Along with Tools to Facilitate Evaluations and Lessons Learned

The Government Performance and Results Modernization Act of 2010 strengthened the mandate to evaluate programs, requiring agencies to include a discussion of evaluations in their strategic plans and performance reports. In part to comply with the requirements of this Act, State established a policy in February 2012 to evaluate programs and projects. In addition, as we reported in May 2012, according to officials from State’s Bureau of International Narcotics and Law Enforcement, the policy was established to comply with a June 2009 directive from the Secretary of State for systematic evaluation and to promote a culture change among program offices.February 2012 policy superseded an evaluation policy dating from September 2010 that did not fully comply with a recommendation later According to State officials, the detailed in State’s December 2010 Quadrennial Diplomacy and Development Review that State adopt an evaluation framework consistent with that of the U.S. Agency for International Development.

State’s 2012 evaluation policy outlines requirements and provides a framework and justification for evaluations of all State programs, including both diplomatic and development programs, projects, and activities. For example, the policy notes that a robust, coordinated, and targeted evaluation policy is essential to State’s ability to measure and monitor program performance, document program impact, and identify best practices and lessons learned. It also states that such a policy can help assess return on investment and provide input for policy, planning, and budget decisions.

State’s evaluation policy assigns a key role to the bureaus and requires them to evaluate two to four programs, projects, or activities over a 24- month period starting in fiscal year 2012 and all large programs, projects, and activities at least once in their lifetime or every 5 years, whichever is less. It also requires the bureaus to appoint a coordinator to ensure that the bureaus meet the new policy’s requirements; requires bureaus to develop and submit a bureau evaluation plan as an annex to their multiyear strategic plans, but gives bureaus flexibility in determining the specific programs to evaluate, as well as the timing and manner of evaluations they will perform; and notes that bureaus should integrate evaluation findings into decision making about strategies, program priorities, and project design, as well as into the planning and budget formulation process.

State’s evaluation policy also draws a clear distinction between monitoring and evaluation. State defines monitoring as a continual process designed to assess the progress of a program, project, or activity. By comparison, evaluations go beyond monitoring to identify the underlying factors and forces that affect the implementation process, as well as the efficiency, sustainability, and effectiveness of the intervention and its outcomes. As our previous work, State, and other organizations have noted, evaluations also require a measure of independence. According to State, this can be promoted in several ways, including entrusting the evaluation to an outside research and evaluation organization or fostering a professional culture that emphasizes the need for rigorous and independent evaluations.

To complement the new evaluation policy and provide further direction, State issued new guidance in March 2012 that describes several types of evaluations that bureaus can conduct and outlines data collection methods. The March 2012 guidance also defines the information that must be included in each bureau evaluation plan. For example, bureaus must include in the first plan a list of evaluations to be initiated or completed between fiscal years 2012 and 2014. Bureaus are expected to update these plans annually, according to State officials.

In addition to the guidance, State has developed or is in the process of developing other resources and tools to complement and support the new evaluation policy. These include an internal website containing resources to assist bureaus with their evaluation responsibilities and the establishment of a community of practice where officials can share their expertise and discuss evaluation issues.

ISN Did Not Include NDF in Its Fiscal Years 2012 through 2014 Bureau Evaluation Plan

ISN submitted its first bureau evaluation plan in April 2012, but the plan did not include any NDF projects. According to ISN officials, the bureau had a short amount of time in which to submit its bureau evaluation plan and for that reason the plan focused on programs that already had projects scheduled for evaluation. After the State evaluation guidance was finalized in late March 2012, the bureaus only had 1 month to submit their bureau evaluation plans for fiscal years 2012 through 2014, according to ISN officials. In canvassing ISN’s five program offices, ISN determined that some offices were already planning evaluations for certain projects within their programs, according to ISN officials and documents. For example, according to the ISN bureau evaluation plan, State’s Global Threat Reduction (GTR) Program plans to contract for four evaluations during the fiscal years 2012 through 2015 period. GTR has in the past contracted for evaluations of its projects in Iraq, Ukraine, and Russia.

State Currently Lacks Information Useful in Conducting an NDF Program Evaluation and Managing NDF’s Program

State currently lacks information that would be useful in conducting a program evaluation of NDF and in improving the management of its program. NDF uses project close-out reports to document its final monitoring of a project.the importance of preparing good monitoring reports since these both complement evaluations and can provide valuable information for use in preparing evaluations. They can also be a key source of information that can be used to improve the management of a program, such as the State’s March 2012 evaluation guidance notes results of a project and lessons learned. However, NDF’s project close- out reports did not document information that could be useful to NDF and the NDF Review Panel. The reports also varied in content and format.

Project management standards note the importance of documenting results in project close-out documents, but not all of the project close-out reports that we examined discussed the results of the project. Of the 23 project close-out reports that we examined, 2 did not address project results at all. In addition, for the other 21, we found some instances where the discussions of results were fairly minimal and other instances where the reports did not state whether all intended outcomes or goals had been achieved. According to the Project Management Body of Knowledge Guide, a recognized standard for project managers, project close-out documents or reports should include formal documentation that indicates completion of a project, including results. Moreover, according to NDF officials, NDF and the NDF Review Panel consider potential results in determining whether to fund future projects.

Project management standards note the importance of documenting project results and entering this information into a database of lessons learned. However, 13 of the 23 project close-out reports that we examined did not discuss lessons learned. Moreover, NDF officials stated that they did not have a database of lessons learned. To document and share lessons learned, NDF officials said that they primarily used informal mechanisms such as e-mails or face-to-face meetings. The Project Management Body of Knowledge Guide notes the importance of documenting lessons learned and entering this information into a lessons- learned database for use in future projects. Some agencies that implement projects or with an interest in communicating lessons learned have formal databases that they use to enter lessons learned and communicate this information to project implementers. For example, the U.S. Agency for International Development and the U.S. Army Center for Lessons Learned have both established lessons-learned databases.State Bureau of Budgeting and Planning officials told us that as part of its effort to implement the new evaluation policy, State is considering the establishment of a lessons-learned database that could include information from NDF.

In addition, the close-out reports often did not address other criteria that the NDF Review Panel considers in assessing future projects for NDF funding. For example, 11 of the 23 project close-out reports that we examined did not discuss cost, and 17 of the 23 did not discuss the timeliness of the project. In one instance, the final cost of the project was approximately 66 percent under the amount notified to Congress, but the close-out report did not provide a reason why this had occurred. Of the 23 reports we examined, 19 did not discuss the appropriateness of using NDF funding for the project and none discussed the project’s return on According to the guidelines promulgated by State when investment. NDF was established in 1994, NDF criteria used to assess a project’s suitability for NDF funding include the cost and the appropriateness of using NDF as a source of funding. In addition, according to NDF officials, the NDF Review Panel also considers the project’s return on investment and timeliness as part of its criteria. Moreover, according to NDF officials, the NDF Review Panel has sometimes modified its initial assessment of a project’s cost based on past experience.

Return on investment is a measure of the benefits gained by implementing a project. use of a standard format in project close-out reports might not always be appropriate or useful given the wide variety of projects that NDF funds and undertakes. However, it may be difficult to obtain information useful to future evaluations from reports that vary so significantly in content and format.

Recognizing the need for NDF project managers to prepare a close-out report to ensure that information is consistently documented, in December 2010, NDF established the expectation that NDF project managers produce a project close-out report. NDF also produced a project management guide designed to encourage project managers to standardize their procedures. The NDF project management guide, which according to NDF officials is based on the Project Management Body of Knowledge Guide, among other things lists the preparation of a project close-out report as one of the steps for closing out a project. However, NDF officials stated in July 2012 that while project managers are expected to write project close-out reports, they are not required to do so. In addition, NDF officials stated that NDF encourages but does not require the use of the project management guide and the guide does not detail the information that project managers need to include in their reports or specify the report format. Partly in response to our work, NDF officials stated that they plan to develop standard operating procedures to address the issues we identified in the project close-out reports, which will also include a requirement for project managers to identify lessons learned. However, as of November 2012, they had not made any changes to their procedures.

Conclusion

Over its lifetime, NDF has responded to pressing nonproliferation and disarmament needs, helping to address significant threats to international security. To support NDF in accomplishing its mission, U.S. law has provided NDF with an unusual degree of flexibility in how it manages its resources and conducts its work. While the critical nature of NDF’s mission provides a strong rationale for such flexibility, it also increases the need for State to effectively manage its program resources to ensure that NDF is achieving its intended results. However, State has not taken the necessary steps to do so. For example, unlike some programs, NDF lacks a formal process for determining how much carryover balance it needs to maintain in reserve to meet unanticipated program requirements. Without such a process, NDF cannot know to what extent its carryover balances, which have increased in the past few years to historically high levels, may be exceeding its unanticipated funding needs. In addition, NDF has taken years to close some projects, delaying the availability of unexpended funds for other projects and likely understating NDF’s carryover balances. A methodical process for determining NDF’s needed carryover balances and for closing projects could help ensure that NDF’s budget requests accurately reflect program needs. Additionally, NDF lacks a process to identify and incorporate lessons learned into future projects. State has never performed a program evaluation of NDF in its 18-year history to determine lessons learned for better designing projects that contribute to U.S. nonproliferation goals. State has implemented a new evaluation policy that could encourage the bureaus to more rigorously rationalize and prioritize their resources over time and identify and incorporate lessons learned. Nonetheless, State is not including NDF among the programs to be evaluated during fiscal years 2012 through 2014. Finally, NDF’s project close-out reports could provide useful information to inform future program evaluations’ identification of lessons learned that could be systematically incorporated into future projects.

Recommendations for Executive Action

To more effectively manage NDF’s resources, increase program accountability, and ensure that NDF has the information necessary to improve program performance, we recommend that the Secretary of State take the following four actions: direct NDF to develop a methodology for determining the amount of reserves that it should carry over annually to meet program requirements to address unanticipated nonproliferation and disarmament opportunities; direct NDF to develop guidance for determining when inactive NDF projects should be closed and the remaining, unexpended funds made available for use on other projects; direct ISN and NDF to periodically and systematically conduct and document program evaluations of NDF; direct NDF to revise its project management guide to establish requirements for project managers’ close-out reports to include information useful for improving the management of NDF projects.

Agency Comments and Our Evaluation

We provided a draft of our report to DOD, DOE, OMB, and State for their review and comment. DOD and OMB did not provide comments. State provided written comments, which we have reprinted in appendix II. State concurred with all four of our recommendations and identified several actions it intends to take in response to the recommendations. For example, State said that it will direct NDF to develop a methodology that the NDF Review Panel can then use to make an annual recommendation on the appropriate level of carryover balances for the next fiscal year to the Under Secretary for Arms Control and International Security. State also said that NDF has begun implementing the recommendation to revise its project management guidance to establish requirements for close-out reports, by creating a standard operating procedure for these reports. State and DOE provided technical comments, which we incorporated in the report, as appropriate.

We are sending copies of this report to interested congressional committees, the secretaries and agency heads of the departments addressed in this report, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-9601 or melitot@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Scope and Methodology

This report examines (1) the Department of State’s (State) use of Nonproliferation and Disarmament Fund (NDF) authorities in developing and implementing NDF projects and (2) the extent to which State has conducted a program evaluation of NDF and used this information to improve program performance.

To assess how State has used NDF’s authorities in developing and implementing NDF projects, we obtained program-wide and project-level data from NDF’s Financial and Information Management System (FIMS) for fiscal years 1994 through 2012. To assess the reliability of data in FIMS, we reviewed NDF documentation on the system, reviewed previous audits that assessed the reliability of FIMS data, compared FIMS data to data from other sources to confirm FIMS data’s accuracy, and interviewed cognizant State officials. To gain additional information on the reliability of data in FIMS, we met with a private contractor conducting a review for NDF under the supervision of the State’s Office of the Inspector General. The scope of the contractor’s work included a review of the reliability of FIMS data. On the basis of the information we obtained, we determined that the FIMS data were sufficiently reliable for our purposes. We analyzed NDF program-wide data to determine program appropriations, commitments, obligations, and carryover balances for fiscal years 1994 through 2012. We analyzed NDF project data to determine project funding amounts, locations, objectives, and time frames for fiscal years 1994 and 2012. Additionally, we reviewed NDF project documentation including project proposals, approval memos, and congressional notifications, for all NDF projects initiated since the beginning of fiscal year 2007 to assess the types of projects NDF has funded and how it used its authorities in developing and implementing these projects. To gain additional information on NDF projects, we also reviewed State press releases, speeches by State officials, and fact sheets describing NDF activities. To identify NDF’s key legal authorities, we reviewed relevant laws and regulations, including the FREEDOM Support Act and NDF appropriations legislation for fiscal years 1994 through 2012. Additionally, we examined congressional committee and conference reports from 1999 through 2012 to identify relevant congressional guidance regarding NDF. We also reviewed key NDF documents discussing the program’s authorities, including the 1994 memorandum pursuant to the FREEDOM Support Act establishing the program and the accompanying NDF Guidelines. To gather additional information on NDF’s authorities and how it develops and implements projects, we conducted a series of interviews with NDF officials and also met with officials from other agencies that proposed or implemented NDF projects, including the Departments of Defense and Energy. We also interviewed officials from the Office of Management and Budget to gain additional information on NDF’s budget planning process. Finally, we reviewed previous GAO reports, as well as reports by the State Inspector General, the Congressional Research Service, and the National Academies of Science, to identify relevant findings regarding NDF and related U.S. nonproliferation and disarmament programs.

To assess the extent to which State has evaluated NDF and used this information to improve program performance, we interviewed State officials with the Bureaus of International Security and Nonproliferation (ISN) and Budgeting and Planning. We also obtained copies of State’s February 2012 evaluation policy and March 2012 evaluation guidance, as well as a copy of ISN’s April 2012 bureau evaluation plan. NDF officials described their project close-out reports as evaluations, but based on our discussion with State ISN and Budgeting and Planning officials, our review of GAO reports discussing evaluations, and State’s February 2012 evaluation policy, we determined that NDF’s project close-out reports fit more closely the standard of a monitoring report. GAO defines evaluations as individual, systematic studies that are conducted periodically or on an as-required basis to assess how well a program is working. State’s evaluation policy notes that in addition to assessing the progress of a program, project, or activity, evaluations go beyond monitoring to identify the underlying factors and forces that affect the implementation process, as well as the efficiency, sustainability, and effectiveness of the program or project and its outcomes. As such, State’s policy draws a clear distinction between evaluation and monitoring. As previous GAO reports, State, and other organizations have noted, evaluations require a measure of independence, which can be promoted in several ways, including entrusting the evaluation to an outside research and evaluation organization or fostering a professional culture that emphasizes the need for rigorous and independent evaluations. By comparison, State defines monitoring as a continual process designed to assess the progress of a program, project, or activity. The Project Management Body of Knowledge Guide notes that project close-out documentation consists of formal documentation indicating the completion of a project or phase of a project. For all these reasons, on the basis of our analysis of NDF’s project close-out reports, we made the determination that NDF’s project close-out reports better fit the standard of a monitoring report than an evaluation. While project close-out reports serve a different purpose from evaluations, based on our review of the Project Management Body of Knowledge Guide and NDF’s Project Management Guide, we determined that we could assess the project close-out reports to determine their usefulness in enabling NDF to improve its management of the program. For this purpose, we obtained a judgmental sample of 23 project close-out reports—14 of which we selected and 9 of which State selected. In selecting our sample, we chose only to consider projects that NDF had closed out since the beginning of fiscal year 2007—of which there were 61—in order to ensure that all close-out documentation was completed for the projects. Our selection criteria for our sample included project cost, location, and type. For example, we selected a variety of projects from all four categories of projects that NDF funds—destruction and conversion, safeguards and verification, enforcement and interdiction, and education and training. State selected its projects using similar criteria; however, State did not limit itself to projects that were closed out. In some cases, State selected projects where work was completed, but the project was not yet officially closed out. In reviewing the documentation for the projects State selected, we determined that these projects were broadly similar to the ones that we selected and the inclusion of these projects in our analysis did not alter our overall findings or compromise the independence of our work. To conduct our analysis of the close-out reports, we developed a list of key terms, such as “results,” “completion,” and “lessons learned.” Our inclusion of these terms was based on our analysis of project management standards, which note the importance of the project close- out process in the project management cycle and the importance of obtaining information about the results of the project and lessons learned. We also included other terms such as “cost,” “timeliness,” “on time,” “return on investment,” and “appropriateness of using NDF funding.” We included these terms because NDF officials told us that NDF and the NDF Review Panel include these criteria in determining a project’s suitability for NDF funding. Because NDF does not have any requirement to use a standard terminology in its reports, we used a dictionary to obtain other synonyms of these terms as well. We examined each of the project close- out reports to determine the presence of these key terms. We also examined each of the project close-out reports to determine the author, content, and format. We did this on the basis of discussions with NDF officials, who told us that they had established an expectation that NDF project managers complete a project close-out report and had developed a project manager’s guide that contained a checklist. While NDF does not have a requirement for project reports to be written in a standard format, we determined that the close-out reports that we had examined varied widely in their content and format and concluded that such variety could make it more difficult for evaluators to extract key information from these reports. After completing our initial review, the lead analyst submitted the results of his work and the methodology used to two additional levels of review. These reviewers were asked to validate the methodology and results. The sample of 23 project close-out reports cannot be generalized to the entire population of NDF project reports for the period in our review. We also reviewed NDF’s Project Management Guide to determine the extent to which NDF has established specific requirements or guidance regarding how project close-out reporting should be conducted. To obtain the list of 11 similar missile destruction and conversion related projects, we conducted a word search of NDF’s projects using the key terms “missiles” and “rockets.”

We conducted this performance audit from March 2012 through November 2012 in accordance with generally accepted government auditing standards. These standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of State

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, the following staff made key contributions to this report: Jeff Phillips, Assistant Director; Lynn Cothern; Martin De Alteriis; Mark Dowling; José M. Peña, III; and Ryan Vaughan. Etana Finkler and Jeremy Sebest provided graphics support and Debbie Chung provided editorial assistance. Julie Hirshen and Julia Jebo Grant also provided additional technical assistance.