Foreign Assistance: USAID's Reengineering at Overseas Missions
Published: 1997-09-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed the effect of the U.S. Agency for International Development's (USAID) reforms on its overseas missions' operations and delivery of assistance, focusing on how: (1) USAID missions have reengineered their operations; (2) reengineering has affected the content of USAID's assistance program; (3) USAID monitors and evaluates the results of its projects; (4) USAID allocates funds for its projects; and (5) USAID's New Management System supports mission operations.
section_title: What GAO Found
paragraphs: GAO noted that: (1) in reengineering the way assistance is delivered, USAID missions have only recently made the investment and operational changes necessary to bring about long-term change; (2) to date, USAID's reengineering efforts to improve its delivery of assistance have shown some benefits in the areas of planning, implementing, and monitoring projects; (3) notwithstanding this progress, USAID still has major obstacles to overcome in deploying its New Management System and establishing valid and reliable performance measures; (4) overseas missions have changed the way they manage assistance in support of USAID's strategic goals; (5) however, because missions had no baseline data on operations management prior to reengineering their processes, neither they nor GAO can identify measurable increases in efficiency or effectiveness of the delivery of assistance; (6) USAID missions have, to varying degrees, begun to establish more results-oriented indicators and report the results of their projects annually; (7) however, the missions still have difficulty linking their activities to the broad indicators of development--such as a country's rate of economic growth; (8) one way to provide a more complete picture of program performance could be to supplement performance measurement data with impact evaluation studies; (9) although the relative performance of mission programs is clearly a factor in USAID's resource allocation decisions, these decisions are still largely driven by other considerations, such as contributions to foreign policy and agency priorities, country need and commitment, and funding priorities of the executive branch and Congress; (10) USAID's New Management System, one of the agency's key tools in reforming its operations, is not working as intended; (11) this computer system, which is expected to cost at least $100 million by the end of fiscal year 1998, was designed to enable the agency to manage its resources and monitor results more effectively by consolidating accounting, budgeting, personnel, procurement, and program operations into a single, integrated network that can be accessed worldwide; (12) despite warnings that the system had not been tested and did not meet basic federal requirements, USAID activated the system in July 1996 in Washington and deployed it in October 1996 in its missions; (13) USAID suspended much of the system's operation in April 1997, after it failed to work properly; and (14) correcting system deficiencies will be critical to continued progress of the agency's reform effort.

section_title: Letter
paragraphs: In response to your request, we have reviewed the effect of the U.S. Agency for International Development’s (USAID) reforms on its overseas missions’ operations and delivery of assistance. Specifically, we focused on (1) how USAID missions have reengineered their operations, (2) how reengineering has affected the content of USAID’s assistance program, (3) how USAID monitors and evaluates the results of its projects, (4) how USAID allocates funds for its projects, and (5) how USAID’s New Management System supports mission operations. In doing this review, we visited missions in El Salvador, Honduras, Bangladesh, and the Philippines.
This is the second of a two-part review of USAID’s reform efforts. In September 1996, we reported that USAID had adopted a development strategy with priority areas of concentration, reorganized its headquarters units, closed some overseas missions, cut its total workforce, developed an information resource management system for its business and service functions, and had begun to reengineer its procurement and overseas mission operations.
subsections: section_title: Background
paragraphs: USAID has provided foreign aid since 1961 in an effort to improve countries’ economies, health, environment, and democratic processes. In times of crisis, the agency has also provided humanitarian assistance to those in need. USAID currently has 72 overseas missions and offices that manage projects associated with this foreign assistance. These projects are generally implemented by private voluntary organizations, nongovernmental organizations, international agencies, universities, and contractors.
Since the end of the Cold War, Congress has reduced USAID’s appropriations, citing other funding priorities as well as basic questions about the effectiveness of foreign aid. USAID’s fiscal year 1996 obligations of $5.7 billion were 13 percent less than those in the previous year (see app. I for information on USAID obligations for fiscal years 1995 through 1997). To accommodate budget reductions, USAID cut its total workforce from 11,150 in September 1993 to 7,609 in June 1997 and will have closed 24 missions by the end of fiscal year 1997. Under the executive branch’s decision to consolidate the Department of State, the Arms Control and Disarmament Agency, and the U.S. Information Agency, the USAID Administrator will report directly to the Secretary of State rather than, as previously, to the President.
Congress specifically directs how USAID spends much of its funding, earmarking substantial amounts of USAID’s direct appropriation for development assistance for specific purposes, such as child survival. Congress also earmarks funds for economic support and food assistance, which are administered by USAID. Congressional earmarks and executive branch directives (e.g., for the population program) accounted for 59.8 percent of development assistance funds in fiscal year 1995, 66.5 percent in fiscal year 1996, and 69.5 percent for fiscal year 1997 (see app. II for more details on directed and undirected assistance).
In 1993, in response to criticisms about USAID’s inefficient processes and inability to demonstrate a significant impact on developing countries and some calls for the agency’s abolition, USAID’s Administrator volunteered the entire agency as a “reinvention laboratory.” USAID established five strategic goals to meet its agency mission of pursuing sustainable development in developing countries: (1) achieving broad-based economic growth, (2) building democracy, (3) stabilizing world population and protecting human health, (4) protecting the environment, and (5) providing humanitarian assistance.
According to USAID, reengineering the agency’s overseas operations has revolved around (1) increasing its customer focus, (2) managing for results, (3) enhancing staff participation and teamwork, (4) empowering and increasing the accountability of staff, and (5) valuing diversity. The reengineering included reorganizing mission organizational structures, eliminating unnecessary administrative requirements, and including the participation of development partners and recipients in program planning.
In 1994, USAID selected 10 missions to test these reengineering principles, concepts, and approaches prior to deployment worldwide. Based on observations of the experiments and feedback from the participating missions, USAID initiated full implementation of its reengineering approach on October 1, 1995. In fiscal year 1996, USAID deployed the New Management System, which was intended to provide an integrated database of financial and program information to facilitate the management of resources and monitoring of results. (See app. III for a chronology of USAID reforms.)
USAID’s Administrator has noted that the agency’s reforms are consistent with the Government Performance and Results Act of 1993 (GPRA) and have positioned USAID well to meet the act’s requirements for strategic planning and performance monitoring.
section_title: Missions Are Reengineering Their Operations in Line With USAID Reforms
paragraphs: Overseas missions have changed their processes to streamline the way they provide assistance in support of USAID’s strategic goals. These changes include taking a team approach to management, accepting increased accountability through the authority and flexibility given them by USAID, and increasing the participation of development partners in the design and implementation of projects. USAID has supported the missions’ reform efforts with information and advice. Mission officials believe that these changes will result in more effective assistance programs. Moreover, our review indicated that the changes show promise that USAID is making progress in better managing the process of providing assistance.
We could not fully assess the impact of the missions’ operational changes for the following reasons: (1) missions are at varying stages of implementing changes; (2) the impact of these changes may not be seen for several years, especially those that affect the design of new projects; and (3) the missions we visited had not established baseline information on the efficiency or effectiveness of their management operations before reengineering their processes and therefore could not demonstrate many measurable improvements. In some missions, officials indicated that some baseline data could be compiled from historical records but that doing so would demand an inordinate amount of scarce mission resources.
subsections: section_title: Missions Have Established Team Structures
paragraphs: Overseas missions we visited have reengineered their organizational structure to focus on achieving strategic objectives. Most missions have replaced their former office structure with a team approach to management, and each team is responsible for one of the mission’s strategic objectives. Team members at the missions we visited commented that the team approach has improved the management of their activities. USAID said that overseas missions have observed that teams are now more likely to (1) remain focused, (2) rethink tactics quickly when activities do not go as planned, and (3) terminate activities that only marginally contribute to results.
Prior to reengineering, missions had several offices: technical offices, such as the education, health, and agriculture offices, which had day-to-day responsibility for implementing assistance projects, and support offices, such as the contracts, controller, and legal adviser’s offices, which assisted the technical offices. According to mission staff, in the past, each of these offices had its own management hierarchy and separately reviewed proposed actions such as expenditure requests or new project starts. This review was often done sequentially, and as each office raised problems or issues, action was delayed until they could be resolved; the proposal was then passed to the next office for review.
The four missions that we visited had reorganized their offices into teams to manage assistance activities that support each mission’s strategic objectives. Staff members that previously worked for different technical and support offices now work together on a team toward a specific common objective. In most missions, the teams replaced much of the traditional office structure. For example, the mission in El Salvador combined the staff of its economic, education, and productive resources offices into a single economic growth strategic objective team, which also includes members from the contracts and controller offices.
At the missions we visited, both technical and support staff said that issues that had previously delayed progress under the old process can now be resolved within the teams. For example, contract office staff are now involved in the early stages of project design and implementation, suggesting appropriate procurement instruments and helping draft scopes of work for proposed contracts. According to contract officers, this approach has reduced the number of sequential reviews and rewrites of proposals and the tensions that have historically existed between staff and technical offices.
section_title: Mission Management and Staff Have More Authority and Flexibility
paragraphs: Under reengineering, USAID increased the authority and flexibility of missions in managing their activities by eliminating many administrative requirements previously imposed on the missions. For example, USAID eliminated the requirement for missions to prepare elaborate project papers and obtain bureau approval for new projects. Once a regional bureau approves a mission’s strategic objectives, the mission is authorized to develop activities to meet those objectives without the bureau’s further review and approval. USAID also eliminated the 10-year time limit on all projects, allowing the missions the flexibility to extend projects when it is advantageous to do so. For example, in Honduras, the mission extended an ongoing policy project in order to continue to provide high-level policy advice to the newly elected Honduran administration. Before reengineering, the mission would have had to design and authorize a completely new project or apply for a waiver from USAID to extend the project.
Furthermore, USAID delegated to the missions a host of executive authorities previously held by officials at USAID headquarters. For example, missions may now independently issue implementation letters and negotiate and implement agreements with other U.S. government agencies.
In accordance with the agency’s emphasis on staff empowerment and the assignment of responsibility and accountability under reengineering, mission directors delegated new authorities to their missions’ strategic objective teams. For example, the mission director in Honduras authorized team leaders to approve individual expenditures of up to $100,000 in program funds without her review.
section_title: Missions Are Implementing a System to Report Results of Activities
paragraphs: In the past, USAID was more concerned with the required inputs and expected outputs of a particular project than the results to be achieved. Projects had individual purposes that often related to an overall goal or goals in only a very general way, and the goals themselves were in broad categories rather than specific objectives against which progress could be measured. Also, missions reported on their activities in a variety of reports, including semiannual reviews and project implementation reviews, that did not necessarily focus on development results but rather focused on the status of project implementation.
USAID is increasingly holding missions accountable for results. In fiscal year 1996, it began requiring the missions to report annually on the results of their programs in a report called the Results Review and Resource Request (R4). Missions must now report to high-level management on their progress in meeting agreed-upon strategic objectives and agency goals and quantify that progress through performance indicators developed by the missions, often with assistance from USAID’s Center for Development Information and Evaluation. Missions are to supplement the data in the annual R4 document with analyses of performance data and other evidence that the mission is making progress in meeting its strategic objectives.
section_title: Participation of Development Partners Has Increased
paragraphs: USAID has given its development partners a more active role in developing country strategies and designing and managing projects. Many of these partners were enthusiastic about this change. In some cases, the increased interaction has accelerated the design and management process. Also, with input from customers, the missions believe they have been better able to respond to a country’s needs and develop a strategy for meeting those needs.
Missions previously presented completed project designs to host countries for acceptance. Today, however, the missions invite the host country, other development partners, and customers to participate in planning and designing activities. For example, according to several representatives from nongovernmental organizations in El Salvador, the mission has invited them to its planning and designing meetings, and their input was included in the mission’s development strategy. In Bangladesh, the mission brought together the customers and development partners to redesign a health and population program that for the first time combines services in both family planning and maternal child health care.
Missions also include development partners as members of strategic objective teams. For example, Bangladesh government officials from the Rural Electrification Board told us that they actively participated with USAID team members, contractor staff, and representatives from local rural electric cooperatives in designing a rural power for the poor project and selecting the indicators by which results will be measured.
Many nongovernment organizations and donors welcomed the missions’ new collaborative approach to designing and implementing activities and were very positive about the missions’ interaction with them. The benefit of this increased interaction can be seen in Bangladesh, for example, where the democracy strategic objective team, working with nongovernmental organizations and staff offices, designed a new project in 6 months compared to the 2 years that project designs had generally required before.
Some donors have also observed increased cooperation with USAID missions. In Bangladesh, an official of the British Office of Development Assistance said that the mission was instrumental in influencing the host government’s development of a national health and population strategy. Increased cooperation with other donors was not always evident at the missions we visited, however. For example, in El Salvador, mission officials told us they have tried to coordinate with the Inter-American Development Bank but have received little response. Officials also said that some other donors had been reluctant to share information.
Teams are increasingly relying on surveys of customers to determine their needs and interests when designing activities and developing strategies. For example, in Bangladesh, the mission has focused its democracy and governance activities on local government rather than the national government because customers indicated that local government was a more effective agent of change. In El Salvador, strategic objective teams, with the assistance of local contractors, held focus groups throughout the country and used the results to develop customer service plans and as input for the mission’s country strategy.
section_title: USAID Headquarters Has Supported Reengineering at Missions
paragraphs: USAID has established new systems and practices to support missions’ implementation of its reengineering efforts by providing advice and information on reengineering. However, it has not provided training needed to learn new skills in team dynamics and personnel matters.
USAID provides information, advice, and answers to staff questions on reengineering through a monthly newsletter, best practices reports and other publications, and an electronic help desk. The missions can more easily access up-to-date agency guidance through an on-line computer system that has replaced 33 manuals on policies, procedures, and program operations guidance. Also, USAID’s Management Bureau sent a team to visit several missions in 1997 to identify “significant management, organizational, and personnel-related constraints being encountered by USAID missions as they embrace reengineering principles” and to provide assistance to resolve them.
At the time of our visits to the four missions, USAID was not providing needed training in new job skills and team operations. For example, USAID has promoted the use of performance-based contracting without providing the requisite training to the missions. As a result, the mission contracts officer in Honduras was developing his own training course. Also, USAID support in personnel matters has lagged behind mission needs, according to mission managers. Thus, the missions have, on their own, developed foreign employees’ evaluations and position descriptions and classifications but are still waiting for USAID’s reengineered position descriptions for American staff.
USAID acknowledged that it had not provided adequate guidance to the field in training and personnel matters. However, USAID noted that there is broad agreement that training requirements and Foreign Service position classification should be the next issues that the agency examines.
section_title: Portfolios Are Beginning to Change but Are Subject to Constraints
paragraphs: Missions have begun to change their project portfolios in response to agency reengineering. However, three factors have constrained the restructuring of portfolios. First, reduced mission budgets have limited the availability of funds for new projects. Second, the missions do not have the authority, without headquarters approval, to retain deobligated funds and shift them from one project to another under a restructured portfolio, according to USAID officials. Third, during the first year of reengineering, the missions concentrated more on reorganization activities and the development of strategic frameworks, including defining performance indicators and obtaining baseline data, which limited the time available for portfolio restructuring and new project development.
The missions we visited were still restructuring their portfolios to focus on projects that more directly support their strategic objectives. They were doing this largely by designing new projects specifically to help achieve each strategic objective and by shifting funds to activities that are more effective at achieving the objective. To date, only a few new projects have been started. In the Philippines, the mission is reshaping its portfolio by selecting one project for each strategic objective to serve as a funding mechanism for all activities under the objective. In Bangladesh, the mission initiated one new project for each of the mission’s strategic objectives, which were designed using reengineering principles. In 1996, USAID approved a new country strategy that refocused activities in El Salvador on the rural poor rather than war-to-peace transition activities and macroeconomic reform at the national level, which were the past focus. The mission in Honduras largely restructured its portfolio in 1994 and 1995, prior to the start of reengineering, and was awaiting approval of its new country strategy before it initiated any new projects.
Reengineering efforts were undertaken during a period of severe budget reductions. Between 1994 and 1997, expenditures at the four missions we visited declined between 43 and 69 percent. A change in the number of projects can be used as a rough approximation of the change in mission activities and the restructuring of the portfolio. In general, the number of active projects declined at these missions, mostly between 1994 and 1995, before the formal start of reengineering agencywide. In three missions, a limited number of projects had been initiated since the start of reengineering.
Because most current mission projects were designed and funded before the reengineering began and because USAID’s budget has been reduced, a limited amount of funding has been available for new projects. According to mission officials, the inability to automatically retain funds that are deobligated from a project limits a mission’s flexibility to use funds most effectively. It may also serve as a disincentive for terminating projects. Consequently, missions may allow projects to continue, even though they could identify other projects that may contribute more to their strategic objectives.
During the first year of reengineering, the missions were busy reorganizing into strategic objective teams, learning how to function as teams, selecting measurable indicators, educating their partners about their new approach, and training staff in the use of the New Management System. When we visited the four missions in February 1997, the mission in El Salvador had just completed the development of a country strategy and was still refining its indicators for the economic growth strategic objective. The mission in Honduras was developing a new country strategy and had not implemented any new projects since the start of reengineering.
section_title: Performance Indicators Are Improving, but USAID’s Impact on Overall Development Is Still Unclear
paragraphs: Under reengineering, missions have made varying degrees of progress in developing performance indicators through which they can measure the short-term results of individual projects or groups of projects, but these indicators have not been in use long enough to have made a significant impact on program management. Missions have had difficulty in developing indicators that tally up the results of individual or groups of projects to demonstrate achievement of strategic objectives and overall, long-term development goals, such as stimulating broad-based economic growth. USAID officials acknowledge that in only a few cases have their programs been directly linked to changes in country-level results. USAID management indicated that the ability of a program to affect country-level indicators will depend on the size of the country, the budget available to support specific strategic objectives, and other factors such as the context in which programs are implemented. USAID is now developing common indicators through which it hopes to combine mission results into overall agency accomplishments worldwide. We recently reported that program evaluations can be valuable management tools for demonstrating the programs’ effectiveness. Such evaluations are an option for the agency to consider to help in its efforts to meet GPRA requirements.
subsections: section_title: Missions at Various Stages in Adopting Measures of Intermediate Project Results
paragraphs: USAID missions have been developing performance indicators on two levels. First, missions are developing project-level indicators intended to monitor the performance and intermediate results of individual projects or groups of projects in their portfolio. Second, missions are developing strategic objective indicators that are intended to gauge progress in achieving the missions’ long-term or strategic objectives, such as increasing national rural household incomes or improving national systems of trade and investment.
The four missions we visited were at various stages in developing and using performance measures for the projects intended to achieve one of their strategic objectives—economic growth. In most cases, the project-level indicators were too new for the missions to show impact on program management. Although one mission had demonstrated a thorough integration of its measures into its program management processes, another was still defining the indicators it would use. Some of the missions had fundamentally revised their project-level performance measures in 1996 to make them more closely correspond to their strategic objectives at the country level.
In Honduras, the mission has been using indicators to measure project results since before reengineering. For example, to measure the results of its agricultural assistance activities, the mission has monitored the volume and value of exports for six crops: sweet onions, ginger, okra, snow peas, asparagus, and plantains. The mission has several activities aimed at promoting these exports, including providing technical assistance to farmers for production and marketing. Indicators the mission uses to measure intermediate results for other activities include the number of land titles issued by the government and the number of vocational center graduates employed. Before adopting these types of results indicators, the mission focused primarily on measuring the physical outputs supported by USAID assistance, such as the number of agricultural research projects conducted or the number of schools built. The mission has been using results indicators extensively for program management, including incorporating them in annual work plans for contractors and other development partners and in its annual performance review.
In El Salvador, the mission recently established performance indicators that are intended to measure the short-term results of its assistance projects. In 1996, the missions relied almost exclusively on macroeconomic statistics to report on the results of its economic growth-related projects. The mission revised its indicators in 1996 to better reflect the direct outcomes of its program activities. For example, new indicators include the number of people in rural areas that are active clients (that have an outstanding loan and/or a savings account) in participating financial institutions and the number of people in rural areas receiving assistance in management, agricultural technology, and marketing. However, because these indicators are new, they have not been used extensively by the mission for program management.
In the Philippines, the mission had developed an extensive set of performance indicators, which it reported in 1996. However, mission officials acknowledged that these indicators measured development results that were beyond the scope of their activities in that country. Since then, the mission has improved many indicators. For example, indicators for projects in Mindanao have been revised to more accurately reflect the desired results of the individual activities. The indicators include the total value of USAID-facilitated or -assisted private investments and the number of government policies or practices modified to facilitate rapid and equitable economic growth. Although many of the mission’s indicators are new, the mission has collected historical data against which to measure future progress.
In Bangladesh, the mission was restructuring its strategic objectives and revising its indicators. Many of the mission’s indicators for its economic growth objective had focused on results at the national level and not specifically on areas targeted in USAID’s activities. The mission is now more closely aligning its indicators with its activities to reflect what can feasibly be achieved, given its economic growth resources. Some of the mission’s more meaningful indicators include the amount of fertilizer and improved seed marketed nationally and the percentage of the population with access to disaster relief supplies within 72 hours. Some historical data were available for comparison with the current indicators, but the mission plans to use a new data collection methodology for one key indicator being developed, and baseline data are not yet available.
section_title: USAID Often Unable to Demonstrate Impact of Projects on Mission Objectives and Agency Goals
paragraphs: Projects are expected to contribute to the achievement of missions’ strategic objectives and USAID’s overall, long-term goals. USAID is working to develop better ways to measure the extent of its contribution to a country’s development. According to USAID officials, internal and external factors, including political instability and the level of other donors’ assistance, affect the missions’ country-level indicators. According to USAID management, it is not possible to measure, in a quantifiable and precise way, the impact of USAID activities alone on country-level development indicators. Therefore, missions’ strategic objective indicators are intended to reflect the results not only of mission projects but also of the efforts of other development partners. The collective impact of activities at the four missions we visited was not significant enough to have a measurable impact on country-level indicators. USAID does not claim responsibility for the development results measured by its strategic objective indicators but rather claims a “plausible association” with the results.
Missions usually measure progress toward their strategic objectives using country-level development indicators. In the countries we visited, indicators of progress toward economic growth objectives included the number of people employed nationally in the agricultural, industrial, and service sectors in Honduras; percentage of the total population in El Salvador with access to potable national ratios of total exports and imports to gross domestic product in per capita gross domestic product in Bangladesh.
At each of the four missions we visited, mission documents showed or USAID officials acknowledged that the activities did not significantly affect country-level indicators of economic growth. For example, for 1997, the Bangladesh mission reported that the per capita gross domestic product growth and agricultural and industrial employment had improved but that “given USAID’s relatively small investments in this strategic objective, we cannot claim major achievements at this national level.” In El Salvador, the mission reported that its projects were not sufficient to achieve its economic growth strategic objective. The mission’s economic growth portfolio is composed of a limited number of projects that focus primarily on microfinance, small business, basic education, small-scale agriculture, and policy reform. The mission in El Salvador reported that achieving the mission’s strategic objective depends on major contributions from other partners, especially international banks, which are expected to provide most of the funding required for activities relating to land, policy, and infrastructure.
Internal and external factors can have a profound impact on macroeconomic conditions and thus on country-level indicators. Such factors include political instability, the commitment of political leaders to necessary reforms, the magnitude and effectiveness of assistance from other bilateral and multilateral donors, weather conditions that affect crop yields, and the stability of international markets. Recent mission reports for both Bangladesh and the Philippines cited external factors as explanations for the failure to achieve country-level development results. The Bangladesh mission reported that political turmoil and power shortages were the principal causes of shortfalls in expected economic growth. The Philippines mission reported that the value of direct exports from Mindanao was below targeted levels, primarily due to the decline in pineapple prices and the loss of European markets for bananas.
Other observers have recently noted USAID’s difficulty in demonstrating its impact on broad development indicators. In March 1997, USAID’s Inspector General concluded that in most cases, USAID’s goals and objectives exceeded the agency’s span of influence and that it was therefore extremely difficult for USAID to take credit for improving country-level results. The Office of Management and Budget similarly noted in 1996, after reviewing USAID’s strategic plan and indicators, that it was very difficult to credit USAID projects with progress in country-level indicators.
USAID is developing common indicators that missions will use to consistently measure progress in key areas worldwide. These measures are intended to help the agency aggregate the results of its various missions’ programs to show progress in achieving overall agency goals. However, the use of common indicators will not resolve USAID’s difficulty in attributing gains to its programs at the country level. According to a senior agency official, a key criterion for using these common indicators, as with the mission-specific strategic objective indicators currently used, will be that USAID can show a “plausible association” with the results, not that the results are attributable to USAID assistance.
The extent of USAID’s plausible association with the country-level results is not reflected in the missions’ documents. In some cases, especially in countries where USAID is the largest donor and projects are showing some results, this association may be very strong, while in other cases, the association may be tenuous, based only on token USAID involvement. The agency has not clearly and consistently differentiated between levels of association with development results in its mission performance reports. USAID management acknowledged that it needs to do a better job of making the plausible associations clear in its strategic documents.
Citing broad development indicators can result in misleading reports on USAID’s performance. The reports we reviewed showed that when missions reported that specific strategic objective performance targets were met, they rarely mentioned other factors outside of USAID’s control that contributed to the results, such as assistance from other donors, actions taken by the host governments independently, or favorable international market developments. Furthermore, the USAID Inspector General recently reported that USAID seemed to be taking credit in its 1996 Agency Performance Report for some high-level impact merely because it had projects in that program area.
section_title: Program Evaluations Can Demonstrate Effectiveness but Are Not Routinely Done
paragraphs: Although some of the missions we visited had done program evaluations that could be used to link project results with country-level development indicators, USAID’s current performance measurement system relies largely on indicators to assess the agency’s impact. However, in our recent report on GPRA implementation, we noted that demonstrating the impact of a government program on outcome indicators is difficult and a common problem among federal agencies that are implementing performance measurement systems. The most difficult aspect of analyzing and reporting performance data is separating the impact of a program from the impact of external factors to measure the program’s effect. We noted that in these cases “it may be important to supplement performance measurement data with impact evaluation studies to provide an accurate picture of program effectiveness.”
Systematic evaluations of how a program was implemented can provide managers with important information about a program’s success or failure. For example, in Honduras, a USAID contractor’s evaluation of a basic education project confirmed that USAID activities in that country, implemented from 1986 to 1995, had a significant impact on key country-level indicators of improvement of the educational system in that country relative to other factors.
Before October 1995, missions were required to evaluate each project upon completion and submit the results to USAID’s Center for Development Information and Evaluation. However, current USAID guidance allows missions to decide whether they will do performance evaluations of their activities. In one mission we visited, a senior USAID official said that the mission would rely on performance indicators, not on evaluations, to monitor its projects’ progress and results. He said that evaluations would be done primarily when implementation problems arise. At other missions, USAID officials also suggested that evaluations would probably be done less often than in the past.
According to the Director of USAID’s Center for Development Information and Evaluation, the agency’s revised policy on evaluation was not necessarily intended to reduce the frequency of evaluations at the missions, but to better target them to the management needs of the mission. He said that missions are expected to do adequate evaluations to support statements about performance included in mission reports. However, documents we reviewed rarely cited evaluations to support their descriptions of performance in economic growth strategic objectives. Officials from the Center and a mission we visited told us that they believed the agency needed to reassess its policies on evaluation and mission practices to ensure that the agency is sufficiently and appropriately using evaluations for effective management.
According to USAID management, as missions improve performance monitoring, there will be less need for routine impact evaluations at the activity level. Eventually, fully operational performance monitoring will result in an adequate assessment of the impact of USAID’s activities in a country. In addition, missions could supplement performance measurement data by obtaining reviews of other donor programs and host-country efforts. Nevertheless, periodic, independent evaluations of USAID’s funded activities can be a useful tool to validate program accomplishments.
section_title: Mission Performance Is Being Linked to Resource Allocation
paragraphs: USAID has begun using program performance in its resource allocation process in a more systematic manner. However, the process continues to evolve, and resource allocation decisions are affected primarily by foreign policy considerations of the executive branch and congressional priorities. Each of USAID’s regional bureaus used different formulas and assigned different weights for performance when constructing their position on USAID’s fiscal year 1998 budget request. However, guidance for the fiscal year 1999 budget cycle attempts to standardize the bureaus’ assessment processes, including the weight accorded to performance in budget deliberations.
subsections: section_title: Revised Resource Allocation Process Focuses on Program Contributions Within Each Agency Goal
paragraphs: A basic principle underlying USAID’s reform efforts is to divide its assistance program funding among its strategic goals in line with carefully crafted strategic plans and systems for performance measurement. USAID now includes more explicit and systematic consideration of program performance in its resource allocation process.
Mission officials told us USAID’s emphasis on performance and results was good, but expressed mixed views on USAID’s allocation process. Officials generally said political and foreign policy considerations continue to dominate the budget process. While mission officials said the process is more transparent than in the past, some said that USAID should be more consultative with the missions regarding the allocation of earmarked and/or directed funding. One mission director said there is little linkage between the mission’s performance, as measured against the mission’s “results contract” with USAID, and the allocation of resources to the mission.
USAID’s process for assessing performance and allocating resources continues to evolve, and modifications continue to be made in reporting requirements and procedures for weighing program results. Performance has been a factor in allocating resources. For example, when we observed headquarters technical sessions for rating performance, we learned that USAID was eliminating an agricultural loan program in Guinea due to low rates of loan disbursements. We also learned that USAID was cutting assistance to El Salvador’s electoral commission due to its poor performance in implementing needed reforms. Performance assessments may become more meaningful in the process as missions provide more complete performance information. In the Asia and Near East Bureau, the number of activities that reported complete performance information rose from 31 percent in 1995 to 59 percent in 1996.
USAID allowed regional bureaus latitude in how they implemented the reporting process for the 1998 fiscal year budget request, as well as for identifying priorities for fiscal year 1997 resource allocations. The Asia and the Near East Bureau scored and ranked, within each of the agency’s four sustainable development goals, each strategic objective by performance, contribution to agency and bureau priorities, contribution to foreign policy goals, and the extent to which the host country had been a good development partner for that objective. The ranking became a starting point for further discussions on what activities should be funded. The Bureau for Africa also used a similar approach, scoring and ranking strategic objectives within each sector on performance, pipeline, host-country performance in the sector, sectoral need and magnitude of the problem, and contribution to regional initiatives and priorities for that sector. The Bureau for Latin America and the Caribbean scored and ranked each strategic objective by performance, then scored and ranked countries by performance, foreign policy interest (such as funding Haitian democratization efforts or the Guatemalan peace process), severity of need, and commitment to free market policies and democratic governance. These scores were used to place countries in four funding categories; those in the highest category received funding priority, after allowing for funding needs based on analysis of the pipeline and for meeting targets for earmarks and directives.
The USAID Administrator issued guidance in January 1997 to standardize the bureaus’ resource requests, performance assessments, and resource allocation processes for the fiscal year 1999 budget request. Among other things, resource requests and allocations are expected to be based on strategic objective performance and its significance to the strategic plan. Bureaus are to use three common factors and common weights to measure performance and allocate resources: performance (35 percent), contribution to agency goals (30 percent), and contribution to development initiatives (35 percent). While this guidance provides a more standard ranking and scoring format, these groupings still allow wide latitude for including factors that are still considered in the allocation process. For example, contribution to agency goals includes foreign policy objectives, and contribution to development initiatives includes bureau initiatives, country and/or sectoral need, and the quality of the development partnership in general and within specific goal areas.
We attended several USAID program review meetings during this year’s deliberations on the fiscal year 1999 budget at the Africa, Asia/Near East, and Latin America bureaus. Generally, the initial discussions on strategic objective performance were chaired by technical staff from the regional bureaus with varying representation from the Global, Management, and Policy and Program Coordination bureaus. The Asia/Near East and Latin America bureaus held several levels of reviews in Washington for each country, often with senior mission management and mission program staff. Due to the large number of missions in Africa, the Washington teams conducted the initial assessment based on the submissions and, in some cases, additional information sent from the field. The Africa Bureau held subsequent 1-day program reviews with staff from full sustainable development missions and with missions and operating units with upcoming strategy reviews. Following the rounds of technical and country meetings, bureaus conducted regional wrap-up sessions to reach resource allocation decisions and prepare for budget reviews with agency management.
At the technical meetings we attended, we noted that the teams had reviewed the documents, and, in some cases, received additional information from the missions. Team members that had been given guidance on the criteria for scoring performance appeared well prepared, usually supporting their scores with specific examples of how strategic objectives were or were not meeting targets or making an impact. In some cases, team members discussed the quality and relevance of performance indicators and targets and the extent to which impact could be attributed to USAID assistance. We also heard some limited discussion on the quality of the data used to measure performance.
This is only the second year of USAID’s revised budget process, and USAID officials acknowledged that modifications will continue to be made as best practices are identified. We noted at several meetings that the team members and mission staff were confused about some of the mission reporting. For example, the guidance asked missions to report on the intermediate results that were most central to achieving the goals of the strategic objectives. However, there was discussion about whether the reported information was the best reflection of performance. Some team members cabled to missions for additional information and used this in their scoring; others did not. The quality of the reports and the extent to which they provide an accurate and complete story of each strategic objective are key factors in performance assessment.
section_title: New Management System Is Not Working
paragraphs: Although USAID will have spent about $100 million by the end of fiscal year 1998 to develop its New Management System (NMS), the system does not work as intended and has created serious problems in mission operations and morale. The system is key to successful information sharing required under reengineering for accountability and control. The agency deployed the system worldwide, knowing that it was not fully operational or adequately tested. Indeed, USAID’s Inspector General reported 6 months after implementation of the system that it was not complete, had not been demonstrated to work effectively, and had not been adequately tested.Because of problems with the system, USAID suspended use of part of the system in the missions in April 1997 and is now taking corrective steps recommended by the Inspector General.
subsections: section_title: System Is Key to USAID Reforms
paragraphs: Since 1994, USAID has been developing the NMS to support its organizational reforms. The system is designed to consolidate USAID’s accounting, budget, personnel, procurement, program operations, and property management into a single, integrated network that the agency’s missions and offices worldwide can access and to aid in the effective management and monitoring of its programs. The purpose of the system is to make financial management more efficient by streamlining business processes, eliminating paper forms, ensuring its compliance with federal accounting and financial management requirements, and providing managers with information needed to make appropriate decisions and reliably report the status of USAID projects; facilitate missions’ program delivery by providing the means for missions and offices to share information on-line about program management; and empower missions and provide USAID management with a means for maintaining accountability.
The system was activated at USAID headquarters in July 1996 and at the missions in October 1996. As of October 1996, four of the six system modules—the accounting, budget, operations, and procurement modules—were operational, with some limitations, at USAID headquarters. The personnel and property management modules are still in development. According to USAID’s Office of the Inspector General, the agency will have invested about $89 million by the end of fiscal year 1997 and about $100 million by the end of fiscal year 1998 to develop the system.
section_title: Dysfunctional NMS Has Hampered Operations at Missions
paragraphs: The four missions we visited could not routinely use NMS to successfully execute financial management functions such as obligating funds and recording procurement actions. According to mission officials, having a dysfunctional system limits the benefits of the missions’ reengineered operations. The Honduras mission director told us that “the reform process has now reached a point where some reforms as contemplated simply cannot go much further without a functioning NMS.”
For several months, the mission in Honduras could not obligate allocated funds for disaster relief on the country’s north coast due to problems in the system’s accounting and procurement modules. Also, because of the system’s long response times, entering information into the system was often very time-consuming and frustrating for mission staff. Officials in Honduras attempted to run some procurement actions while we observed: the actions took from 2 to 6 hours to process, and none could be successfully completed. In fact, because of these problems, some mission officials we visited had two personal computers on their desks: one for using NMS and one to do other work. In El Salvador, some mission personnel told us that all other work frequently stopped while NMS was in use and the system was having a negative impact on implementation of mission projects.
In addition to citing excessive system response times as a major problem, the agency’s Chief of Staff noted potential problems with the system development effort. He said that because each module was independently developed, the database used for NMS had some inconsistencies. For example, in one module, for some data fields the system will accept eight characters, while another module will accept only six characters for the same information. Such inconsistencies can cause problems when attempting to transfer data from one module to another.
In March 1997, USAID’s Inspector General reported that the system’s premature deployment had increased the risk of fraud and abuse, had not met users’ needs, and did not meet basic federal financial system requirements. These findings were consistent with our observations. The Inspector General made a number of recommendations to USAID management, including suspension of part of the system until many of its problems were resolved.
In April 1997, following the Inspector General’s report and comments from system users and us, USAID suspended the NMS accounting and procurement modules and left the budget and operations modules operational in the missions. These systems are expected to remain shut down until at least fiscal year 1998. All four modules will still be used by headquarters offices, which have not had as much trouble in using NMS as have the missions.
USAID said it is taking the following steps to address problems with the NMS:
A new Director of Information Resources Management, who will focus exclusively on managing information resource activities, has been hired.
The Administrator has announced that he will designate a senior official as the agency’s Chief Information Officer, who will be certified by the Office of Management and Budget as qualified for the position and report directly to the Administrator.
A full-time NMS project manager has been selected to supervise the development efforts until the system is operational.
Efforts have been undertaken to analyze the technical and implementation problems that currently limit NMS from achieving its full potential.
A goal to achieve level 2 of the Software Engineering Institute’s Capability Maturity Model (CMM) for software development has been established.
A statement of work has been developed to assess the accounting and procurement components of NMS and their quality and to identify the risks and opportunities in the application code that the components use.
A transition to performance-based contracting for NMS and a reduction in both the number of contract entities and contractors are underway.
Development activities are limited to the portions of NMS that are still operational and needed to establish the core functionality in the accounting module.
section_title: Conclusions
paragraphs: It is too early to discern the full extent of the impact of reengineering efforts at USAID missions, as they have only recently made the investment and operational changes necessary to bring about long-term change. However, USAID’s reengineering efforts have produced some tangible benefits in the areas of planning, implementing, and monitoring assistance projects. Missions are adapting to the strategic approach to planning, but missions’ progress in changing their portfolios of projects has been constrained by factors such as funding limitations and obligation authority. The extent to which USAID can capitalize on operational changes will depend on continued vigilance by agency management to ensure that these changes take hold and continue. The ultimate test will be whether USAID can achieve sustained efficiencies in project planning and implementation and demonstrate the impact of its projects. With limited resources available for foreign assistance, USAID’s ability to target its assistance effectively and deliver it efficiently is especially important.
Since information-sharing is critical to USAID’s new operating system, diligence in properly deploying NMS is key to the reform process. While NMS promises to aid in more efficient management of USAID resources, many of the efficiencies expected to result from this system have been delayed and offset by unsuccessful implementation. USAID has taken positive steps to address the concerns discussed in this report and the recommendations in its Inspector General’s report. However, whether these actions will correct the NMS’ problems will depend on the successful implementation of these efforts. In addressing the problems with NMS, it is USAID management’s responsibility to ensure that NMS facilitates, rather than impedes, the reform process.
section_title: Agency Comments
paragraphs: In commenting on a draft of this report, USAID stated that the report was “a fair, balanced, and thoughtful assessment of the state of overseas missions’ reengineering efforts.” USAID comments provided some suggested clarifications to the draft report and additional information on agency efforts to address problems associated with the New Management System, which we have incorporated in the text of our report, as appropriate. USAID’s letter, without attachments, is reprinted in appendix IV.
section_title: Scope and Methodology
paragraphs: As requested, we examined how USAID missions have reformed their operations and developed a results-oriented program. In making our selections for overseas fieldwork, we asked USAID to recommend five countries in each of its regions where reengineering has progressed to the point that we could see results and the economic growth objective was a priority. USAID provided us program budget data on all countries in which it operates and five country recommendations per region, along with explanations for their selections. Following discussions with USAID officials and congressional Committee staff, we selected El Salvador, Honduras, Bangladesh, and the Philippines for fieldwork. We based our country selections on the size of the USAID program, importance of the economic growth strategic objective in the overall program, and pipeline size. We had done fieldwork in the Dominican Republic for the first part of the assignment. We focused on Latin America and Asia because of the greater emphasis on economic growth in these regions; we excluded Central Europe and the former Soviet Union region from consideration since these countries’ programs are relatively new, transitional, and expected to end soon.
To assess (1) how USAID missions have reengineered their operations and (2) how USAID monitors and evaluates the results of its projects, we interviewed USAID officials in the Management, Global, and regional bureaus and the Bureau for Policy and Program Coordination, including the Center for Development Information and Evaluation in Washington, D.C.; reviewed agency policies, practices, and procedures; and analyzed agency documentation, including evaluations, reporting documents, and Inspector General reports. At the missions visited, we interviewed USAID management and staff, held discussions with foreign service nationals on the impact of reforms and reengineering, and collected and analyzed relevant documentation. We also interviewed representatives of nongovernmental and private voluntary organizations and USAID contractors that implement USAID programs and officials of international financial institutions, other donor countries, and the host governments.
To assess how reengineering has affected the delivery of development assistance, we interviewed USAID officials, partners, and customers about reengineered program design and delivery and reviewed documentation on mission strategies, plans, and projects. In reviewing portfolio changes at the missions we visited, we assessed the active, completed, terminated, or initiated projects for each year between fiscal year 1994 and 1997.
To determine how USAID allocates funds for its projects, we interviewed USAID officials, attended regional bureau budget sessions, examined the budget process, and reviewed recent and current budget documents and agency guidance. We assessed how the budget process relates the various factors that the agency uses to measure and rank its performance to provide some insight as to whether the budget process is meeting expectations.
To assess how USAID’s NMS supports mission operations, we met with officials of USAID’s Office of the Inspector General and reviewed their reports on NMS. We reviewed agency documentation, including agency cable traffic and e-mails in Washington, D.C., and the field missions visited; USAID’s general notice system and policy directives, guidance, and statements; and congressional testimony by the Administrator. We interviewed USAID officials and agency staff who use NMS in Washington, D.C., and in the field missions and observed mission officials’ and staff members’ use and/or tests of NMS. We reviewed our prior reports: a December 1992 report identifying weaknesses in federal information resources management that frequently led to costly projects that show disappointing results, numerous reports and testimonies since 1993 on information management overall, and a February 1997 report noting that failure to follow disciplined management and system development practices could lead to costly failures.
We performed our work from October 1996 to May 1997 in accordance with generally accepted government auditing standards.
We are providing copies of this report to the Chairmen and Ranking Minority Members of the House and Senate Committees on Appropriations, the Senate Committee on Foreign Relations, the House Committee on Government Reform and Oversight, and the Senate Committee on Governmental Affairs, and to the Ranking Minority Member of the House Committee on International Relations. We are also sending copies to the Administrator, U.S. Agency for International Development, and to the Director, Office of Management and Budget. Copies will be made available to others upon request.
Please contact me at (202) 512-4128 if you or your staff have any questions about this report. Major contributors to this report are listed in  appendix V.
section_title: USAID Obligations, Fiscal Years 1995-97
paragraphs: 1997 (est.)
Population, health, and nutrition.
The amounts for “Other” include the $1.2 billion provided to Israel each year.
Includes the U.S. Agency for International Development (USAID) and USAID Office of Inspector General operating expenses and Foreign Service Retirement and Disability funds.
Not applicable.
section_title: Directed and Undirected Funds for Development Assistance and Related Accounts, Fiscal Years 1995-97
section_title: Chronology of USAID Reforms
subsections: section_title: March: Strategies for sustainable development announced
section_title: March: Agency strategic implementation guidelines issued
paragraphs: September:  One-year experiment on reengineered program  operation processes completed September:  Agency strategic framework and indicators for FY  1995-96 issued October:   New program operations procedures issued October:  Agency Automated Directives System implemented October:  Partial deployment of New Managment System  (NMS)
section_title: March: Reengineering best practices report issued (based on test of reengineered processes in 10 missions)
paragraphs: July: NMS activated in Washington, D.C.
October:  NMS deployed worldwide February:  REFORM Advisory Group visits selected missions to  assist with reengineering 3 mission closings planned Staff reduced by 417 (as of  6/30/97)
April: Two modules of NMS suspended at missions  $5.9 billion (estimated) 7,609  employees  (as of  6/30/97)
section_title: Comments From the U.S. Agency for International Development
section_title: Major Contributors to This Report
subsections: section_title: National Security and International Affairs Division, Washington, D.C.
section_title: Office of General Counsel, Washington, D.C.
paragraphs: Ernie E. Jackson The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (202) 512-6061, or TDD (202) 512-2537.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.