International Food Assistance: USAID Is Taking Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid, but Weaknesses in Planning Could Impede Efforts
Released: 2009-09-28T00:00:00

Letter

Background

USAID Plans to Administer Larger Nonemergency Food Aid Programs in Fewer Priority Countries

FFP is shifting to larger programs in fewer countries in order to leverage nonemergency food aid resources more effectively. FFP’s Strategic Plan for 2006-2010 concentrates on a single objective of reducing food insecurity in vulnerable populations with an aim to fund programs that can be modified when circumstances change. In fiscal year 2008, under the Food for Peace Act, the United States provided $354 million in nonemergency food aid and FFP administered nonemergency food aid programs in 28 countries. Beginning in fiscal year 2010, FFP will limit its nonemergency food aid programs to 20 priority countries. Of these, 16 are in sub-Saharan Africa (Burkina Faso, Burundi, Chad, the Democratic Republic of Congo, Ethiopia, Liberia, Madagascar, Malawi, Mali, Mauritania, Mozambique, Niger, Sierra Leone, Sudan, Uganda, and Zambia), 2 are in Central and South Asia (Afghanistan and Bangladesh), and 2 are in Latin America and the Caribbean (Guatemala and Haiti). Figure 1 lists the countries that received nonemergency food aid in fiscal year 2008 and the most recent available statistics on the number of undernourished people and the prevalence of undernourishment in these countries, respectively.

Food for Peace Nonemergency Food Aid Programs Are Implemented through Grants for Single-Year and Multiyear Assistance Programs

FFP provides food aid funding under the Food for Peace Act using two primary funding mechanisms—Single-Year Assistance Programs (SYAP) and Multiyear Assistance Programs (MYAP). SYAPs address emergency needs and are used for direct distribution of food aid in emergency contexts. MYAPs, long-term programs with duration of 3 to 5 years, are used to implement development activities that target chronically food- insecure populations. In carrying out nonemergency programs through its implementing partners, FFP uses a combination of Food for Peac e commodities for direct distribution, local currency generated through and cash resources to implement activities that target monetization, chronic food insecurity. (For illustrative examples of the types of activities supported by nonemergency food aid, such as project activities we observed in Bangladesh and Haiti, see app. III.)

Monitoring and Evaluation Are Essential to Effective Program Management

Monitoring and evaluation perform two separate but related functions. Monitoring is the collection of data to determine whether programs are being implemented as intended and the tracking of progress using preselected indicators throughout the life of the program. Evaluation consists of periodic or ad hoc studies to assess how and whether programs achieved their expected goals. It can also consider what outcomes can be attributed to the program, as well as program cost-effectiveness. Evaluations can rely on a range of quantitative and qualitative measures as well as preselected indicators, comprehensive research designs, and the use of appropriate statistical analysis of the data. Figure 2 outlines a framework based on our review of Food for Peace nonemergency programs, as well as AEA and WFP guidance that illustrates how monitoring and evaluation could be integrated with program management of nonemergency food aid.

Program Management

Determine if necessary inp uts (equipment, supplies, and personnel) are in place. – Ensure that food is properly stored. – Check to see if health clinics are sufficiently stocked for intended beneficiaries.

Determine if programs are achieving their expected outputs by tracking performance indicators.

Ensure that target number of beneficiaries are receiving food and attending clinics.

Focus on the extent to which program objectives were achieved and what outcomes can be directly attributed to programs.

Mother and child health and nutrition programs may consider how the ration is affecting the health and nutritional status of beneficiaries. programs may focus on how the activity is addressing temporary food insecurity and developing a sustainable community asset.

FFP Relies on a Range of Staff to Perform Various Types of Monitoring and Evaluation Activities

FFP relies on a range of staff to perform its monitoring and evaluation— including FFP officers in the field, implementing partners, contractors, and FANTA-2 staff. According to FFP, field staff visit project sites to ensure that programs are being implemented as expected. They also approve and oversee the implementing partners’ selection of performance indicators and data collection. In addition to regularly collecting performance data, the implementing partners perform baseline studies and midterm evaluations for their multiyear programs and hire external evaluators to conduct the final evaluations for the multiyear programs. In addition, FFP relies on appropriately credentialed consultants and researchers for some cross-cutting evaluations, which, for example, consider the agency’s exit strategies for nonemergency food aid programs in Bolivia, Honduras, and Kenya.

Conducting evaluations can present many challenges, such as obtaining reliable data and employing qualified evaluators. These challenges can be compounded in developing countries, where local conditions may be difficult and local capacity is often limited. For example, a recent review of evaluations conducted by an international NGO cited numerous challenges in-country including climate, poor economic status, civil unrest, migration of beneficiaries, cultural biases, poor recordkeeping, and turnover in government and NGO staff.

Table 1 summarizes the various types of monitoring and evaluation activities that FFP performs. The table also indicates which staff have primary responsibility for the activities, briefly describes the activities, and provides examples of them.

Weaknesses in Planning Could Hinder USAID’s Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid Programs

USAID has begun taking a series of actions to improve its monitoring and evaluation of nonemergency food aid programs, as mandated by the Food for Peace Act passed in July 2008. These efforts include (1) increasing the number of field staff responsible for the monitoring of nonemergency food aid programs; (2) expanding FEWS NET; (3) upgrading FFP’s information technology system; (4) expanding the use of Layers, a computerized monitoring system for quality assurance; (5) improving the quality of evaluations, assessments, and targeting approaches; and (6) launching a pilot project to improve market analysis in countries receiving U.S. nonemergency food aid. Weaknesses in planning could hinder USAID’s ability to effectively implement these actions, but it is too early to assess impact because the agency’s efforts are still in progress. As of August 2009, USAID estimated that, in fiscal year 2009, it will have spent about $13 million of the $22 million authorized for monitoring and evaluation of nonemergency food aid under the Food for Peace Act. Figure 3 summarizes the allocation of the funds and the implementation timelines.

USAID Has Allocated New Funding to Increase Staffing Levels and Upgrade Information Systems but Has Not Developed Key Processes and Plans in These Areas

Staffing Levels

USAID has announced plans to increase the number of field staff responsible for the monitoring of food aid programs, which would provide minimal FFP presence in all priority countries. The agency plans to use approximately $5 million of its new Food for Peace Act funding to add 21 full-time field staff whose responsibilities include the monitoring of nonemergency food aid programs. According to USAID, 1 of the new positions is located in USAID’s Southern Africa Regional Office in Pretoria (South Africa) while the other 20 new positions are or will be located in designated priority countries for nonemergency food aid programs (including 19 in sub-Saharan Africa and 1 in Guatemala). As of August 2009, all 21 of the new positions had been established, of which 9 had been filled. Figure 4 shows the location of the new positions, including those filled, in FFP’s 20 priority countries for nonemergency food aid.

We found that USAID does not have a systematic process in place to determine appropriate staffing levels for effective monitoring and evaluation. According to USAID officials, creating new positions in field missions where FFP has lacked staff was the primary consideration for their allocation. Hence, 17 of the new positions are in countries where USAID runs nonemergency food aid programs but has lacked FFP program management staff to date, namely: Burkina Faso, Burundi, Liberia, Madagascar, Malawi, Mali, Mauritania, Niger, Sierra Leone, and Zambia. The remaining 4 new positions augment existing FFP program management staff in the Democratic Republic of Congo, Guatemala, Sudan, and the Southern Africa Regional Office in Pretoria.

Information Technology System

USAID plans to replace its current Food for Peace Information System with a new Food for Peace Management System (FFPMS) that it expects will address several of FFP’s current information technology issues, including the collection, tracking, and interfacing of food aid program data. According to USAID’s December 2008 report to Congress, the new FFPMS is expected to involve collaboration and testing with users in each business area in which the system is used. These areas include budgeting, proposal, document management, commodity purchase, commodity booking and shipping, tracking, monitoring and evaluation, reporting, closeout, and interface with internal and external information systems. In addition, USAID officials explained that, unlike FFPIS, FFPMS is intended to be an all-inclusive system that will fully capture the movement of U.S. food aid commodities, used for both emergency and nonemergency purposes, from the initial transfer authorization to the commodity’s final distribution site. FFPMS is also expected to (1) capture implementing partners’ data on Food for Peace resources, including revenues from monetization transactions as well as costs incurred; (2) allow FFP to aggregate information for its implementing partners’ annual results reports; and (3) have the capability to interface with USAID internal systems—such as USAID’s core financial system, known as Phoenix, which captures all USAID transactions—as well as with external agency information systems with which FFP conducts business to allow the collection and sharing of information among and between agencies. With systematic and electronic collection of information, USAID anticipates being able to monitor the relative efficiency of its nonemergency food aid programs and adequately monitor the degree to which monetization revenues cover costs.

USAID is in the initial stages of this planned information systems upgrade. In June 2009, USAID awarded a contract for $637,000 to develop a management strategy and framework for implementing FFPMS by the end of fiscal year 2010. The contractor’s responsibilities, as defined in the contract, are to revalidate assumptions made within the 2005 requirements and architecture diagram; revalidate FFP’s technical environment; and update the requirements, documents, and architectural design. The contractor is also expected to make a recommendation to USAID by December 2009 for a commercial off-the-shelf software package that would best fit FFP’s needs.

A concept of operations document is a critical element in developing information system requirements and testing activities. However, USAID has not developed such a document for this initiative. This document is normally one of the first documents produced during a disciplined development effort because it describes system characteristics for a proposed system from a user’s point of view. The key elements that should be included in the document are major system components, interfaces to external systems, and performance characteristics, such as speed and volume. A concept of operations document defines how an organization’s day-to-day activities will be carried out to meet mission needs and includes high-level descriptions of information systems, their interrelationships, and information flows. It also describes the operations that must be performed, who must perform them, and where and how the operations will be carried out.

Famine Early Warning Systems Network

To provide timely and rigorous early warning and vulnerability information on emerging food security issues, USAID is making a contribution to the FEWS NET fiscal year 2009 operational budget. FEWS NET monitors and analyzes relevant data, using multiple technologies such as satellites and field observations, to identify threats to food security and their impacts on livelihoods and markets. Chemonics, the private firm responsible for implementing FEWS NET along with several U.S. government agencies, currently maintains 20 national and 3 regional field offices in sub-Saharan Africa, Central America, Haiti, and Afghanistan. Anticipating an increase in food insecurity in countries where there is no FEWS NET presence, Chemonics plans to implement three remote monitoring pilot projects—in Burundi, Pakistan, and Yemen—that are expected to be completed by the end of fiscal year 2010.

In fiscal year 2009, USAID is providing $7.6 million of the newly authorized Food for Peace Act funding to support FEWS NET. This funding is not designated to support the remote monitoring pilot projects but is a contribution to FEWS NET’s operational budget of $20 million, with the remaining $12.4 million coming from the Development Assistance (DA), the Office of Foreign Disaster Assistance (OFDA), and other sources.

Legal Restrictions on Usage of Food for Peace Act Funding Could Hamper USAID’s Plans to Expand FFP’s New Monitoring System and Improve Evaluations

Plans to Expand FFP’s Computerized Monitoring System for Quality Assurance

Legal restrictions on the allowable uses of Food for Peace Act funding could hamper USAID’s plans to expand FFP’s new monitoring system for quality assurance. USAID plans to expand the use of Layers—a computerized system using personal data assistant (PDA) devices for monitoring the implementation and management of nonemergency food aid programs—from 3 pilot countries to 20 countries by the end of 2012, through its multiyear cooperative agreement with AED. The purpose of Layers is to help FFP staff responsible for the monitoring of food aid programs to collect data to assess the quality of implementing partner operations in the field. Layers can be used to monitor a wide range of programs and activities, including the conditions of storage in commodity warehouses and the quality of services delivered by FFP’s implementing partners, such as food-for-work activities, provision of community health services, and maternal and child health and nutrition. Prior to 2009, Layers was piloted in Haiti, Ethiopia, and Madagascar. USAID estimates the one- time cost for the start-up to introduce Layers to a new country to be approximately $144,000. It began rolling out the system in Guatemala, Mali, and Uganda in 2009 and plans to do so in all the remaining priority countries by the end of 2012 as their multiyear assistance programs are approved. However, in May 2009, USAID’s General Counsel determined that the Food for Peace Act precludes the agency from using the newly authorized funding for nonemergency food aid monitoring and evaluation for grants and cooperative agreements. Section 3012(f)(5) of the Food for Peace Act authorizes USAID to enter into personal services contracts to carry out monitoring and evaluation but does not refer to grants or cooperative agreements. As a result, FFP cannot move forward with its original plans to use Food for Peace Act funding to expand the implementation of Layers and will instead need to use other sources of funding that could be made available for this purpose. According to FFP officials, the current plan is to use other funding streams, such as the DA account, to fund the costs of the start-up of Layers and the first year of the system’s operation. However, these officials noted that due to competing priorities for DA, it cannot be relied upon as a stable source of funding.

Plans to Improve the Quality of Evaluations

USAID plans to improve the quality of evaluations through a number of ongoing and future studies. Using $5,000 of the newly authorized funding for monitoring and evaluation, the agency has signed a contract with Michigan State University for a study of monetization that is expected to be available for release in December 2009. It has also commissioned several cross-cutting evaluations and related studies that are expected to improve the quality of assessments, targeting approaches, design, and evaluation of nonemergency food aid programs. For example, through a cooperative agreement with AED, USAID is funding the FANTA-2 project to initiate the following multiyear studies in 2009: a study of exit strategies in Bolivia, Honduras, and Kenya, which will produce recommendations for effective ways to sustain program benefits after U.S. nonemergency food aid programs have been phased out; a study to identify key components of success in Preventing Malnutrition in Children Under 2 Approach (PM2A) programs in Guatemala and Burundi; a study to improve the early warning and response capacity in FFP’s multiyear assistance programs in Haiti, Niger, and Uganda; an update of the 2002 Food Aid and Food Security Assessment, which will evaluate progress made under FFP’s Strategic Plan for 2006-2010; and a study testing the efficacy and safety of emergency food products (EFP) that is co-funded by USAID’s Bureau for Global Health and FFP.

In addition to these cross-cutting evaluations and studies, FANTA-2 prepared four food security country frameworks—for Bangladesh, Burkina Faso, Liberia, and Sierra Leone—in 2009. Food security country frameworks are assessments that provide country-specific guidance to partners developing new MYAP proposals and describe how Food for Peace resources should be programmed and integrated with other U.S. government funds by defining objectives, approaches, regions, and institutional partnerships for effective use of U.S. government resources dedicated to reducing food insecurity. FFP estimates the cost of FANTA- 2’s work on the cross-cutting evaluations, related studies, and food security country frameworks in fiscal year 2009 to be $4.4 million.

To improve the quality of assessments and targeting approaches, FFP is also initiating a new cooperative agreement to identify best practices that will increase the efficiency and effectiveness of Food for Peace food aid program management and capacity building. The aim is to promote the broader adoption of best practices and collaboration among FFP’s implementing partners. As of August 2009, FFP had issued a Request for Applications seeking to identify potential partners’ interests, priorities, and capacities and estimates the cost of the award at $3.5 million.

However, USAID has determined that the Food for Peace Act precludes the agency from using the newly authorized funding for monitoring and evaluation of nonemergency food aid for grants and cooperative agreements. As a result, the completion of the multiyear cross-cutting evaluations, as well as future food security country frameworks and “best practices” assessments is not assured. FFP’s current plan is to fund these studies from the DA account.

USAID Is Using Other Funding to Improve Assessments Conducted Prior to Approving U.S. Food Aid Programs in Recipient Countries but Faces Implementation Challenges

In August 2008, USAID launched a 3-year Bellmon Estimation for Title II (BEST) pilot project intended to improve the market analysis required before food aid programs are approved in recipient countries—commonly referred to as the Bellmon analysis. The project is funded through resources other than the newly authorized Food for Peace Act funding for monitoring and evaluation of nonemergency food aid. It aims to improve how nonemergency food aid data are collected and analyzed, to determine the appropriate level of food aid, as well as the appropriate commodities that can be distributed and monetized in each country.

The pilot project reduces the potential for conflict of interest because the organization that does the analysis is no longer the same organization whose interest is directly affected by the outcome of the analysis. In the past, USAID’s implementing partners for nonemergency food aid distribution or monetization conducted and submitted the Bellmon analysis to USAID. However, the outcome of the analysis affected whether they could conduct monetization and, if so, how much food aid they could monetize—which, in turn, directly affected their operating budgets. Under the pilot project, USAID will have one organization, Fintrac, carry out the initial market analysis for all 20 of USAID’s priority countries. According to Fintrac, having one organization do the analysis for all countries could help to improve consistency in analytical methodology, reporting structure, and format.

BEST is still in its early stage of implementation. It encountered some difficulties in its first year of implementation and has yet to address some additional methodological challenges. One key challenge is ensuring the quality and consistency of the analysis, which can be impaired by a lack of reliable and up-to-date data. For example, Fintrac was able to use data from a recent livelihoods survey done by the World Food Program and FEWS NET in the Malawi assessment but had to conduct additional research and trips to Guatemala because it had no comparable updated data for that country. In addition, Fintrac officials told us that they have not developed quantifiable measures and thresholds agreed upon by USAID and its implementing partners to help guide USAID’s programming decisions on levels of monetization. Without quantifiable measures and thresholds, implementing partners could challenge USAID’s determination of monetization levels in a country. Finally, the law requires that the distribution of commodities will not result in a substantial disincentive or interference with domestic production or marketing in a recipient country—but the information Fintrac uses for its analysis, such as the historic levels of monetization relative to a country’s imports or production, does not directly answer the question of whether U.S. food aid will cause such disincentives. USAID and Fintrac officials acknowledged that additional research is needed in this area. As of July 2009, Fintrac had completed analyses for 7 priority countries: Burundi, Ethiopia, Guatemala, Madagascar, Malawi, Sudan, and Uganda, with plans to complete analyses for the remaining 13 priority countries through 2011.

USAID’s Monitoring and Evaluation of Nonemergency Food Aid Are Not Fully Integrated with Program Management

USAID’s monitoring and evaluation of its nonemergency food aid programs are consistent to varying degrees with some of the principles established by AEA to integrate evaluation into program management. We found that USAID’s actions were (1) generally consistent with the AEA principles for policies and procedures and for independence; (2) partially consistent with the principles for scope and coverage, dissemination of results, professional competence, and resources; and (3) not consistent with the principles for monitoring and evaluation plans.

USAID’s Actions Were Generally Consistent with the AEA Principles for Policies and Procedures and for Independence

FFP Has Issued Policies and Procedures for Monitoring and Evaluation

procedures and adopt quality standards to guide the conduct of evaluations. the kinds of evaluations to be performed and the criteria and administrative steps for developing evaluation plans and setting priorities; selecting the appropriate evaluation approaches and methods to be used; consulting subject matter experts; ensuring the quality of the evaluation products; publishing evaluation reports; and ensuring the independence of the evaluation function.

FFP has established policies and procedures for monitoring and evaluation that are contained primarily in several information bulletins for staff and implementing partners. Guidance can also be found in the FFP Process and Procedures Manual, which outlines field and headquarters staff’s roles and responsibilities for monitoring and evaluation. In addition, FFP has issued a series of technical papers by FANTA-2 that discuss the methodological standards and techniques that can be used for many aspects of monitoring and evaluation, such as statistical sampling for surveys of beneficiaries and preparing survey questions for key indicators. For example, one technical paper discusses in detail a key FFP indicator, household dietary diversity, and explains how to collect data through a questionnaire, tabulate the responses, and set appropriate targets to track progress over time. FFP’s policies and procedures, in conjunction with USAID’s official guidance, cover the types of evaluations that can be performed, methodologies that can be employed, ways to plan evaluations, the means to disseminate studies, and the independence of the staff performing the evaluations.

USAID’s new central evaluation office collaborated with State’s Office of the Director of Foreign Assistance (State/F) to develop evaluation guidelines and standards that were placed on the United States Foreign Assistance Web site in March 2009. These standards and guidance elaborate on the official USAID and FFP guidance, and they specifically address some elements that are not directly addressed in the official guidance, most notably the standards for quality. Collectively, the FFP- specific, USAID-wide, and Foreign Assistance policies and procedures are generally consistent with the AEA principles.

FFP Requires External Evaluations to Help Ensure Independence

To help ensure independence as outlined in the AEA principle, FFP requires that each multiyear assistance program’s final evaluation be performed by external evaluators. It also relies on external entities to perform cross-cutting evaluations using the Food for Peace Act and other funding sources.

Independence in the context of final evaluations for FFP multiyear assistance programs means that the key evaluation experts, the team leader, and sectoral leads, should be external consultants hired by the project. Our review of the 16 final evaluations that we found on USAID’s Development Experience Clearinghouse (DEC) Web site determined that external consultants were generally listed as the prime authors. In some instances, the external evaluators were affiliated with a particular consulting firm and, in other instances, several external evaluators formed a team to perform the final evaluation.

In addition, the cross-cutting evaluations and other FFP-funded studies are to be conducted by FANTA-2 and external consultants. FANTA-2 is part of AED, a nonprofit organization that cites integrity as one of its core values. Our review of selected task orders, signed and made available by FANTA-2, found them in line with the AEA principle for independence. For example, not only did its task order for a subrecipient of a new 3-year cross-cutting evaluation describe the project’s phases, activities, and methodologies, but it also clearly specified that the subrecipient’s staff would develop preliminary conclusions based on the collected data, draft interim reports, and submit a final report to USAID.

USAID’s Actions Were Partially Consistent with the AEA Principles for Scope and Coverage, Dissemination of Results, Professional Competence, and Resources

USAID Requires Few Cross- Cutting Evaluations

programs and policies, making it integral to planning, developing, and managing government programs at all stages of their development. should be subject to evaluation, which is a responsible and necessary part of good management. submit annual questionnaires that respond to questions about FFP’s food security indicators; and submit midterm and final project evaluations.

In addition, USAID project officers must review the evaluation plans and final reports. Finally, the final evaluation must include questionnaires that are comparable to the baseline surveys, along with information about sample design, universe, methods of analysis, and sampling frame.

During our fieldwork in Bangladesh and Haiti, we saw several examples of FFP’s implementing partners following these directives. For example, we visited a project in Bangladesh that had collected baseline data on a wide range of indicators that measured aspects of land ownership, height and weight-for-age measures, maternal and child health practices, access to water and sanitation, household food security, and disaster preparedness. In addition, we visited a project in Haiti that had produced a quarterly report including targets for more than 50 measures, along with data on whether or not those targets had been achieved, and reasons why some targets had not been met.

FFP’s required monitoring and evaluation of the multiyear programs focus on whether the programs are achieving their expected outcomes—but they do not assess program impacts, or the effectiveness and efficiency, of particular approaches. To achieve this, FFP would need to conduct periodic, cross-cutting evaluations that compare what actually happened with what would have happened in the absence of the program or activity. While FFP has commissioned some of these studies, according to USAID officials, it has no requirements to do so and has conducted only a relatively small number over the last 5 years. In contrast, FFP required every one of more than 100 multiyear projects concluded over the last 5 years to be evaluated for outcomes.

Implementing Partner Evaluation Results Are Not Routinely and Widely Disseminated

Although USAID policy requires final evaluations of its nonemergency food aid programs to be posted on a special Web site known as DEC, the implementing partners responsible for these evaluations often do not adhere to this requirement. The DEC Web site is accessible to the general public and contains both simple and more advanced search engines. For example, a user interested in Food for Peace nonemergency programs could perform a search for “agriculture,” “maternal and child health,” or “food security” final evaluations conducted in recent years. A user could also search for final evaluations sponsored by FFP. However, FFP officials told us that the MYAP implementing partners often do not submit their final evaluations to DEC, even though this is a program requirement. According to these officials, there is no penalty if the implementing partners do not submit their reports to DEC. We were unable to find in DEC a final evaluation of an integrated food security program that FFP staff in Bangladesh provided to us in hard copy. In addition, more than 100 FFP-sponsored final evaluations of multiyear programs should have been conducted from 2004 to 2009, according to FFP staff, but our search in DEC found only 16.

Professional Competence Varies among the Different Groups of Staff Who Monitor and Evaluate FFP Programs

Although the entities that USAID has selected to perform cross-cutting evaluations have staff with appropriate professional training and experience, questions have been raised about the technical skills of some staff that perform monitoring and evaluation for the implementing partners. As FFP does not have reliable data on the number of its field staff that perform monitoring and evaluation functions, and the specific skills and competencies of those staff, it is not possible to determine their overall competence. In addition, as FFP’s ability to hire monitoring and evaluation specialists in Washington, D.C., is limited, and as USAID has only recently reestablished its central evaluation unit, there has been an absence of skilled evaluators in headquarters that could assist FFP’s officers in the field and implementing partners’ staff. recognize the multidisciplinary nature of evaluation and assure that the diversity of disciplines is appropriately represented in both internal and independent consulting teams.

FFP’s Partners Have Training and Experience Consistent with the AEA Principles The partner organizations that FFP relies on to provide technical assistance and support and to conduct cross-cutting evaluations have training and expertise that are consistent with the AEA principles for professional competence. The AED FANTA-2 project, with which USAID has a cooperative agreement for various types of technical assistance, employs staff experienced in monitoring and evaluation, with backgrounds in economics, sociology, nutrition, anthropology, public health, and epidemiology. In addition, FANTA-2 has partnered with universities and research institutes, such as Tufts University and the International Food Policy Research Institute, for particular evaluations and other research projects.

Concerns Have Been Raised about Certain Aspects of Implementing Partners’ Monitoring and Evaluation Skills FFP’s implementing partners’ staff perform important monitoring and evaluation functions, such as gathering data on performance indicators and tracking these indicators. Yet FFP officials in Washington, D.C., and in the field, as well as some external evaluators, have expressed concerns about the competency of these staff to perform certain aspects of monitoring and evaluation, as follows: Staff can lack key evaluation skills. Implementing partners’ staff can lack key evaluation skills, according to an AED employee who provided training and support to FFP’s implementing partners and several external evaluators. According to the AED employee, implementing partners’ staff could execute research designs adequately but often were not capable of designing the research themselves, or of taking responsibility for key aspects of data collection, such as developing surveys or selecting statistical samples. FFP officials stated that they recognize the need to build capacity among the implementing partners, and are providing monitoring and evaluation training in the field for these FFP and implementing partner staff.

Difficulties in hiring and retaining qualified staff. Adequacy of staffing levels and staff turnover have been cited as a concern in some countries. For example, the implementing partners in Haiti told us that it was difficult to attract and retain technically qualified staff. They stated that Haiti has only a limited supply of people with the appropriate skills and training for monitoring and evaluation. To address the shortage, one implementing partner recently hired a new monitoring and evaluation program officer from an African country who had appropriate methodological skills but did not speak either French or Creole, Haiti’s official languages, at the time of his arrival in-country.

FFP Lacks Data on the Monitoring and Evaluation Competencies of Its Field Staff FFP does not have the data necessary to assure that it has enough staff with the right skills. According to FFP officials, USAID does not collect an inventory of its staff’s skills and abilities, which would help the agency to determine whether it has staff with sufficient training and skills in monitoring and evaluation. Furthermore, neither USAID nor FFP have data that can indicate which staff spend the majority of their time on monitoring and evaluation, or how much staff time is spent on various monitoring and evaluation activities, such as checking on warehouses, advising implementing partners on performance indicators, or participating in evaluation designs. While FFP attempted to gather some data for us on its field staff who perform monitoring and evaluation, these data were not reliable.

In Haiti, for example, FFP officials initially told us that nine staff acted as field monitors. However, we found that this number included staff members who did not spend much of their time involved in routine monitoring and evaluation, such as the FFP officer, his deputy, and two secretaries. Of the FFP staff in country, only four appeared to spend substantial parts of their time on specific monitoring activities, such as performing routine checks of warehouses or health clinics and giving implementing partners advice on indicators.

One reason that it is difficult for FFP to determine which staff spend the majority of their time on monitoring is that definitions of this function vary. For example, while a recent FFP bulletin defined monitoring as the regular tracking of program implementation, FFP guidance to field staff defined monitoring more broadly, to include activities such as reviewing contracts and ensuring that financial audits are performed. If the broader definition of monitoring is used, more staff members can be classified as performing this function, but then it is not possible to determine which of those staff are spending the majority of their time on monitoring program inputs and program implementation.

FFP Lacks Staff in Headquarters Who Are Skilled in Monitoring and Evaluation AEA has recommended that to support evaluations agencies either establish centers of evaluation or appoint evaluation coordinators for individual programs that conduct their own evaluation. According to FFP officials, FFP has been unable to hire monitoring and evaluation staff at headquarters to assist and support its officers in the field. FFP officials attributed this to restrictions in the Food for Peace Act, which precludes them from hiring any staff in Washington, D.C., to monitor and evaluate nonemergency food aid programs. Moreover, as USAID has only recently reestablished its central evaluation unit, FFP has not been able to seek advice and guidance from professional, experienced evaluators at headquarters, or to ask them to perform evaluations of FFP’s programs.

Greater Resources Are Now Available for Monitoring and Evaluation, but Their Sufficiency Is Difficult to Determine Absent a Needs Assessment

The Food for Peace Act provides FFP with a stable source of new funding of up to $22 million annually for each of the fiscal years 2009 through 2012 to improve, monitor, and evaluate the effectiveness and efficiency of nonemergency food aid programs. In addition, FFP has access to other funding streams, such as DA funds, which have recently increased. As a result, FFP officials told us that they currently have more resources to devote to monitoring and evaluation than in the past.

Principle: Sufficient resources should be made available for evaluation, including stable annually recurring sources of funds and special one-time funds for evaluation projects of interest to executive branch and congressional policymakers.

Although availability of the newly authorized funding for monitoring and evaluation as well as funding from other sources is generally consistent with the AEA principle on resources, it is not possible to determine if funding is sufficient because FFP has not systematically determined its research needs, prioritized those needs, or estimated the costs associated with them. In the course of our audit work, we identified several areas where the current resource levels might not be sufficient. For example, experts we interviewed provided examples of potential evaluations or studies that could help improve FFP’s programs. However, according to USAID officials, FFP has not undertaken the formal exercise of compiling a list of these studies, or prioritizing its needs for them. In addition, FANTA-2 staff who review many of the implementing partners’ multiyear plans for evaluation raised concerns that FFP does not sufficiently ensure that these evaluations are adequately funded. However, FFP has not reviewed or estimated the amounts its implementing partners are spending on monitoring and evaluating their multiyear projects because, according to FFP, that would be burdensome as it would require a review of each project’s contract. Without such an assessment, it is not possible to determine whether sufficient resources are being spent on the required final evaluations.

USAID’s Actions Were Not Consistent with the AEA Principles for Monitoring and Evaluation Plans

According to FFP officials, FFP plans some of its monitoring and evaluation activities, but lacks an overall plan that integrates its monitoring and evaluation with program management. FFP officials reported that they make decisions about monitoring and evaluation based on discussion and a thorough consideration of options but have not undertaken the rigorous planning process that is recommended by AEA. Therefore, FFP does not have an integrated plan for all of its oversight activities to facilitate strategic planning, budgeting, and program management, and to indicate how it will use monitoring and evaluation results to improve program management. Moreover, FFP does not have a mechanism to link resources to needs based on risk, according to USAID officials. As a result, when FFP allocates resources for monitoring and evaluation, it does not take into account factors such as the country context, vulnerability to fraud, waste, and abuse, program efficiency, and critical issues. As we noted earlier, we cannot determine whether FFP has sufficient resources because this type of planning has not been conducted, which means that FFP’s monitoring and evaluation needs have not been prioritized and compared against available resources.

A senior FFP official told us that he could see the potential value of such a plan but wondered whether FFP—which has a staff of about 60 people and which he characterized as being small in relation to other USAID units—could support such an effort. However, FFP appears to have the resources for such a planning effort, as the Food for Peace Act has provided it with up to $88 million to improve, monitor, and evaluate the effectiveness and efficiency of nonemergency food aid programs over a 4- year period, and has access to other resources that can be used for monitoring and evaluation, such as DA funds.

Conclusions

Nonemergency food aid programs are an essential component of the U.S. strategy to reduce global food insecurity, and Congress has authorized new funding for USAID to improve its oversight of these programs. In response, USAID’s FFP has initiated actions over the past year to strengthen its monitoring and evaluation of food assistance. However, weak planning could impede these efforts, including the upgrade of FFP’s information technology system. In addition, USAID has yet to determine how it intends to assure stable resources, within existing authorities, to fully implement all the actions planned by FFP to improve monitoring and evaluation of nonemergency food aid. As a result, planned actions that are essential to effective program management, such as the expansion of a computerized monitoring system for food aid programs and cross-cutting evaluations, could be jeopardized. Finally, FFP lacks an integrated and comprehensive monitoring and evaluation plan that links resources to needs based on risk. In the absence of such a plan, it is difficult to determine whether USAID is allocating resources for monitoring and evaluation to its priority needs, such as program efficiency, critical issues, and programs vulnerable to fraud, waste, and abuse.

Recommendations for Executive Action

To facilitate the implementation of ongoing efforts to improve monitoring and evaluation and ensure effective program management of nonemergency food aid, we recommend that the Administrator of USAID take the following two actions: Develop a concept of operations document to help reduce the risks associated with upgrading FFP’s information technology system. Such a document should adhere to industry best practices and include key elements such as major system components, interfaces to external systems, and performance characteristics. It would also describe system characteristics for a proposed system from a user’s point of view and include high-level descriptions of information systems, their interrelationships, and information flows.

Develop an integrated monitoring and evaluation plan to ensure that the results of its oversight activities are used for effective management of nonemergency food aid programs. Such a plan should, among other things, (1) link monitoring and evaluation to key USAID and FFP goals; (2) establish a systematic process for determining appropriate budget levels and staff resources for monitoring and evaluation based on an analysis of the nature, scope, and size of the programs, and the unique conditions presented, in the recipient countries; (3) examine all available options for funding monitoring and evaluation at headquarters and in recipient countries; and (4) establish time frames for implementing and evaluating the plan.

Agency Comments and Our Evaluation

USAID provided written comments on a draft of this report. We have reprinted these comments in appendix IV, along with our responses. Additionally, USAID provided technical comments on a draft of this report, which we have addressed as appropriate.

USAID concurred with our first recommendation to develop a concept of operations document to help reduce the risks associated with upgrading FFP’s information technology system. USAID disagreed with the second recommendation, asserting that we did not adequately take into account documents that detail its planning processes. We considered all the documents that USAID cited and found that they did not constitute plans, but only provided guidance for planning. Furthermore, the weaknesses we identified in FFP’s monitoring and evaluation efforts, including in the areas of resources and professional competence, demonstrate the need for a comprehensive monitoring and evaluation plan.

USAID acknowledged that it does not have an integrated monitoring and evaluation plan, but said that it does not see the benefits of developing it since the elements of such a plan exist in some form already. USAID noted that we did not mention FFP’s Strategic Plan for 2006-2010 and a related document, which link monitoring and evaluation to agency goals and lay out performance indicators. We considered the strategic plan irrelevant to our analysis because it was prepared in 2005, prior to the passage of the 2008 Food for Peace Act, and covers only some aspects of monitoring. USAID also stated that its planned studies form part of an agency agenda to inform program management. We reported on FFP’s planned studies but noted that, because funding for these studies is not assured due to legal restrictions, USAID may need to find alternative funding sources. Moreover, USAID officials reported that they did not use almost $9 million of last year’s available funds authorized by the 2008 Food for Peace Act for oversight of nonemergency food aid. We consider these to be compelling reasons for USAID to develop an integrated monitoring and evaluation plan.

In terms of staffing, FFP officials said that their first priority is to place staff in all the countries where they have nonemergency food aid programs. FFP acknowledged that it does not have data on field staff that perform monitoring and evaluation functions, but said that such data would be costly to collect, and that we had not made the case for collecting them. However, we found that FFP currently does not know what percentage of time its staff spend on monitoring and evaluation and what specific monitoring and evaluation functions they perform. We believe that because FFP had a substantial new funding authority for oversight activities, it should invest in a survey of staff skills and an assessment of monitoring and evaluation needs, which would provide FFP with essential data for future staffing decisions. We believe that this basis for our recommendation for an integrated monitoring and evaluation plan remains valid.

We are sending copies of this report to interested Members of Congress and the Administrator of USAID. The report is also available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-9601 or melitot@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

In passing the Food for Peace Act in 2008, Congress required the Comptroller General of the United States to review the U.S. Agency for International Development’s (USAID) Report to Congress on the agency’s efforts to strengthen the monitoring and evaluation of its nonemergency food aid programs and provide recommendations for improvement. To address this congressional mandate, we reviewed (1) USAID’s plans and actions to improve its monitoring and evaluation of nonemergency food aid programs provided under the Food for Peace Act and (2) the extent to which USAID has integrated its monitoring and evaluation of nonemergency food aid with program management, particularly with regard to established principles. In this review, we focused on oversight practices of USAID’s Office of Food for Peace (FFP), not of the agency as a whole.

To address USAID’s plans to improve the monitoring and evaluation of nonemergency food aid programs using the funds provided under the Food for Peace Act, we reviewed the list of activities that the agency reported to Congress in December 2008. This included plans to make improvements in the number of field staff monitoring food aid programs, computerized systems for monitoring food aid programs, the market analysis required before food aid programs are approved in recipient countries, cross-cutting evaluations of nonemergency food aid projects, FFP’s information technology system, and the Famine Early Warning Systems Network (or FEWS NET).

We interviewed knowledgeable USAID officials and partners responsible for implementing these plans and reviewed relevant documents, including contracts and plans. We also examined FFP data on the allocations of the Food for Peace Act funding for oversight of nonemergency food aid in fiscal year 2009. Based on interviews, and our analysis of the available documents, we determined that the data were sufficiently reliable to indicate the relative amounts allocated to each of the oversight activities, and to demonstrate that almost $9 million were not used in fiscal year 2009.

In addition, we conducted fieldwork in Bangladesh and Haiti. We selected these two countries because they have the largest Food for Peace nonemergency food aid programs, but they differ in their approaches to monitoring and evaluation. Haiti has more staff working on these programs than does Bangladesh and uses the Layers computerized monitoring system, which Bangladesh does not. Haiti’s staff spend relatively more time monitoring programs directly, while FFP in Bangladesh relies more on contract officers and implementing partners to perform its monitoring and evaluation. In addition, Haiti is in the Caribbean while Bangladesh is in South Asia, which provided a regional contrast. While we could not generalize from these two countries to all the countries that have Food for Peace programs, we gained useful insights into monitoring and evaluation from our fieldwork.

To consider the extent to which USAID’s monitoring and evaluation of nonemergency food aid are aligned with program management, we selected the American Evaluation Association’s (AEA) principles for integrating evaluation into program management. We selected these principles as criteria because they are consistent with the goal set by the Secretary of State, who has ultimate responsibility for USAID, to ensure evidence-based decisions in development programs.

In this review, we combined the AEA standards for scope and for coverage because we found overlap between them. In addition, we did not assess one of the AEA standards on methods because that would have involved a review of studies that was beyond the scope of this report, and because that principle was less specific than those for the other elements, essentially noting that a mix of methods should be used, but that the actual methods used should depend upon the topic to be evaluated.

The AEA principles differ in length and detail, with some including several parts and clauses, while others are more succinct. For the more detailed principles, we identified the sentences that summarized them at the highest level and used those as our operating criteria. For the sake of transparency, we published the exact principles we used in our report next to our discussion of them.

To gather evidence to apply against the principles, we interviewed cognizant FFP officials in Washington, D.C., and in Bangladesh and Haiti. We also interviewed the Chief of USAID’s Evaluation Office, senior staff at the Food and Nutrition Technical Assistance (FANTA-2) project of the Academy for Educational Development (AED), with which USAID has ongoing cooperative agreement, as well as experts knowledgeable about USAID’s evaluation and monitoring. We reviewed USAID and FFP policy guidance and bulletins that discuss and describe evaluation. We reviewed the evaluations that we found on USAID’s Development Experience Clearinghouse (DEC) Web site and that were given to us by USAID and its implementing partners. We also examined related monitoring and evaluation documents, such as performance plans and annual project reports, and documentation of the training USAID provides on monitoring and evaluation.

We reviewed this evidence and compared it against each of the AEA principles to make a general, high-level determination on the broad degree to which FFP was consistent or inconsistent with them. In the course of this review, we recognized that FFP has its own guidance and practices, but that it follows the overall guidance and practices set by USAID for the agency as a while, and considered each source as appropriate.

In addition to the AEA principles, we also considered GAO standards and prior GAO work, particularly on workforce planning; USAID guidance; and guidelines that the Institute of Electrical and Electronics Engineers (IEEE) has established for information technology.

As USAID performs a wide range of activities to monitor and evaluate its Food for Peace program, we defined evaluation broadly for the purposes of this report. This definition includes evaluating inputs and implementation, outcomes, and impacts. We also considered other research studies that FFP had conducted to help manage its programs, such as prospective studies on the impacts of providing particular food commodities to particular markets, and analyses of individual countries’ food strategies. In addition, we included the selection and tracking of performance indicators in our review because FFP and USAID define monitoring to include the tracking of these indicators during the life cycle of their programs, and use them in the final evaluations.

We conducted this performance audit from March 2009 to September 2009, in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: USAID Report to Congress

Appendix III: Examples of Activities Supported with Nonemergency Food Aid

Figure 5 provides examples of activities we observed in Bangladesh and Haiti that are illustrative of the various activities supported with nonemergency food aid.

Appendix IV: Comments from USAID

The following are GAO’s comments on the U.S. Agency for International Development letter dated September 17, 2009.

GAO Comments

1. We reviewed the eight documents listed and acknowledged several of them in our report’s section on policies and procedures. However, none of these documents constitutes a plan. Instead, they are policy or technical documents that describe how USAID and its awardees should plan for aspects of monitoring and evaluation. 2. We considered FFP’s Strategic Plan for 2006-2010 but determined it to be irrelevant to our analysis because it was prepared in 2005, prior to the passage of the 2008 Food for Peace Act, and covers only some aspects of monitoring. The Performance Management Plan is related to the strategic plan and provides a set of indicators to measure progress. Neither of these documents discusses funding, resources, staffing needs, or how results would be used to improve program management. 3. We reported on the planned studies earlier in this report. However, USAID provided us not with a plan, but rather with a list of selected studies, their implementers, costs, and a high-level description. In the list, USAID did not discuss alternatives, outline priorities, or explain how the selected studies would address FFP’s most pressing needs or be used for program management. As we noted, several studies do not have assured funding due to the legal restrictions on FFP’s funding, which makes the need for integrated planning even more critical. 4. We noted that FFP allocated the new staff positions in order to ensure coverage in all countries where it has programs. While this is one factor to consider in workforce planning, FFP did not consider other important factors, such as existing staff skills or country-specific needs for monitoring and evaluation because it did not collect data on them. Lacking a systematic process to collect such data, FFP is unable to determine if it has adequate staff with sufficient training and skills in monitoring and evaluation. 5. FFP officials reported that they had not used almost $9 million of the $22 million available in fiscal year 2009. Better planning could ensure that the available funds are more fully used in future years for activities such as surveying FFP staff on their monitoring and evaluation skills and conducting an assessment of FFP monitoring and evaluation needs.

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Phillip Thomas, Martin De Alteriis, and Michael LaForge, Assistant Directors; and Sada Aksartova; Carol Bray; Ming Chen; Debbie Chung; Lynn Cothern; Mark Dowling; Brian Egger; Ian Ferguson; Etana Finkler; Kendall Helm; Joy Labez; Jeff Miller; Yvonne Moss; Ulyana Panchishin; Lisa Reijula; Julia Ann Roberts; Barbara Shields; Stephanie Shipman; and George Taylor made key contributions to this report.

Related GAO Products

International Food Assistance: Key Issues for Congressional Oversight. GAO-09-977SP. Washington, D.C.: September 30, 2009.

International Food Assistance: Local and Regional Procurement Provides Opportunities to Enhance U.S. Food Aid, but Challenges May Constrain Its Implementation. GAO-09-757T. Washington, D.C.: June 4, 2009.

International Food Assistance: Local and Regional Procurement Can Enhance the Efficiency of U.S. Food Aid, but Challenges May Constrain Its Implementation. GAO-09-570. Washington, D.C.: May 29, 2009.

International Food Security: Insufficient Efforts by Host Governments and Donors Threaten Progress to Halve Hunger in Sub-Saharan Africa by 2015. GAO-08-680. Washington, D.C.: May 29, 2008.

Foreign Assistance: Various Challenges Limit the Efficiency and Effectiveness of U.S. Food Aid. GAO-07-905T. Washington, D.C.: May 24, 2007.

Foreign Assistance: Various Challenges Impede the Efficiency and Effectiveness of U.S. Food Aid. GAO-07-560. Washington, D.C.: April 13, 2007.

Foreign Assistance: U.S. Agencies Face Challenges to Improving the Efficiency and Effectiveness of Food Aid. GAO-07-616T. Washington, D.C.: March 21, 2007.

Darfur Crisis: Progress in Aid and Peace Monitoring Threatened by Ongoing Violence and Operational Challenges. GAO-07-9. Washington, D.C.: November 9, 2006.

Maritime Security Fleet: Many Factors Determine Impact of Potential Limits of Food Aid Shipments. GAO-04-1065. Washington, D.C.: September 13, 2004.

United Nations: Observations on the Oil for Food Program and Iraq’s Food Security. GAO-04-880T. Washington, D.C.: June 16, 2004.

Foreign Assistance: Lack of Strategic Focus and Obstacles to Agricultural Recovery Threaten Afghanistan’s Stability. GAO-03-607. Washington, D.C.: June 30, 2003.

Foreign Assistance: Sustained Efforts Needed to Help Southern Africa Recover from Food Crisis. GAO-03-644. Washington, D.C.: June 25, 2003.

Food Aid: Experience of U.S. Programs Suggest Opportunities for Improvement. GAO-02-801T. Washington, D.C.: June 4, 2002.

Foreign Assistance: Global Food for Education Initiative Faces Challenges for Successful Implementation. GAO-02-328. Washington, D.C.: Feb. 28, 2002.

Foreign Assistance: U.S. Food Aid Program to Russia Had Weak Internal Controls. GAO/NSIAD/AIMD-00-329. Washington, D.C.: September 29, 2000.

Foreign Assistance: U.S. Bilateral Food Assistance to North Korea Had Mixed Results. GAO/NSIAD-00-175. Washington, D.C.: June 15, 2000.

Foreign Assistance: Donation of U.S. Planting Seed to Russia in 1999 Had Weaknesses. GAO/NSIAD-00-91. Washington, D.C.: March 9, 2000.

Foreign Assistance: North Korea Restricts Food Aid Monitoring. GAO/NSIAD-00-35. Washington, D.C.: October 8, 1999.

Food Security: Factors That Could Affect Progress toward Meeting World Food Summit Goals. GAO/NSIAD-99-15. Washington, D.C.: March 22, 1999.

Food Security: Preparations for the 1996 World Food Summit. GAO/NSIAD-97-44. Washington, D.C.: November 7, 1996.