International Food Assistance: USAID Is Taking Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid, but Weaknesses in Planning Could Impede Efforts
Published: 2009-09-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In passing the Food for Peace Act in 2008, Congress authorized up to $22 million annually for fiscal years 2009 to 2012 to the U.S. Agency for International Development (USAID) to improve, monitor, and evaluate the effectiveness and efficiency of nonemergency food aid programs. Congress also required USAID to report on its oversight of these programs and the Comptroller General to review and report to Congress on USAID's report. Through analysis of agency documents; interviews with agency officials, experts, and partners; and visits to Bangladesh and Haiti, this mandated report reviews (1) USAID's plans and actions to improve its monitoring and evaluation of nonemergency food aid programs and (2) the extent to which USAID has integrated its monitoring and evaluation of nonemergency food aid with program management.
section_title: What GAO Found
paragraphs: USAID's actions to improve its monitoring and evaluation of these programs could be hindered by weak planning. Monitoring is essential to ensuring that USAID's nonemergency food aid programs in developing countries are implemented as intended, and evaluation helps to assure that these programs achieve their goal of reducing global food insecurity. First, with funding from the Food for Peace Act, USAID's Office of Food for Peace (FFP) plans to increase the number of field staff responsible for the monitoring of nonemergency food aid programs, has provided funding for the Famine Early Warning Systems Network, and has initiated an upgrade of its information technology system. However, FFP's plans for the information technology upgrade lack a concept of operations document, which describes system characteristics for a proposed system from a user's point of view and includes high-level descriptions of information systems, their interrelationships, and information flows. Second, with funding from other sources, USAID plans to carry out additional actions in an effort to improve its oversight of food aid, including the expansion of a computerized system for monitoring the implementation and management of nonemergency food aid programs. However, USAID has not determined a stable source of funding for these initiatives beyond the first year of operations due to legal restrictions that preclude the agency from using the newly authorized funding for grants and cooperative agreements. USAID's monitoring and evaluation of its nonemergency food aid programs are consistent to varying degrees with some of the principles established by the American Evaluation Association's Task Force on Evaluation Policy to integrate evaluation into program management. GAO found that (1) FFP's actions were generally consistent with the principles for policies and procedures and for independence. For example, FFP has issued policies and procedures for monitoring and evaluating food aid programs and generally uses external evaluators to assess its multiyear assistance programs. (2) FFP's actions were partially consistent with the principles for scope and coverage, dissemination of results, professional competence, and resources. For example, FFP relies on a range of staff to perform its monitoring and evaluation, but does not have reliable data on the numbers of field staff who have competencies in monitoring and evaluation, or their specific skills. (3) FFP's actions were not consistent with the principles for monitoring and evaluation plans. While FFP plans some of its monitoring and evaluation activities--such as final evaluations for multiyear assistance programs--it lacks an integrated plan to ensure that monitoring and evaluation results will be used to improve program management.

section_title: Letter
paragraphs: The number of undernourished people worldwide has been growing and  now exceeds 1 billion, according to the estimates of the United Nations  (UN) Food and Agriculture Organization (FAO). This calls into question  the feasibility of attaining the 1996 World Food Summit target of reducing  the number of hungry people by half to no more than 420 million by 2015.  The United States is the world’s largest donor of food aid, and the U.S.  Agency for International Development (USAID) Office for Food for Peace  (FFP) administers the largest nonemergency food aid program funded  under the Food for Peace Act. FFP has adopted as its strategic objective  the reduction of food insecurity in vulnerable populations. Specifically,  FFP’s Strategic Plan for 2006-2010 identifies two intermediate results:  (1) enhancing FFP’s global leadership in reducing global food insecurity  and (2) increasing FFP’s program impact in the field.
Monitoring and evaluation of nonemergency food aid—one of the largest  sources of U.S. funding focused on food security—are critical oversight  and program management tools that would help to ensure that FFP meets  its strategic objective. The size and scope of FFP’s activities are  substantial—in fiscal year 2008, FFP supported 20 grantees that  implemented 92 activities in 28 countries—but a number of recent reviews  have found that monitoring and evaluation of these activities have been  insufficient.  In 2007, we found that the USAID Inspector General had  previously reported that monitoring had not been regular and systematic  and that, in some cases, intended recipients had not received food aid or  the number of recipients could not be verified. In April 2007, we reported  that USAID had only 23 Title II-funded staff assigned to missions and  regional offices in just 10 countries to monitor programs that cost about  $1.7 billion in 55 countries in the previous fiscal year. Our audit work also  indicated that monitoring had been insufficient due to various factors  including limited staff, competing priorities, and legal restrictions on the  use of food aid resources. To ensure the effective use of food aid, we  recommended that USAID improve its monitoring of food aid programs to  strengthen proper management and implementation of these programs.  USAID agreed, noting that monitoring of all its grant programs is required  and that its implementing partners are legally required to monitor and  report their findings to USAID. Like monitoring, evaluation is crucial to  ensuring that best practices and lessons learned are considered in the  management and implementation of existing programs and in designing  new ones. However, for many years, USAID did not place a great deal of  importance or invest many resources in evaluation. In a June 2009 speech,  the Secretary of State, whose department coordinates U.S. development  assistance, recognized that “…too often, program evaluation is treated as  an afterthought, and sometimes is overlooked entirely.” To rectify this  situation, the Secretary announced that the Department of State (State)  had for the first time developed a policy for integrating evaluation into its  work worldwide.
In passing the Food for Peace Act in 2008, Congress authorized up to $22  million annually in funding for fiscal years 2009 through 2012 to USAID for  program oversight to improve, monitor, and evaluate the effectiveness and  efficiency of nonemergency food aid programs—and, consistent with our  2007 recommendation, required the USAID Administrator to submit an  implementation report on the agency’s efforts in these areas. Congress  also required the Comptroller General of the United States to review  USAID’s report and provide recommendations for improvement. To  address this congressional mandate, we reviewed (1) USAID’s plans and  actions to improve its monitoring and evaluation of nonemergency food  aid programs provided under the Food for Peace Act and (2) the extent to  which USAID has integrated its monitoring and evaluation of  nonemergency food aid with program management, particularly with  regard to established principles. In this review, we focused on oversight  practices of FFP, not of USAID as a whole.
To address these objectives, we examined USAID’s December 2008 report  to Congress, which outlines FFP’s plans to improve its monitoring and  evaluation of nonemergency food aid programs, and actions FFP has taken  to date. We interviewed USAID officials, including FFP field staff  responsible for monitoring the implementation of nonemergency food aid  programs, and implementing partners such as nongovernmental  organizations (NGO) and the World Food Program (WFP). We considered  established principles for monitoring and evaluation—especially the  “Roadmap” principles of the American Evaluation Association (AEA),  which have been developed to integrate monitoring and evaluation with  program management; GAO standards and prior GAO work; USAID  guidance; and guidelines that the Institute of Electrical and Electronics  Engineers (IEEE) has established for information technology. We  conducted fieldwork in Bangladesh and Haiti, where we met with U.S.  mission officials, host government officials, bilateral and multilateral  donor representatives, and NGO representatives. In Bangladesh and Haiti,  we also visited various WFP and USAID project sites supported by  nonemergency food aid and food aid logistics facilities.
We conducted this performance audit from March 2009 to September 2009,  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. (Appendix I provides a detailed  discussion of our objectives, scope, and methodology.)
subsections: section_title: Background
subsections: section_title: USAID Plans to Administer Larger Nonemergency Food Aid Programs in Fewer Priority Countries
paragraphs: FFP is shifting to larger programs in fewer countries in order to leverage  nonemergency food aid resources more effectively. FFP’s Strategic Plan  for 2006-2010 concentrates on a single objective of reducing food  insecurity in vulnerable populations with an aim to fund programs that can  be modified when circumstances change. In fiscal year 2008, under the  Food for Peace Act, the United States provided $354 million in  nonemergency food aid and FFP administered nonemergency food aid  programs in 28 countries. Beginning in fiscal year 2010, FFP will limit its  nonemergency food aid programs to 20 priority countries. Of these, 16  are in sub-Saharan Africa (Burkina Faso, Burundi, Chad, the Democratic  Republic of Congo, Ethiopia, Liberia, Madagascar, Malawi, Mali,  Mauritania, Mozambique, Niger, Sierra Leone, Sudan, Uganda, and  Zambia), 2 are in Central and South Asia (Afghanistan and Bangladesh),  and 2 are in Latin America and the Caribbean (Guatemala and Haiti).  Figure 1 lists the countries that received nonemergency food aid in fiscal  year 2008 and the most recent available statistics on the number of  undernourished people and the prevalence of undernourishment in these  countries, respectively.
section_title: Food for Peace Nonemergency Food Aid Programs Are Implemented through Grants for Single-Year and Multiyear Assistance Programs
paragraphs: FFP provides food aid funding under the Food for Peace Act using two  primary funding mechanisms—Single-Year Assistance Programs (SYAP)  and Multiyear Assistance Programs (MYAP). SYAPs address emergency  needs and are used for direct distribution of food aid in emergency  contexts. MYAPs, long-term programs with duration of 3 to 5 years, are  used to implement development activities that target chronically food- insecure populations. In carrying out nonemergency programs through  its implementing partners, FFP uses a combination of Food for Peac e  commodities for direct distribution, local currency generated through   and cash resources to implement activities that target  monetization, chronic food insecurity.  (For illustrative examples of the types of  activities supported by nonemergency food aid, such as project activities  we observed in Bangladesh and Haiti, see app. III.)
section_title: Monitoring and Evaluation Are Essential to Effective Program Management
paragraphs: Monitoring and evaluation perform two separate but related functions.  Monitoring is the collection of data to determine whether programs are  being implemented as intended and the tracking of progress using  preselected indicators throughout the life of the program. Evaluation  consists of periodic or ad hoc studies to assess how and whether programs  achieved their expected goals. It can also consider what outcomes can be  attributed to the program, as well as program cost-effectiveness.  Evaluations can rely on a range of quantitative and qualitative measures as  well as preselected indicators, comprehensive research designs, and the  use of appropriate statistical analysis of the data. Figure 2 outlines a  framework based on our review of Food for Peace nonemergency  programs, as well as AEA and WFP guidance that illustrates how  monitoring and evaluation could be integrated with program  management of nonemergency food aid.
section_title: Program Management
paragraphs: Determine if necessary inp uts (equipment,  supplies, and personnel) are in place.     – Ensure that food is properly stored.   – Check to see if health clinics are sufficiently   stocked for intended beneficiaries.
Determine if programs are achieving their  expected outputs by tracking performance  indicators.
Ensure that target number of beneficiaries are  receiving food and attending clinics.
Focus on the extent to  which program  objectives were  achieved and what  outcomes can be  directly attributed to  programs.
Mother and child health and nutrition programs may  consider how the  ration is affecting the  health and nutritional  status of beneficiaries. programs may focus  on how the activity is  addressing temporary  food insecurity and  developing a  sustainable  community asset.
section_title: FFP Relies on a Range of Staff to Perform Various Types of Monitoring and Evaluation Activities
paragraphs: FFP relies on a range of staff to perform its monitoring and evaluation—  including FFP officers in the field, implementing partners, contractors,  and FANTA-2 staff. According to FFP, field staff visit project sites to  ensure that programs are being implemented as expected. They also  approve and oversee the implementing partners’ selection of performance  indicators and data collection. In addition to regularly collecting  performance data, the implementing partners perform baseline studies and  midterm evaluations for their multiyear programs and hire external  evaluators to conduct the final evaluations for the multiyear programs. In  addition, FFP relies on appropriately credentialed consultants and  researchers for some cross-cutting evaluations, which, for example,  consider the agency’s exit strategies for nonemergency food aid programs  in Bolivia, Honduras, and Kenya.
Conducting evaluations can present many challenges, such as obtaining  reliable data and employing qualified evaluators. These challenges can be  compounded in developing countries, where local conditions may be  difficult and local capacity is often limited. For example, a recent review  of evaluations conducted by an international NGO cited numerous  challenges in-country including climate, poor economic status, civil unrest,  migration of beneficiaries, cultural biases, poor recordkeeping, and  turnover in government and NGO staff.
Table 1 summarizes the various types of monitoring and evaluation  activities that FFP performs. The table also indicates which staff have  primary responsibility for the activities, briefly describes the activities, and  provides examples of them.
section_title: Weaknesses in Planning Could Hinder USAID’s Actions to Improve Monitoring and Evaluation of Nonemergency Food Aid Programs
paragraphs: USAID has begun taking a series of actions to improve its monitoring and  evaluation of nonemergency food aid programs, as mandated by the Food  for Peace Act passed in July 2008. These efforts include (1) increasing the  number of field staff responsible for the monitoring of nonemergency food  aid programs; (2) expanding FEWS NET; (3) upgrading FFP’s information  technology system; (4) expanding the use of Layers, a computerized  monitoring system for quality assurance; (5) improving the quality of  evaluations, assessments, and targeting approaches; and (6) launching a  pilot project to improve market analysis in countries receiving U.S.  nonemergency food aid. Weaknesses in planning could hinder USAID’s  ability to effectively implement these actions, but it is too early to assess  impact because the agency’s efforts are still in progress. As of August 2009,  USAID estimated that, in fiscal year 2009, it will have spent about $13  million of the $22 million authorized for monitoring and evaluation of  nonemergency food aid under the Food for Peace Act. Figure 3  summarizes the allocation of the funds and the implementation timelines.
subsections: section_title: USAID Has Allocated New Funding to Increase Staffing Levels and Upgrade Information Systems but Has Not Developed Key Processes and Plans in These Areas
subsections: section_title: Staffing Levels
paragraphs: USAID has announced plans to increase the number of field staff  responsible for the monitoring of food aid programs, which would provide  minimal FFP presence in all priority countries. The agency plans to use  approximately $5 million of its new Food for Peace Act funding to add 21  full-time field staff whose responsibilities include the monitoring of  nonemergency food aid programs. According to USAID, 1 of the new  positions is located in USAID’s Southern Africa Regional Office in Pretoria  (South Africa) while the other 20 new positions are or will be located in  designated priority countries for nonemergency food aid programs  (including 19 in sub-Saharan Africa and 1 in Guatemala). As of August  2009, all 21 of the new positions had been established, of which 9 had been  filled. Figure 4 shows the location of the new positions, including those  filled, in FFP’s 20 priority countries for nonemergency food aid.
We found that USAID does not have a systematic process in place to  determine appropriate staffing levels for effective monitoring and  evaluation. According to USAID officials, creating new positions in field  missions where FFP has lacked staff was the primary consideration for  their allocation. Hence, 17 of the new positions are in countries where  USAID runs nonemergency food aid programs but has lacked FFP  program management staff to date, namely: Burkina Faso, Burundi,  Liberia, Madagascar, Malawi, Mali, Mauritania, Niger, Sierra Leone, and  Zambia. The remaining 4 new positions augment existing FFP program  management staff in the Democratic Republic of Congo, Guatemala,  Sudan, and the Southern Africa Regional Office in Pretoria.
section_title: Information Technology System
paragraphs: USAID plans to replace its current Food for Peace Information System  with a new Food for Peace Management System (FFPMS) that it expects  will address several of FFP’s current information technology issues,  including the collection, tracking, and interfacing of food aid program  data. According to USAID’s December 2008 report to Congress, the new  FFPMS is expected to involve collaboration and testing with users in each  business area in which the system is used. These areas include budgeting,  proposal, document management, commodity purchase, commodity  booking and shipping, tracking, monitoring and evaluation, reporting,  closeout, and interface with internal and external information systems. In  addition, USAID officials explained that, unlike FFPIS, FFPMS is intended  to be an all-inclusive system that will fully capture the movement of U.S.  food aid commodities, used for both emergency and nonemergency  purposes, from the initial transfer authorization to the commodity’s final  distribution site. FFPMS is also expected to (1) capture implementing  partners’ data on Food for Peace resources, including revenues from  monetization transactions as well as costs incurred; (2) allow FFP to  aggregate information for its implementing partners’ annual results  reports; and (3) have the capability to interface with USAID internal  systems—such as USAID’s core financial system, known as Phoenix,  which captures all USAID transactions—as well as with external agency  information systems with which FFP conducts business to allow the  collection and sharing of information among and between agencies. With  systematic and electronic collection of information, USAID anticipates  being able to monitor the relative efficiency of its nonemergency food aid  programs and adequately monitor the degree to which monetization  revenues cover costs.
USAID is in the initial stages of this planned information systems upgrade.  In June 2009, USAID awarded a contract for $637,000 to develop a  management strategy and framework for implementing FFPMS by the end  of fiscal year 2010. The contractor’s responsibilities, as defined in the  contract, are to revalidate assumptions made within the 2005 requirements  and architecture diagram; revalidate FFP’s technical environment; and  update the requirements, documents, and architectural design. The  contractor is also expected to make a recommendation to USAID by  December 2009 for a commercial off-the-shelf software package that  would best fit FFP’s needs.
A concept of operations document is a critical element in developing  information system requirements and testing activities. However, USAID  has not developed such a document for this initiative. This document is  normally one of the first documents produced during a disciplined  development effort because it describes system characteristics for a  proposed system from a user’s point of view. The key elements that  should be included in the document are major system components,  interfaces to external systems, and performance characteristics, such as  speed and volume. A concept of operations document defines how an  organization’s day-to-day activities will be carried out to meet mission  needs and includes high-level descriptions of information systems, their  interrelationships, and information flows. It also describes the operations  that must be performed, who must perform them, and where and how the  operations will be carried out.
section_title: Famine Early Warning Systems Network
paragraphs: To provide timely and rigorous early warning and vulnerability  information on emerging food security issues, USAID is making a  contribution to the FEWS NET fiscal year 2009 operational budget. FEWS  NET monitors and analyzes relevant data, using multiple technologies  such as satellites and field observations, to identify threats to food security  and their impacts on livelihoods and markets. Chemonics, the private firm  responsible for implementing FEWS NET along with several U.S.  government agencies, currently maintains 20 national and 3 regional field  offices in sub-Saharan Africa, Central America, Haiti, and Afghanistan.  Anticipating an increase in food insecurity in countries where there is no  FEWS NET presence, Chemonics plans to implement three remote  monitoring pilot projects—in Burundi, Pakistan, and Yemen—that are  expected to be completed by the end of fiscal year 2010.
In fiscal year 2009, USAID is providing $7.6 million of the newly authorized  Food for Peace Act funding to support FEWS NET. This funding is not  designated to support the remote monitoring pilot projects but is a  contribution to FEWS NET’s operational budget of $20 million, with the  remaining $12.4 million coming from the Development Assistance (DA),  the Office of Foreign Disaster Assistance (OFDA), and other sources.
section_title: Legal Restrictions on Usage of Food for Peace Act Funding Could Hamper USAID’s Plans to Expand FFP’s New Monitoring System and Improve Evaluations
subsections: section_title: Plans to Expand FFP’s Computerized Monitoring System for Quality Assurance
paragraphs: Legal restrictions on the allowable uses of Food for Peace Act funding  could hamper USAID’s plans to expand FFP’s new monitoring system for  quality assurance. USAID plans to expand the use of Layers—a  computerized system using personal data assistant (PDA) devices for  monitoring the implementation and management of nonemergency food  aid programs—from 3 pilot countries to 20 countries by the end of 2012,  through its multiyear cooperative agreement with AED. The purpose of  Layers is to help FFP staff responsible for the monitoring of food aid  programs to collect data to assess the quality of implementing partner  operations in the field. Layers can be used to monitor a wide range of  programs and activities, including the conditions of storage in commodity  warehouses and the quality of services delivered by FFP’s implementing  partners, such as food-for-work activities, provision of community health  services, and maternal and child health and nutrition. Prior to 2009, Layers  was piloted in Haiti, Ethiopia, and Madagascar. USAID estimates the one- time cost for the start-up to introduce Layers to a new country to be  approximately $144,000. It began rolling out the system in Guatemala,  Mali, and Uganda in 2009 and plans to do so in all the remaining priority  countries by the end of 2012 as their multiyear assistance programs are  approved. However, in May 2009, USAID’s General Counsel determined  that the Food for Peace Act precludes the agency from using the newly  authorized funding for nonemergency food aid monitoring and evaluation  for grants and cooperative agreements. Section 3012(f)(5) of the Food for  Peace Act authorizes USAID to enter into personal services contracts to  carry out monitoring and evaluation but does not refer to grants or  cooperative agreements. As a result, FFP cannot move forward with its  original plans to use Food for Peace Act funding to expand the  implementation of Layers and will instead need to use other sources of  funding that could be made available for this purpose. According to FFP  officials, the current plan is to use other funding streams, such as the DA  account, to fund the costs of the start-up of Layers and the first year of the  system’s operation.  However, these officials noted that due to competing  priorities for DA, it cannot be relied upon as a stable source of funding.
section_title: Plans to Improve the Quality of Evaluations
paragraphs: USAID plans to improve the quality of evaluations through a number of  ongoing and future studies. Using $5,000 of the newly authorized funding  for monitoring and evaluation, the agency has signed a contract with  Michigan State University for a study of monetization that is expected to  be available for release in December 2009. It has also commissioned  several cross-cutting evaluations and related studies that are expected to  improve the quality of assessments, targeting approaches, design, and  evaluation of nonemergency food aid programs. For example, through a  cooperative agreement with AED, USAID is funding the FANTA-2 project  to initiate the following multiyear studies in 2009:    a study of exit strategies in Bolivia, Honduras, and Kenya, which will  produce recommendations for effective ways to sustain program benefits  after U.S. nonemergency food aid programs have been phased out;    a study to identify key components of success in Preventing Malnutrition  in Children Under 2 Approach (PM2A) programs in Guatemala and  Burundi;    a study to improve the early warning and response capacity in FFP’s  multiyear assistance programs in Haiti, Niger, and Uganda;    an update of the 2002 Food Aid and Food Security Assessment, which will  evaluate progress made under FFP’s Strategic Plan for 2006-2010; and    a study testing the efficacy and safety of emergency food products (EFP)  that is co-funded by USAID’s Bureau for Global Health and FFP.
In addition to these cross-cutting evaluations and studies, FANTA-2  prepared four food security country frameworks—for Bangladesh,  Burkina Faso, Liberia, and Sierra Leone—in 2009. Food security country  frameworks are assessments that provide country-specific guidance to  partners developing new MYAP proposals and describe how Food for  Peace resources should be programmed and integrated with other U.S.  government funds by defining objectives, approaches, regions, and  institutional partnerships for effective use of U.S. government resources  dedicated to reducing food insecurity. FFP estimates the cost of FANTA- 2’s work on the cross-cutting evaluations, related studies, and food  security country frameworks in fiscal year 2009 to be $4.4 million.
To improve the quality of assessments and targeting approaches, FFP is  also initiating a new cooperative agreement to identify best practices that  will increase the efficiency and effectiveness of Food for Peace food aid  program management and capacity building. The aim is to promote the  broader adoption of best practices and collaboration among FFP’s  implementing partners. As of August 2009, FFP had issued a Request for  Applications seeking to identify potential partners’ interests, priorities,  and capacities and estimates the cost of the award at $3.5 million.
However, USAID has determined that the Food for Peace Act precludes  the agency from using the newly authorized funding for monitoring and  evaluation of nonemergency food aid for grants and cooperative  agreements. As a result, the completion of the multiyear cross-cutting  evaluations, as well as future food security country frameworks and “best  practices” assessments is not assured. FFP’s current plan is to fund these  studies from the DA account.
section_title: USAID Is Using Other Funding to Improve Assessments Conducted Prior to Approving U.S. Food Aid Programs in Recipient Countries but Faces Implementation Challenges
paragraphs: In August 2008, USAID launched a 3-year Bellmon Estimation for Title II  (BEST) pilot project intended to improve the market analysis required  before food aid programs are approved in recipient countries—commonly  referred to as the Bellmon analysis. The project is funded through  resources other than the newly authorized Food for Peace Act funding for  monitoring and evaluation of nonemergency food aid. It aims to improve  how nonemergency food aid data are collected and analyzed, to determine  the appropriate level of food aid, as well as the appropriate commodities  that can be distributed and monetized in each country.
The pilot project reduces the potential for conflict of interest because the  organization that does the analysis is no longer the same organization whose  interest is directly affected by the outcome of the analysis. In the past,  USAID’s implementing partners for nonemergency food aid distribution or  monetization conducted and submitted the Bellmon analysis to USAID.  However, the outcome of the analysis affected whether they could conduct  monetization and, if so, how much food aid they could monetize—which, in  turn, directly affected their operating budgets. Under the pilot project, USAID  will have one organization, Fintrac, carry out the initial market analysis for all  20 of USAID’s priority countries. According to Fintrac, having one  organization do the analysis for all countries could help to improve  consistency in analytical methodology, reporting structure, and format.
BEST is still in its early stage of implementation. It encountered some  difficulties in its first year of implementation and has yet to address some  additional methodological challenges. One key challenge is ensuring the  quality and consistency of the analysis, which can be impaired by a lack of  reliable and up-to-date data. For example, Fintrac was able to use data  from a recent livelihoods survey done by the World Food Program and  FEWS NET in the Malawi assessment but had to conduct additional  research and trips to Guatemala because it had no comparable updated  data for that country. In addition, Fintrac officials told us that they have  not developed quantifiable measures and thresholds agreed upon by  USAID and its implementing partners to help guide USAID’s programming  decisions on levels of monetization. Without quantifiable measures and  thresholds, implementing partners could challenge USAID’s determination  of monetization levels in a country. Finally, the law requires that the  distribution of commodities will not result in a substantial disincentive or  interference with domestic production or marketing in a recipient  country—but the information Fintrac uses for its analysis, such as the  historic levels of monetization relative to a country’s imports or  production, does not directly answer the question of whether U.S. food aid  will cause such disincentives. USAID and Fintrac officials acknowledged  that additional research is needed in this area. As of July 2009, Fintrac had  completed analyses for 7 priority countries: Burundi, Ethiopia, Guatemala,  Madagascar, Malawi, Sudan, and Uganda, with plans to complete analyses  for the remaining 13 priority countries through 2011.
section_title: USAID’s Monitoring and Evaluation of Nonemergency Food Aid Are Not Fully Integrated with Program Management
paragraphs: USAID’s monitoring and evaluation of its nonemergency food aid  programs are consistent to varying degrees with some of the principles  established by AEA to integrate evaluation into program management.  We  found that USAID’s actions were (1) generally consistent with the AEA  principles for policies and procedures and for independence; (2) partially  consistent with the principles for scope and coverage, dissemination of  results, professional competence, and resources; and (3) not consistent  with the principles for monitoring and evaluation plans.
subsections: section_title: USAID’s Actions Were Generally Consistent with the AEA Principles for Policies and Procedures and for Independence
subsections: section_title: FFP Has Issued Policies and Procedures for Monitoring and Evaluation
paragraphs: procedures and adopt quality standards to  guide the conduct of evaluations. the kinds of evaluations to be performed  and the criteria and administrative steps  for developing evaluation plans and setting  priorities; selecting the appropriate  evaluation approaches and methods to be  used; consulting subject matter experts;  ensuring the quality of the evaluation  products; publishing evaluation reports;  and ensuring the independence of the  evaluation function.
FFP has established policies and procedures for monitoring and  evaluation that are contained primarily in several information bulletins for  staff and implementing partners. Guidance can also be found in the FFP  Process and Procedures Manual, which outlines field and headquarters  staff’s roles and responsibilities for monitoring and evaluation. In addition,  FFP has issued a series of technical papers by FANTA-2 that discuss the  methodological standards and techniques that can be used for many  aspects of monitoring and evaluation, such as statistical sampling for  surveys of beneficiaries and preparing survey questions for key indicators.  For example, one technical paper discusses in detail a key FFP indicator,  household dietary diversity, and explains how to collect data through a  questionnaire, tabulate the responses, and set appropriate targets to track  progress over time. FFP’s policies and procedures, in conjunction with  USAID’s official guidance, cover the types of evaluations that can be  performed, methodologies that can be employed, ways to plan evaluations,  the means to disseminate studies, and the independence of the staff  performing the evaluations.
USAID’s new central evaluation office collaborated with State’s Office of  the Director of Foreign Assistance (State/F) to develop evaluation  guidelines and standards that were placed on the United States Foreign  Assistance Web site in March 2009. These standards and guidance  elaborate on the official USAID and FFP guidance, and they specifically  address some elements that are not directly addressed in the official  guidance, most notably the standards for quality. Collectively, the FFP- specific, USAID-wide, and Foreign Assistance policies and procedures are  generally consistent with the AEA principles.
section_title: FFP Requires External Evaluations to Help Ensure Independence
paragraphs: To help ensure independence as outlined in the AEA principle, FFP  requires that each multiyear assistance program’s final evaluation be  performed by external evaluators. It also relies on external entities to  perform cross-cutting evaluations using the Food for Peace Act and other  funding sources.
Independence in the context of final evaluations for FFP multiyear  assistance programs means that the key evaluation experts, the team  leader, and sectoral leads, should be external consultants hired by the  project. Our review of the 16 final evaluations that we found on USAID’s  Development Experience Clearinghouse (DEC) Web site determined that  external consultants were generally listed as the prime authors. In some  instances, the external evaluators were affiliated with a particular  consulting firm and, in other instances, several external evaluators formed  a team to perform the final evaluation.
In addition, the cross-cutting evaluations and other FFP-funded studies are  to be conducted by FANTA-2 and external consultants. FANTA-2 is part of  AED, a nonprofit organization that cites integrity as one of its core  values. Our review of selected task orders, signed and made available by  FANTA-2, found them in line with the AEA principle for independence. For  example, not only did its task order for a subrecipient of a new 3-year  cross-cutting evaluation describe the project’s phases, activities, and  methodologies, but it also clearly specified that the subrecipient’s staff  would develop preliminary conclusions based on the collected data, draft  interim reports, and submit a final report to USAID.
section_title: USAID’s Actions Were Partially Consistent with the AEA Principles for Scope and Coverage, Dissemination of Results, Professional Competence, and Resources
subsections: section_title: USAID Requires Few Cross- Cutting Evaluations
paragraphs: programs and policies, making it integral  to planning, developing, and managing  government programs at all stages of their  development. should be subject to evaluation, which is a  responsible and necessary part of good  management. submit annual questionnaires that respond to questions about FFP’s  food security indicators; and    submit midterm and final project evaluations.
In addition, USAID project officers must review the evaluation plans and  final reports. Finally, the final evaluation must include questionnaires that  are comparable to the baseline surveys, along with information about  sample design, universe, methods of analysis, and sampling frame.
During our fieldwork in Bangladesh and Haiti, we saw several examples of  FFP’s implementing partners following these directives. For example, we  visited a project in Bangladesh that had collected baseline data on a wide  range of indicators that measured aspects of land ownership, height and  weight-for-age measures, maternal and child health practices, access to  water and sanitation, household food security, and disaster preparedness.  In addition, we visited a project in Haiti that had produced a quarterly  report including targets for more than 50 measures, along with data on  whether or not those targets had been achieved, and reasons why some  targets had not been met.
FFP’s required monitoring and evaluation of the multiyear programs focus  on whether the programs are achieving their expected outcomes—but  they do not assess program impacts, or the effectiveness and efficiency, of  particular approaches. To achieve this, FFP would need to conduct  periodic, cross-cutting evaluations that compare what actually happened  with what would have happened in the absence of the program or  activity. While FFP has commissioned some of these studies, according  to USAID officials, it has no requirements to do so and has conducted  only a relatively small number over the last 5 years. In contrast, FFP  required every one of more than 100 multiyear projects concluded over the  last 5 years to be evaluated for outcomes.
section_title: Implementing Partner Evaluation Results Are Not Routinely and Widely Disseminated
paragraphs: Although USAID policy requires final evaluations of its nonemergency  food aid programs to be posted on a special Web site known as DEC, the  implementing partners responsible for these evaluations often do not  adhere to this requirement. The DEC Web site is accessible to the general  public and contains both simple and more advanced search engines. For  example, a user interested in Food for Peace nonemergency programs  could perform a search for “agriculture,” “maternal and child health,” or  “food security” final evaluations conducted in recent years. A user could  also search for final evaluations sponsored by FFP. However, FFP officials  told us that the MYAP implementing partners often do not submit their  final evaluations to DEC, even though this is a program requirement.  According to these officials, there is no penalty if the implementing  partners do not submit their reports to DEC. We were unable to find in  DEC a final evaluation of an integrated food security program that FFP  staff in Bangladesh provided to us in hard copy. In addition, more than 100  FFP-sponsored final evaluations of multiyear programs should have been  conducted from 2004 to 2009, according to FFP staff, but our search in  DEC found only 16.
section_title: Professional Competence Varies among the Different Groups of Staff Who Monitor and Evaluate FFP Programs
paragraphs: Although the entities that USAID has selected to perform cross-cutting  evaluations have staff with appropriate professional training and  experience, questions have been raised about the technical skills of some  staff that perform monitoring and evaluation for the implementing  partners. As FFP does not have reliable data on the number of its field  staff that perform monitoring and evaluation functions, and the specific  skills and competencies of those staff, it is not possible to determine their  overall competence. In addition, as FFP’s ability to hire monitoring and  evaluation specialists in Washington, D.C., is limited, and as USAID has  only recently reestablished its central evaluation unit, there has been an  absence of skilled evaluators in headquarters that could assist FFP’s  officers in the field and implementing partners’ staff. recognize the multidisciplinary nature of  evaluation and assure that the diversity of  disciplines is appropriately represented in  both internal and independent consulting  teams.
FFP’s Partners Have Training and Experience Consistent with the  AEA Principles  The partner organizations that FFP relies on to provide technical  assistance and support and to conduct cross-cutting evaluations have  training and expertise that are consistent with the AEA principles for  professional competence. The AED FANTA-2 project, with which USAID  has a cooperative agreement for various types of technical assistance,  employs staff experienced in monitoring and evaluation, with backgrounds  in economics, sociology, nutrition, anthropology, public health, and  epidemiology. In addition, FANTA-2 has partnered with universities and  research institutes, such as Tufts University and the International Food  Policy Research Institute, for particular evaluations and other research  projects.
Concerns Have Been Raised about Certain Aspects of  Implementing Partners’ Monitoring and Evaluation Skills  FFP’s implementing partners’ staff perform important monitoring and  evaluation functions, such as gathering data on performance indicators  and tracking these indicators. Yet FFP officials in Washington, D.C., and in  the field, as well as some external evaluators, have expressed concerns  about the competency of these staff to perform certain aspects of  monitoring and evaluation, as follows:    Staff can lack key evaluation skills. Implementing partners’ staff can lack  key evaluation skills, according to an AED employee who provided  training and support to FFP’s implementing partners and several external  evaluators. According to the AED employee, implementing partners’ staff  could execute research designs adequately but often were not capable of  designing the research themselves, or of taking responsibility for key  aspects of data collection, such as developing surveys or selecting  statistical samples. FFP officials stated that they recognize the need to  build capacity among the implementing partners, and are providing  monitoring and evaluation training in the field for these FFP and  implementing partner staff.
Difficulties in hiring and retaining qualified staff. Adequacy of staffing  levels and staff turnover have been cited as a concern in some countries.  For example, the implementing partners in Haiti told us that it was  difficult to attract and retain technically qualified staff. They stated that  Haiti has only a limited supply of people with the appropriate skills and  training for monitoring and evaluation. To address the shortage, one  implementing partner recently hired a new monitoring and evaluation  program officer from an African country who had appropriate  methodological skills but did not speak either French or Creole, Haiti’s  official languages, at the time of his arrival in-country.
FFP Lacks Data on the Monitoring and Evaluation Competencies of  Its Field Staff  FFP does not have the data necessary to assure that it has enough staff  with the right skills. According to FFP officials, USAID does not collect an  inventory of its staff’s skills and abilities, which would help the agency to  determine whether it has staff with sufficient training and skills in  monitoring and evaluation. Furthermore, neither USAID nor FFP have  data that can indicate which staff spend the majority of their time on  monitoring and evaluation, or how much staff time is spent on various  monitoring and evaluation activities, such as checking on warehouses,  advising implementing partners on performance indicators, or  participating in evaluation designs. While FFP attempted to gather some  data for us on its field staff who perform monitoring and evaluation, these  data were not reliable.
In Haiti, for example, FFP officials initially told us that nine staff acted as  field monitors. However, we found that this number included staff  members who did not spend much of their time involved in routine  monitoring and evaluation, such as the FFP officer, his deputy, and two  secretaries. Of the FFP staff in country, only four appeared to spend  substantial parts of their time on specific monitoring activities, such as  performing routine checks of warehouses or health clinics and giving  implementing partners advice on indicators.
One reason that it is difficult for FFP to determine which staff spend the  majority of their time on monitoring is that definitions of this function  vary. For example, while a recent FFP bulletin defined monitoring as the  regular tracking of program implementation, FFP guidance to field staff  defined monitoring more broadly, to include activities such as reviewing  contracts and ensuring that financial audits are performed. If the broader  definition of monitoring is used, more staff members can be classified as  performing this function, but then it is not possible to determine which of  those staff are spending the majority of their time on monitoring program  inputs and program implementation.
FFP Lacks Staff in Headquarters Who Are Skilled in Monitoring  and Evaluation  AEA has recommended that to support evaluations agencies either  establish centers of evaluation or appoint evaluation coordinators for  individual programs that conduct their own evaluation. According to FFP  officials, FFP has been unable to hire monitoring and evaluation staff at  headquarters to assist and support its officers in the field. FFP officials  attributed this to restrictions in the Food for Peace Act, which precludes  them from hiring any staff in Washington, D.C., to monitor and evaluate  nonemergency food aid programs. Moreover, as USAID has only recently  reestablished its central evaluation unit, FFP has not been able to seek  advice and guidance from professional, experienced evaluators at  headquarters, or to ask them to perform evaluations of FFP’s programs.
section_title: Greater Resources Are Now Available for Monitoring and Evaluation, but Their Sufficiency Is Difficult to Determine Absent a Needs Assessment
paragraphs: The Food for Peace Act provides FFP with a stable source of new funding  of up to $22 million annually for each of the fiscal years 2009 through 2012  to improve, monitor, and evaluate the effectiveness and efficiency of  nonemergency food aid programs. In addition, FFP has access to other  funding streams, such as DA funds, which have recently increased. As a  result, FFP officials told us that they currently have more resources to  devote to monitoring and evaluation than in the past.
Principle: Sufficient resources should be made  available for evaluation, including stable  annually recurring sources of funds and  special one-time funds for evaluation projects  of interest to executive branch and  congressional policymakers.
Although availability of the newly authorized funding for monitoring and  evaluation as well as funding from other sources is generally consistent  with the AEA principle on resources, it is not possible to determine if  funding is sufficient because FFP has not systematically determined its  research needs, prioritized those needs, or estimated the costs associated  with them. In the course of our audit work, we identified several areas  where the current resource levels might not be sufficient. For example,  experts we interviewed provided examples of potential evaluations or  studies that could help improve FFP’s programs. However, according to  USAID officials, FFP has not undertaken the formal exercise of compiling  a list of these studies, or prioritizing its needs for them. In addition,  FANTA-2 staff who review many of the implementing partners’ multiyear  plans for evaluation raised concerns that FFP does not sufficiently ensure  that these evaluations are adequately funded. However, FFP has not  reviewed or estimated the amounts its implementing partners are spending  on monitoring and evaluating their multiyear projects because, according  to FFP, that would be burdensome as it would require a review of each  project’s contract. Without such an assessment, it is not possible to  determine whether sufficient resources are being spent on the required  final evaluations.
section_title: USAID’s Actions Were Not Consistent with the AEA Principles for Monitoring and Evaluation Plans
paragraphs: According to FFP officials, FFP plans some of its monitoring and  evaluation activities, but lacks an overall plan that integrates its  monitoring and evaluation with program management. FFP officials  reported that they make decisions about monitoring and evaluation based  on discussion and a thorough consideration of options but have not  undertaken the rigorous planning process that is recommended by AEA.  Therefore, FFP does not have an integrated plan for all of its oversight  activities to facilitate strategic planning, budgeting, and program  management, and to indicate how it will use monitoring and evaluation  results to improve program management. Moreover, FFP does not have a  mechanism to link resources to needs based on risk, according to USAID  officials. As a result, when FFP allocates resources for monitoring and  evaluation, it does not take into account factors such as the country  context, vulnerability to fraud, waste, and abuse, program efficiency, and  critical issues. As we noted earlier, we cannot determine whether FFP has  sufficient resources because this type of planning has not been conducted,  which means that FFP’s monitoring and evaluation needs have not been  prioritized and compared against available resources.
A senior FFP official told us that he could see the potential value of such a  plan but wondered whether FFP—which has a staff of about 60 people  and which he characterized as being small in relation to other USAID  units—could support such an effort. However, FFP appears to have the  resources for such a planning effort, as the Food for Peace Act has  provided it with up to $88 million to improve, monitor, and evaluate the  effectiveness and efficiency of nonemergency food aid programs over a 4- year period, and has access to other resources that can be used for  monitoring and evaluation, such as DA funds.
section_title: Conclusions
paragraphs: Nonemergency food aid programs are an essential component of the U.S.  strategy to reduce global food insecurity, and Congress has authorized  new funding for USAID to improve its oversight of these programs. In  response, USAID’s FFP has initiated actions over the past year to  strengthen its monitoring and evaluation of food assistance. However,  weak planning could impede these efforts, including the upgrade of FFP’s  information technology system. In addition, USAID has yet to determine  how it intends to assure stable resources, within existing authorities, to  fully implement all the actions planned by FFP to improve monitoring and  evaluation of nonemergency food aid. As a result, planned actions that are  essential to effective program management, such as the expansion of a  computerized monitoring system for food aid programs and cross-cutting  evaluations, could be jeopardized. Finally, FFP lacks an integrated and  comprehensive monitoring and evaluation plan that links resources to  needs based on risk. In the absence of such a plan, it is difficult to  determine whether USAID is allocating resources for monitoring and  evaluation to its priority needs, such as program efficiency, critical issues,  and programs vulnerable to fraud, waste, and abuse.
section_title: Recommendations for Executive Action
paragraphs: To facilitate the implementation of ongoing efforts to improve monitoring  and evaluation and ensure effective program management of  nonemergency food aid, we recommend that the Administrator of USAID  take the following two actions:    Develop a concept of operations document to help reduce the risks  associated with upgrading FFP’s information technology system. Such a  document should adhere to industry best practices and include key  elements such as major system components, interfaces to external  systems, and performance characteristics. It would also describe system  characteristics for a proposed system from a user’s point of view and  include high-level descriptions of information systems, their  interrelationships, and information flows.
Develop an integrated monitoring and evaluation plan to ensure that the  results of its oversight activities are used for effective management of  nonemergency food aid programs. Such a plan should, among other things,  (1) link monitoring and evaluation to key USAID and FFP goals;  (2) establish a systematic process for determining appropriate budget  levels and staff resources for monitoring and evaluation based on an  analysis of the nature, scope, and size of the programs, and the unique  conditions presented, in the recipient countries; (3) examine all available  options for funding monitoring and evaluation at headquarters and in  recipient countries; and (4) establish time frames for implementing and  evaluating the plan.
section_title: Agency Comments and Our Evaluation
paragraphs: USAID provided written comments on a draft of this report.  We have  reprinted these comments in appendix IV, along with our responses.   Additionally, USAID provided technical comments on a draft of this  report, which we have addressed as appropriate.
USAID concurred with our first recommendation to develop a concept of  operations document to help reduce the risks associated with upgrading  FFP’s information technology system. USAID disagreed with the second  recommendation, asserting that we did not adequately take into account  documents that detail its planning processes.  We considered all the  documents that USAID cited and found that they did not constitute plans,  but only provided guidance for planning. Furthermore, the weaknesses we  identified in FFP’s monitoring and evaluation efforts, including in the  areas of resources and professional competence, demonstrate the need for  a comprehensive monitoring and evaluation plan.
USAID acknowledged that it does not have an integrated monitoring and  evaluation plan, but said that it does not see the benefits of developing it  since the elements of such a plan exist in some form already. USAID noted  that we did not mention FFP’s Strategic Plan for 2006-2010 and a related  document, which link monitoring and evaluation to agency goals and lay  out performance indicators. We considered the strategic plan irrelevant to  our analysis because it was prepared in 2005, prior to the passage of the  2008 Food for Peace Act, and covers only some aspects of monitoring.  USAID also stated that its planned studies form part of an agency agenda  to inform program management. We reported on FFP’s planned studies  but noted that, because funding for these studies is not assured due to  legal restrictions, USAID may need to find alternative funding sources.  Moreover, USAID officials reported that they did not use almost $9 million  of last year’s available funds authorized by the 2008 Food for Peace Act for  oversight of nonemergency food aid. We consider these to be compelling  reasons for USAID to develop an integrated monitoring and evaluation  plan.
In terms of staffing, FFP officials said that their first priority is to place  staff in all the countries where they have nonemergency food aid  programs. FFP acknowledged that it does not have data on field staff that  perform monitoring and evaluation functions, but said that such data  would be costly to collect, and that we had not made the case for  collecting them. However, we found that FFP currently does not know  what percentage of time its staff spend on monitoring and evaluation and  what specific monitoring and evaluation functions they perform.  We  believe that because FFP had a substantial new funding authority for  oversight activities, it should invest in a survey of staff skills and an  assessment of monitoring and evaluation needs, which would provide FFP  with essential data for future staffing decisions. We believe that this basis  for our recommendation for an integrated monitoring and evaluation plan  remains valid.
We are sending copies of this report to interested Members of Congress  and the Administrator of USAID.  The report is also available at no charge  on the GAO Web site at http://www.gao.gov.
If you or your staffs have any questions about this report, please contact  me at (202) 512-9601 or melitot@gao.gov. Contact points for our Office of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: In passing the Food for Peace Act in 2008, Congress required the  Comptroller General of the United States to review the U.S. Agency for  International Development’s (USAID) Report to Congress on the agency’s  efforts to strengthen the monitoring and evaluation of its nonemergency  food aid programs and provide recommendations for improvement. To  address this congressional mandate, we reviewed (1) USAID’s plans and  actions to improve its monitoring and evaluation of nonemergency food aid  programs provided under the Food for Peace Act and (2) the extent to  which USAID has integrated its monitoring and evaluation of nonemergency  food aid with program management, particularly with regard to established  principles. In this review, we focused on oversight practices of USAID’s  Office of Food for Peace (FFP), not of the agency as a whole.
To address USAID’s plans to improve the monitoring and evaluation of  nonemergency food aid programs using the funds provided under the  Food for Peace Act, we reviewed the list of activities that the agency  reported to Congress in December 2008. This included plans to make  improvements in  the number of field staff monitoring food aid programs,    computerized systems for monitoring food aid programs,  the market analysis required before food aid programs are approved in  recipient countries,    cross-cutting evaluations of nonemergency food aid projects,    FFP’s information technology system, and  the Famine Early Warning Systems Network (or FEWS NET).
We interviewed knowledgeable USAID officials and partners responsible for  implementing these plans and reviewed relevant documents, including  contracts and plans. We also examined FFP data on the allocations of the  Food for Peace Act funding for oversight of nonemergency food aid in fiscal  year 2009. Based on interviews, and our analysis of the available documents,  we determined that the data were sufficiently reliable to indicate the relative  amounts allocated to each of the oversight activities, and to demonstrate  that almost $9 million were not used in fiscal year 2009.
In addition, we conducted fieldwork in Bangladesh and Haiti. We selected  these two countries because they have the largest Food for Peace  nonemergency food aid programs, but they differ in their approaches to  monitoring and evaluation. Haiti has more staff working on these  programs than does Bangladesh and uses the Layers computerized  monitoring system, which Bangladesh does not. Haiti’s staff spend  relatively more time monitoring programs directly, while FFP in  Bangladesh relies more on contract officers and implementing partners to  perform its monitoring and evaluation. In addition, Haiti is in the  Caribbean while Bangladesh is in South Asia, which provided a regional  contrast. While we could not generalize from these two countries to all the  countries that have Food for Peace programs, we gained useful insights  into monitoring and evaluation from our fieldwork.
To consider the extent to which USAID’s monitoring and evaluation of  nonemergency food aid are aligned with program management, we  selected the American Evaluation Association’s (AEA) principles for  integrating evaluation into program management. We selected these  principles as criteria because they are consistent with the goal set by the  Secretary of State, who has ultimate responsibility for USAID, to ensure  evidence-based decisions in development programs.
In this review, we combined the AEA standards for scope and for coverage  because we found overlap between them. In addition, we did not assess  one of the AEA standards on methods because that would have involved a  review of studies that was beyond the scope of this report, and because  that principle was less specific than those for the other elements,  essentially noting that a mix of methods should be used, but that the actual  methods used should depend upon the topic to be evaluated.
The AEA principles differ in length and detail, with some including several  parts and clauses, while others are more succinct. For the more detailed  principles, we identified the sentences that summarized them at the  highest level and used those as our operating criteria. For the sake of  transparency, we published the exact principles we used in our report  next to our discussion of them.
To gather evidence to apply against the principles, we interviewed  cognizant FFP officials in Washington, D.C., and in Bangladesh and Haiti.  We also interviewed the Chief of USAID’s Evaluation Office, senior staff at  the Food and Nutrition Technical Assistance (FANTA-2) project of the  Academy for Educational Development (AED), with which USAID has  ongoing cooperative agreement, as well as experts knowledgeable about  USAID’s evaluation and monitoring. We reviewed USAID and FFP policy  guidance and bulletins that discuss and describe evaluation. We reviewed  the evaluations that we found on USAID’s Development Experience  Clearinghouse (DEC) Web site and that were given to us by USAID and its  implementing partners. We also examined related monitoring and  evaluation documents, such as performance plans and annual project  reports, and documentation of the training USAID provides on monitoring  and evaluation.
We reviewed this evidence and compared it against each of the AEA  principles to make a general, high-level determination on the broad degree  to which FFP was consistent or inconsistent with them. In the course of this  review, we recognized that FFP has its own guidance and practices, but that  it follows the overall guidance and practices set by USAID for the agency as  a while, and considered each source as appropriate.
In addition to the AEA principles, we also considered GAO standards and  prior GAO work, particularly on workforce planning; USAID guidance; and  guidelines that the Institute of Electrical and Electronics Engineers (IEEE)  has established for information technology.
As USAID performs a wide range of activities to monitor and evaluate its  Food for Peace program, we defined evaluation broadly for the purposes  of this report. This definition includes evaluating inputs and  implementation, outcomes, and impacts. We also considered other  research studies that FFP had conducted to help manage its programs,  such as prospective studies on the impacts of providing particular food  commodities to particular markets, and analyses of individual countries’  food strategies. In addition, we included the selection and tracking of  performance indicators in our review because FFP and USAID define  monitoring to include the tracking of these indicators during the life cycle  of their programs, and use them in the final evaluations.
We conducted this performance audit from March 2009 to September 2009,  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: USAID Report to Congress
section_title: Appendix III: Examples of Activities Supported with Nonemergency Food Aid
paragraphs: Figure 5 provides examples of activities we observed in Bangladesh and  Haiti that are illustrative of the various activities supported with  nonemergency food aid.
section_title: Appendix IV: Comments from USAID
paragraphs: The following are GAO’s comments on the U.S. Agency for International  Development letter dated September 17, 2009.
subsections: section_title: GAO Comments
paragraphs: 1.  We reviewed the eight documents listed and acknowledged several of  them in our report’s section on policies and procedures.  However,  none of these documents constitutes a plan. Instead, they are policy or  technical documents that describe how USAID and its awardees  should plan for aspects of monitoring and evaluation.  2.  We considered FFP’s Strategic Plan for 2006-2010 but determined it  to be irrelevant to our analysis because it was prepared in 2005, prior  to the passage of the 2008 Food for Peace Act, and covers only some  aspects of monitoring. The Performance Management Plan is related  to the strategic plan and provides a set of indicators to measure  progress. Neither of these documents discusses funding, resources,  staffing needs, or how results would be used to improve program  management.  3.  We reported on the planned studies earlier in this report. However,  USAID provided us not with a plan, but rather with a list of selected  studies, their implementers, costs, and a high-level description. In the  list, USAID did not discuss alternatives, outline priorities, or explain  how the selected studies would address FFP’s most pressing needs or  be used for program management.  As we noted, several studies do not  have assured funding due to the legal restrictions on FFP’s funding,  which makes the need for integrated planning even more critical.  4.  We noted that FFP allocated the new staff positions in order to ensure  coverage in all countries where it has programs. While this is one  factor to consider in workforce planning, FFP did not consider other  important factors, such as existing staff skills or country-specific  needs for monitoring and evaluation because it did not collect data on  them. Lacking a systematic process to collect such data, FFP is unable  to determine if it has adequate staff with sufficient training and skills  in monitoring and evaluation.  5.  FFP officials reported that they had not used almost $9 million of the  $22 million available in fiscal year 2009.  Better planning could ensure  that the available funds are more fully used in future years for  activities such as surveying FFP staff on their monitoring and  evaluation skills and conducting an assessment of FFP monitoring and  evaluation needs.
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Phillip Thomas, Martin De  Alteriis, and Michael LaForge, Assistant Directors; and Sada Aksartova;  Carol Bray; Ming Chen; Debbie Chung; Lynn Cothern; Mark Dowling; Brian  Egger; Ian Ferguson; Etana Finkler; Kendall Helm; Joy Labez; Jeff Miller;  Yvonne Moss; Ulyana Panchishin; Lisa Reijula; Julia Ann Roberts; Barbara  Shields; Stephanie Shipman; and George Taylor made key contributions to  this report.
section_title: Related GAO Products
paragraphs: International Food Assistance: Key Issues for Congressional Oversight.  GAO-09-977SP. Washington, D.C.: September  30, 2009.
International Food Assistance: Local and Regional Procurement  Provides Opportunities to Enhance U.S. Food Aid, but Challenges May  Constrain Its Implementation. GAO-09-757T. Washington, D.C.:  June 4,  2009.
International Food Assistance: Local and Regional Procurement Can  Enhance the Efficiency of U.S. Food Aid, but Challenges May Constrain  Its Implementation. GAO-09-570. Washington, D.C.: May 29, 2009.
International Food Security: Insufficient Efforts by Host Governments  and Donors Threaten Progress to Halve Hunger in Sub-Saharan Africa  by 2015. GAO-08-680. Washington, D.C.: May 29, 2008.
Foreign Assistance: Various Challenges Limit the Efficiency and  Effectiveness of U.S. Food Aid. GAO-07-905T. Washington, D.C.: May 24,  2007.
Foreign Assistance: Various Challenges Impede the Efficiency and  Effectiveness of U.S. Food Aid. GAO-07-560. Washington, D.C.: April 13,  2007.
Foreign Assistance: U.S. Agencies Face Challenges to Improving the  Efficiency and Effectiveness of Food Aid. GAO-07-616T. Washington,  D.C.: March 21, 2007.
Darfur Crisis: Progress in Aid and Peace Monitoring Threatened by  Ongoing Violence and Operational Challenges. GAO-07-9. Washington,  D.C.: November 9, 2006.
Maritime Security Fleet: Many Factors Determine Impact of Potential  Limits of Food Aid Shipments. GAO-04-1065. Washington, D.C.:  September 13, 2004.
United Nations: Observations on the Oil for Food Program and Iraq’s  Food Security. GAO-04-880T. Washington, D.C.: June 16, 2004.
Foreign Assistance: Lack of Strategic Focus and Obstacles to  Agricultural Recovery Threaten Afghanistan’s Stability. GAO-03-607.  Washington, D.C.: June 30, 2003.
Foreign Assistance: Sustained Efforts Needed to Help Southern Africa  Recover from Food Crisis. GAO-03-644. Washington, D.C.: June 25, 2003.
Food Aid: Experience of U.S. Programs Suggest Opportunities for  Improvement. GAO-02-801T. Washington, D.C.: June 4, 2002.
Foreign Assistance: Global Food for Education Initiative Faces  Challenges for Successful Implementation. GAO-02-328. Washington, D.C.:  Feb. 28, 2002.
Foreign Assistance: U.S. Food Aid Program to Russia Had Weak Internal  Controls. GAO/NSIAD/AIMD-00-329. Washington, D.C.: September 29,  2000.
Foreign Assistance: U.S. Bilateral Food Assistance to North Korea Had  Mixed Results. GAO/NSIAD-00-175. Washington, D.C.: June 15, 2000.
Foreign Assistance: Donation of U.S. Planting Seed to Russia in 1999  Had Weaknesses. GAO/NSIAD-00-91. Washington, D.C.: March 9, 2000.
Foreign Assistance: North Korea Restricts Food Aid Monitoring.  GAO/NSIAD-00-35. Washington, D.C.: October 8, 1999.
Food Security: Factors That Could Affect Progress toward Meeting World  Food Summit Goals. GAO/NSIAD-99-15. Washington, D.C.: March 22, 1999.
Food Security: Preparations for the 1996 World Food Summit.  GAO/NSIAD-97-44. Washington, D.C.: November 7, 1996.