Afghanistan Development: Enhancements to Performance Management and Evaluation Efforts Could Improve USAID's Agricultural Programs
Published: 2010-07-14T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Eighty percent of Afghans are dependent on agriculture for their livelihoods. Agricultural assistance is a key U.S. contribution to Afghanistan's reconstruction efforts. Since 2002, the U.S. Agency for International Development (USAID) has awarded about $1.4 billion for agricultural programs to increase agricultural productivity, accelerate economic growth, and eliminate illicit drug cultivation. This report (1) describes the change in U.S. focus on agricultural assistance since 2002, (2) assesses USAID's performance management and evaluation of its agricultural programs, (3) analyzes the extent to which certain programs met targets, and (4) addresses efforts to mitigate implementation challenges. GAO reviewed USAID documents; analyzed program data; and interviewed program implementers and USAID officials in Washington, D.C., and Afghanistan. GAO has prepared this report as part of its ongoing efforts to monitor key aspects of U.S. efforts in Afghanistan.
section_title: What GAO Found
paragraphs: The United States' focus in providing agricultural assistance to Afghanistan shifted from food security programs in 2002 to counternarcotics-related alternative-development programs in 2005. This focus on providing farmers with alternatives to growing opium poppy lasted through 2008. In 2009, the Administration shifted the focus of its agricultural strategy in Afghanistan from counternarcotics to counterinsurgency, noting that economic growth and new job creation were critical to U.S. efforts in Afghanistan because they provide alternatives to narcotics- and insurgent-related activities. USAID's Automated Directives System established planning, monitoring, and evaluation procedures that USAID was expected to follow in Afghanistan. USAID planning efforts prior to 2009 largely follow these procedures. However, since the end of 2008, USAID has operated without a required Mission performance management plan for Afghanistan. In addition, USAID did not approve all implementing partner monitoring plans for the eight USAID agricultural programs, which represented about 75 percent of all USAID agricultural awards since 2002. USAID also did not assure all indicators had targets. USAID undertook efforts to monitor agricultural programs, but due to security concerns could not consistently verify reported data. USAID did not consistently analyze and interpret or document program performance for these eight programs, active between 2007 and 2009, on which our review focused. In the absence of this analysis, USAID did not document decisions linking program performance to changes made to the duration or funding of programs. USAID conducted one evaluation covering three of the eight programs, but the extent to which or whether USAID used the evaluation to enhance current or future programs is unclear. We found that the eight agricultural programs we reviewed did not always establish or achieve their targets for each performance indicator. USAID requires implementing partners to submit information on indicators, targets, and results. We measured performance for the eight programs by comparing annual results against annual targets and determining the extent to which targets were met. Six of the eight programs did not meet their performance targets in the most recent year for which targets were reported. For the two programs that met all their targets, we found they failed to establish targets for several indicators and, thus, we could not fully assess performance for those indicators. We also found that the three longest-running programs in our review showed declines in performance from fiscal years 2006 to 2008. USAID faces several challenges to implementing its agricultural programs in Afghanistan, such as the security environment, and has taken steps to mitigate other challenges, such as working to improve Afghan government capacity. However, while USAID's lack of documentation and high staff turnover have hampered USAID's ability to maintain institutional knowledge, the agency has not taken steps to address this challenge.
section_title: What GAO Recommends
paragraphs: GAO recommends that the USAID Administrator take a number of steps to enhance performance planning, monitoring and evaluation, and knowledge transfer procedures. USAID agreed with our recommendations, highlighted ongoing efforts to improve in these areas, and noted the high-threat environment in which they are operating.

section_title: Letter
paragraphs: An estimated 80 percent of Afghanistan’s citizens depend on agriculture  and related agribusiness for their livelihoods. However, the country’s  agricultural sector has been hampered by decades of conflict; degradation  of land and infrastructure, including poor irrigation systems; and lack of  reliable sources of power and quality inputs, such as improved seed and  fertilizer. The United States considers agricultural assistance a key  contribution to Afghanistan’s reconstruction and stabilization. Moreover,  promoting sustainable development is a priority of the Afghanistan  National Development Strategy and other donor countries. Since 2002, the  U.S. Agency for International Development (USAID) has awarded about  $1.4 billion toward agricultural programs to increase agricultural  productivity, accelerate economic growth, and eliminate illicit drug  cultivation. The USAID Mission to Afghanistan (Mission) relies on  implementing partners to carry out its agricultural programs.
This review assesses (1) how the United States has changed the focus of its  agricultural efforts in Afghanistan since 2002, (2) USAID’s performance  management and evaluation efforts of agricultural programs in  Afghanistan, (3) the extent to which USAID’s agricultural programs in  Afghanistan met targets, and (4) USAID’s efforts to mitigate challenges in  implementing these programs in Afghanistan.
To address these objectives, we reviewed USAID and other U.S.  government planning, funding, and reporting documents related to U.S.  agricultural programs in Afghanistan. We discussed these programs with  officials from USAID and the Departments of Agriculture, Defense, and  State, including the Office of the Special Representative for Afghanistan  and Pakistan, as well as with private contractors and other implementing  partners working on U.S.-funded agricultural programs in Washington,  D.C., and Afghanistan. In Afghanistan, we also met with officials from the  United Nations and the governments of Afghanistan and the United  Kingdom to discuss agricultural-related efforts. We traveled to the  provinces of Badakhshan and Farah in 2009 to meet with U.S. and Afghan  officials, implementing partners, and aid recipients to discuss several U.S.- funded projects. We analyzed program budget data provided by USAID to  report on program funding, as well as changes in USAID’s program  monitoring officials over time. We analyzed program data provided by  USAID and its implementing partners to track performance against targets  over time. We took steps to assess the reliability of the budget,  performance, and attack data and determined they were sufficiently  reliable for the purposes of this report. We focused our analysis on the  eight USAID agricultural programs that were active between 2007 and 2009  and had total awards greater than $15 million. These programs represent  about 75 percent of all USAID agricultural awards since 2002. We did not  address agriculture-related projects carried out independently by other  U.S. government agencies, such the Department of Defense’s Commander’s  Emergency Response Program, or those carried out by multilateral  institutions to which the United States contributes, such as the United  Nations Development Programme.
We conducted this performance audit from March 2009 through July 2010  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. More detailed information on  our scope and methodologies, as well as data reliability assessments, can  be found in appendix I.
subsections: section_title: Background
paragraphs: Afghanistan is one of the world’s poorest countries and ranks near the  bottom of virtually every development indicator category, including life  expectancy; literacy; nutrition; and infant, child, and maternal mortality.  According to the most recent National Risk and Vulnerability Assessment  conducted by the government of Afghanistan between 2007 and 2008, the  Afghan poverty rate was 36 percent. The highest rates of poverty were  among nomads and rural farmers and varied across regions and provinces.  (For additional information on regional poverty in Afghanistan see GAO-10- 756SP.) The survey also found that agricultural activities provided the  Afghan population’s primary livelihood; 55 percent of households were  engaged in farming and 68 percent had livestock. According to the World  Bank, the agricultural sector accounts for 30 percent of Afghanistan’s gross  domestic product. The National Risk and Vulnerability Assessment states  that agricultural productivity is hampered by water shortage, lack of credit,  insufficient outreach of agricultural and veterinary extension services, and  poor access to markets.
Afghanistan suffers from limited means to capture water resources, soil  degradation, overgrazing, deforestation, and desertification. As shown in  figure 1, Afghanistan is mountainous and much of its land is not arable.  According to the National Risk and Vulnerability Assessment, household  access to arable land increased between 2005 and 2007/2008, largely due to  increasing access to irrigated land across urban, rural, and nomadic  households. Additionally, in 2007/2008, 40 percent of households  nationwide had access to irrigated land and 17 percent had access to rain- fed land. Farms in Afghanistan averaged 1.4 hectares for irrigated land and  2.8 hectares for rain-fed land. The survey also found that wheat was the  most frequently cited crop produced on irrigated and rain-fed land during  the summer planting season, followed by opium and potatoes on irrigated  land and cotton and barley on rain-fed land; corn, sorghum, and rice were  grown on irrigated land during the winter planting season. Some  households also grew fruit and nut trees and grapes. (For additional  information on major crops grown in Afghanistan see GAO-10-756SP.)
subsections: section_title: USAID Funds Agricultural Programs in Afghanistan
paragraphs: As figure 2 shows, between fiscal years 2002 and March 31, 2010, USAID  awarded about $1.4 billion to 41 agricultural-assistance programs in  Afghanistan, with almost two-thirds of the amount (about $900 million)  disbursed.
As table 1 shows, disbursements of U.S. funds for agricultural programs  represented 14 percent of all USAID assistance programs in Afghanistan  from fiscal years 2002-March 31, 2010. Moreover, the percentage of USAID’s  total assistance to Afghanistan disbursed to agricultural programs has  increased from 6 percent in fiscal years 2002-2004 to 17 percent in fiscal  years 2008 and 2009. Appendix II has more funding information on USAID’s  agricultural programs, including the eight agricultural programs in our  review.
The administration requested $827 million dollars in fiscal year 2010 for  USAID agricultural-assistance programs.
section_title: U.S. Agricultural Strategy Changing Focus to Support Counterinsurgency Priorities
paragraphs: In fiscal years 2002–2003, to help address the complex humanitarian crisis  in Afghanistan, the U.S. government provided emergency assistance that  helped avert a famine, significantly reduced the suffering of the most  vulnerable Afghans, and assisted the return of refugees. USAID provided  Afghanistan with 355,270 metric tons of wheat and other emergency food  assistance (valued at $206.4 million), and the U.S. Department of  Agriculture provided 79,600 metric tons of surplus wheat (valued at $38.7  million). Over the 2-year period, the United States provided over 60 percent  of all international food assistance received by Afghanistan. According to  the World Food Program, the food assistance provided by the United States  and the international community helped avert famine in Afghanistan.
As we previously reported, from 2002 through 2004, increased opium poppy  cultivation spread and drug trafficking grew as a threat to Afghanistan’s  security and stability. During that time, the United States supported  Afghan- and United Kingdom-led counternarcotics efforts. These efforts  reportedly had little effect on the illicit narcotics industry because of  limited security and stability across Afghanistan. In response, the U.S.  government made counternarcotics a top U.S. priority and developed a  strategy in 2004 to reduce poppy cultivation, drug production, and  trafficking, shifting the emphasis of the United States’ agricultural  assistance programs in Afghanistan from food security programs to  counternarcotics-related ADP. This part of the U.S. counternarcotics  strategy was intended to offer incentives to stop opium poppy production  by helping farmers and farm laborers obtain other ways to earn a living.  The strategy also called for strong disincentives such as forced eradication,  interdiction, and law enforcement, while at the same time spreading the  Afghan government’s antinarcotics message. The United States’ efforts also  were expected to build the Afghan government’s capacity to conduct  counternarcotics efforts on its own. As part of its counternarcotics efforts,  beginning in 2005, USAID awarded most of its new agricultural funds to  alternative-development programs (ADP)—to (1) increase agricultural  productivity, (2) accelerate economic growth, and (3) eliminate illicit drug  cultivation. As figure 3 shows, between 2005 and 2008, $494 million, or 71  percent of new awards were for ADP.
During this time period, USAID funded ADP, as well as other large  agricultural programs. Figure 4 provides a brief description of the goals  and objectives of the eight programs included in this review including,  beginning in 2005, ADP-Northeast, ADP-South, and ADP-East; beginning in  2006, the Accelerating Sustainable Agriculture Program; beginning in 2008,  ADP-Southwest, the Afghanistan Water, Agriculture, and Technology  Transfer program, and the Afghanistan Vouchers for Increased Production  in Agriculture program; and beginning in 2009, the Incentives Driving  Economic Alternatives-North, East and West, as a follow-on program to  ADP-East. USAID identified six of the eight programs as ADP, excluding the  Accelerating Sustainable Agriculture Program and the Afghanistan  Vouchers for Increased Production in Agriculture. Appendix III specifies  the provinces where the eight agricultural programs included in our review  operated and gives examples of the range of projects that these programs  implemented.
In 2009, under the direction of the President and the Special Representative  for Afghanistan and Pakistan, the United States shifted the focus of its  agricultural strategy in Afghanistan from counternarcotics to  counterinsurgency efforts. This shift de-emphasized eradication. According  to the Special Representative for Afghanistan and Pakistan, eradication  unduly punished and alienated farmers for making a “rational economic  decision,” while ignoring the profits gleaned by traffickers and insurgents  from the sale of processed opium and heroin. Further, the Administration  noted that economic growth and new job creation were critical to U.S.  counterinsurgency efforts in Afghanistan because they provide licit  alternatives to narcotics- and insurgent-related activities and connect  people to their government. As a result, the Administration integrated  agricultural programs with other U.S. efforts, including military operations,  and directed more resources to the agricultural sector. For example, the  Afghanistan Vouchers for Increased Production in Agriculture program,  which originally operated outside of the southeastern portion of  Afghanistan, was expanded to Helmand and Kandahar in 2010, where  according to Department of Defense officials, the United States and  Afghanistan have begun military operations to break the momentum of the  insurgency. Furthermore, the Administration increased the involvement of  the U.S. Department of Agriculture and aligned U.S. efforts with the current  agricultural priorities of the Afghan government, as laid out in the Ministry  of Agriculture, Irrigation, and Livestock’s National Agriculture  Development Framework. The strategy focuses on the following four areas:  1. increasing agricultural productivity by increasing farmers’ access to  quality inputs, such as improved seeds and fertilizer, and effective  extension services; 2. regenerating agribusiness by increasing linkages between farmers,  markets, credit, and trade corridors; 3. rehabilitating watersheds and improving irrigation structure; and 4. increasing the Ministry of Agriculture, Irrigation, and Livestock’s  capacity to deliver services and promote the private sector and farmer  associations through direct budget and technical assistance.
section_title: Gaps Exist in USAID’s Efforts to Plan, Monitor, and Evaluate Agricultural Programs in Afghanistan
paragraphs: USAID’s Automated Directives System establishes performance  management and evaluation procedures that USAID expects its staff to  follow in planning, monitoring, and evaluating its agricultural assistance  programs in Afghanistan. USAID operationalized these procedures  through the development of a Mission Performance Management Plan  (PMP). Similarly, USAID requires its implementing partners to develop  monitoring and evaluation plans, which are generally included in  implementing partners’ program PMPs. The collection of planning,  monitoring, and evaluating efforts, when taken together, enable USAID to  manage the performance of its programs. While USAID has noted that  Afghanistan is an insecure environment in which to implement its  programs, the agency has generally required the same performance  management and evaluation procedures as it does in other countries in  which it operates. In October 2008, USAID approved new guidance that  outlined several alternative monitoring methods—especially when site  visits are difficult or not possible—in high threat environments such as  Afghanistan. This guidance, however, was not disseminated to USAID staff  until December 2009, and the USAID Mission to Afghanistan agricultural  staff did not become aware of the guidance until June 2010. This guidance  was included in GAO’s review where applicable. Figure 5 presents a  summary of USAID’s Automated Directives System’s performance  management and evaluation procedures it expects its staff to follow,  grouped into planning, monitoring, and evaluating categories.
subsections: section_title: USAID’s Planning for Agricultural Programs in Afghanistan Generally Included Key Activities, but Lacks a Current Mission Performance Management Plan
paragraphs: USAID’s Automated Directives System requires USAID officials to  complete a Mission PMP for each of its high-level objectives as a tool to  manage its performance management and evaluation procedures. In line  with this requirement, USAID’s Mission to Afghanistan developed its first  PMP in 2006; the document covered the years 2006, 2007, and 2008. The  Mission operated without a PMP to guide its efforts after 2008. According  to USAID, the Mission is in the process of developing a new missionwide  PMP that will reflect the current Administration’s priorities and strategic  shift to counterinsurgency. USAID expects the new PMP to be completed  by the end of fiscal year 2010. The Mission attributed the delay in creating  the new PMP to the process of developing new strategies in different  sectors and gaining approval from the U.S. Embassy in Afghanistan and  from agency headquarters in Washington.
Overall, the 2006-2008 Mission PMP incorporated key planning activities.  For example, the PMP identified indicators, established baselines and  targets, planned for data quality assessments, and described the frequency  of data collection for four high-level objectives for all USAID programs in  Afghanistan, including its agricultural programs. The eight agricultural  programs we reviewed all fell under one of these four high-level objectives  identified in the Mission PMP—“developing a thriving licit economy led by  the private sector.” In addition, the PMP described regular site visits,  random data checks, and data quality assessments as the means to be used  to verify and validate information collected. Furthermore, the Mission PMP  noted that the PMP was developed to enable staff to actively and  systematically assess their contributions to the Mission’s program results  and take corrective action when necessary. It noted that indicators, when  analyzed in combination with other information, provide data for program  decision making. The Mission PMP did not include plans for evaluations  and special studies for the high-level objective that the eight programs  included in this review supported; but according to USAID, the agency has  planned evaluations for seven of the eight agricultural programs included  in this review during fiscal year 2010. In addition, USAID has planned to  conduct evaluations of agricultural depots and veterinarian field units,  activities included in several agricultural programs.
Similar to the Automated Directives System’s requirement that USAID  develop a Mission PMP as a planning tool to manage the process of  monitoring and evaluating progress—including establishing targets for  each performance indicator—implementing partners are required to  develop and submit monitoring and evaluation plans to USAID for  approval. To keep its performance-management system cost-effective,  reduce its burden, and ensure implementing partner activities and USAID  plans are well-aligned, USAID requires its implementing partners to  integrate performance-data collection in their performance-management  activities and work plans. In fulfilling this requirement, the implementing  partners submitted monitoring and evaluation plans for the eight programs  included in our review to USAID for approval. The implementing partners’  plans, among other things, generally contained goals and objectives,  indicators, and targets. However, we found that USAID had not always  approved these plans and did not require implementing partners to set  targets for each of their indicators, which are needed to assess program  performance. Figure 6 shows the number of performance indicators by  fiscal year with targets that the implementing partner developed and  submitted to USAID for approval. The number of indicators with targets  varied over time.
The three programs we reviewed that were active in 2005, identified  indicators and, in some cases, targets in their monitoring and evaluation  plans to track progress; however, according to implementing partners,  USAID did not approve these plans in 2005. Implementing partners for the  eight programs we reviewed were contractually required to submit  monitoring and evaluation plans. According to implementing partners,  USAID was developing a common set of indicators for all three programs to  track. In 2006, all three programs were requested to revise their monitoring  and evaluation plans and develop PMPs that included a common set of  performance indicators. All three programs submitted revised plans in their  2006 PMPs, however, USAID subsequently approved only two out of the three program PMPs. ADP-South’s PMP was never formally approved  during the life of the program. USAID officials were unable to explain or  provide reasons for the lack of approval. In addition, while ADP-South and  ADP-East were intended to end early in fiscal year 2009, when their  contracts were extended into fiscal year 2009, these programs were not  required to set targets for all of their indicators during the additional time  frame. The USAID officials we spoke with were uncertain as to why their  predecessors did not require this of the implementing partners. In addition,  several of the other programs, such as the Afghanistan Water, Agriculture,  and Technology Transfer and the Afghanistan Vouchers for Increased  Production in Agriculture programs did not establish targets for all of their  indicators. As a result, in fiscal year 2009, out of the seven active  agricultural programs we reviewed, two had set targets for all of their  indicators.
section_title: USAID Undertook Efforts to Monitor Agricultural Programs, but Did Not Fully Analyze and Interpret Performance Data
paragraphs: According to USAID’s Automated Directives System, monitoring efforts  should include, among other things, collecting performance data, assessing  data quality, identifying limitations, and taking steps to mitigate data  limitations. USAID regularly collected program reports containing  performance data from implementing partners for the eight programs we  reviewed and assessed data quality, as well as mitigated data limitations, by  conducting site visits when feasible, regularly communicating with  implementing partners, and completing a data quality assessment for  performance data. USAID assigned a monitoring official—known as an  agreement or contracting officer’s technical representative—to oversee  implementing partners’ activities for each of the eight agricultural  programs we reviewed. Monitoring officials identified quarterly reports  submitted by implementing partners as key documents used to collect  performance data. To assess data quality and make efforts to mitigate data  limitations, USAID conducted site visits and documented these efforts by  completing monitoring reports, progress reports, and trip reports.
According to USAID, Afghanistan’s insecure environment limited the  frequency of some site visits and monitoring officials’ ability to consistently  verify reported data. As such, the frequency of site visits varied within and  across programs. Moreover, also according to USAID, formal site visit  reports are seldom completed. As a result of time constraints,  documentation of site visits is often limited to photographic documentation  combined with informal emails from staff participating in site visits. In  2009, USAID conducted site visits for two of the eight programs included in  our review.
In 2008 and 2009, the USAID Mission director cited USAID’s efforts to  monitor project implementation in Afghanistan as a significant deficiency,  in the Mission’s Federal Management Financial Integrity Act of 1982 Annual  Certification. These assessments raised concerns that designated USAID  staff are “prevented from monitoring project implementation in an  adequate manner with the frequency required” and noted that there is a  high degree of potential for fraud, waste, and mismanagement of Mission  resources. USAID further noted that the deficiency in USAID’s efforts to  monitor projects will remain unresolved until the security situation in  Afghanistan improves and stabilizes.
USAID identified several actions to address the limitations to monitoring  project implementation, these include  placement of more staff in the field to improve monitoring capacity,   use of hired security services to provide protection to Mission staff  traveling to project sites,  use of provincial reconstruction team staff to obtain information on the  progress of USAID–funded activities where the provincial  reconstruction teams operate,  use of more Afghan staff, who have greater mobility than expatriate  staff, to monitor projects,  hiring of a contractor to monitor the implementation of construction  projects and conduct regular site visits,   use of Google Earth geospatial mapping to substitute for site visits,  frequent and regular communication with implementing partners,  collection of implementing partner videos or photographs—including   spot checks of implementing partner records or files, and   feedback from Afghan ministries and local officials.
subsections: section_title: USAID Conducted Data Quality Assessments
paragraphs: USAID performance management procedures require that Mission  performance data reported to Washington for Government Performance  and Results Act (GPRA) reporting purposes or for reporting externally on  Agency performance must have had a data quality assessment at some time  within the 3 years before submission. USAID established 10 Mission  agricultural indicators that it reports to the joint Department of State- USAID’s Foreign Assistance Coordination and Tracking System. As  required, USAID completed a data quality assessment for all 10 Mission  agricultural indicators in November 2008. As table 3 shows, USAID’s data  quality assessments generally provided high, medium, or low rankings of  quality for the data collected.
USAID’s assessments also identified actions to mitigate weaknesses in data  quality.
Suggestions for improving data quality included clarifying definitions and  qualifying activities, increasing the frequency of direct monitoring, and  increasing the effort to understand the impact of activities.
section_title: USAID Did Not Consistently Analyze and Interpret Performance Data
paragraphs: In addition to collecting performance data and assessing the data’s quality,  USAID’s Automated Directives System also includes the monitoring  activities of analyzing and interpreting performance data in order to make  program adjustments, inform higher-level decision making and resource  allocation. We found that while USAID collects implementing partner  performance data, or information on targets and results, the agency did not  fully analyze and interpret this performance data for the eight programs in  our review. Some USAID officials in Afghanistan told us that they reviewed  the information reported in implementing partners’ quarterly reports in  efforts to analyze and interpret a program’s performance for the eight  programs in our review, although they could not provide any  documentation of their efforts to analyze and interpret program  performance. Some USAID officials also said that they did not have the  time to fully review the reports. As a result, it is unclear the extent to which  USAID uses performance data to make program adjustments, inform  higher-level decision making and resource allocation.
As previously noted, efforts to monitor program performance as outlined in  USAID’s Automated Directives System should include and document  decisions on how performance data will be used to guide higher-level  decision making and resource allocation. Additionally, USAID is required to  report results to advance organizational learning and demonstrate USAID’s  contribution to overall U.S. government foreign assistance goals. While  USAID did not fully analyze and interpret program data, the Mission does  meet semiannually to examine and document strategic issues and  determine whether the results of USAID-supported agricultural activities  are contributing to progress toward high-level objectives. With respect to  reporting of results, the Mission reported aggregate results in the Foreign  Assistance Coordination and Tracking System—discussed earlier.
section_title: Limited Evaluations Completed for Agricultural Programs GAO Reviewed
paragraphs: USAID’s Automated Directives System requires USAID to undertake at  least one evaluation for each of its high-level objectives; to disseminate the  findings of evaluations; and to use evaluation findings to further  institutional learning, inform current programs, and shape future planning.
As noted earlier, each of the eight agricultural programs included in this  review support the high-level objective of “developing a thriving licit  economy led by the private sector.” In May 2007, USAID initiated an  evaluation covering three of the eight agricultural programs included in our  review—ADP-Northeast, ADP-East, and ADP-South. This evaluation was  intended to assess the progress of the alternative-development initiatives  toward achieving program objectives and offer recommendations for the  coming years. The evaluators found insufficient data to evaluate whether  the programs were meeting objectives and targets and, thus, shifted their  methodology to a qualitative review based on interviews and discussions  with key individuals. As required under USAID’s evaluation requirements,  USAID posted the evaluation to its Web site for dissemination. In addition,  as noted earlier, USAID is planning to conduct evaluations in fiscal year  2010 for all but one of the agricultural programs included in this review.
We are uncertain of the extent to which USAID used the 2007 evaluation to  adapt current programs and plan future programs. Few staff were able to  discuss the evaluation’s findings and recommendations and most noted  that they were not present when the evaluation of the three ADP programs  was completed and, therefore, were not aware of the extent to which  changes were made to the programs. With regard to using lessons learned  to plan future programs, USAID officials told us that, in planning for the  Afghanistan Vouchers for Increased Production in Agriculture program,  key donors met with USAID staff and the Afghan government to share ideas  and lessons learned from other programs. However, the officials could not  provide documentation of this discussion or examples of how programs  were modified as a result of the discussion.
section_title: Several Agricultural Programs Did Not Achieve or Establish Targets
paragraphs: Based on our assessment of USAID implementing partner data, we found  that six of the eight agricultural programs we reviewed fell short of  achieving their established targets for several of their performance  indicators. Additionally, although USAID requires implementing partners to  submit information on indicators, targets, and results, as previously noted,  not all indicators had established targets to allow for performance  assessments. As figure 8 shows, six of the eight programs we reviewed did  not meet their performance targets in the most recent year for which  information was reported on performance targets. For the two programs  that met all their targets, we found, as previously discussed, that they did  not establish targets for several indicators and, thus, we could not fully  assess performance for those indicators. We also found that the three  longest-running programs in our review showed declines in performance  over time. We measured performance for the eight agricultural programs in  our review by comparing annual results against annual targets reported by  USAID’s implementing partners. We assessed the extent to which targets  were fully met. We decided that this measure of performance was  appropriate because implementing partners are allowed to adjust and  revise target levels to better reflect available information in the field. Our  analysis is detailed in appendix IV.
With respect to the three longest-running agricultural programs in our  review—ADP-Northeast, ADP-South, and ADP-East—we found that the  number of indicators that met or exceeded annual targets generally declined from 2006 to 2008. For example, ADP-Northeast met 33 percent  of its targets in fiscal year 2006 and 29 percent of its targets in fiscal year  2008. While ADP-Northeast showed improvements in the percentage of  indicators that met targets between fiscal years 2006 and 2007, the  percentage declined in 2008. Similarly, ADP-South met 79 percent of its  targets in fiscal year 2006, but 50 percent of its targets in fiscal year 2008.  Although ADP-South showed substantial improvements in fiscal year 2009,  the performance assessment was based on 5 out of 25 indicators that had  set targets (see fig. 6 earlier in the report); the remaining 20 indicators  showed results, but did not have annual targets. The Mission noted that  these declines coincided with declines in the security environment;  however, the Mission acknowledged that it had not conducted any analysis  to confirm that the security environment was the reason for the declines in  performance. From 2006 to 2008, the percentage of targets met declined for  indicators, such as the number full-time equivalent jobs; Afghans trained in  business skills; and hectares of improved irrigation as a result of  infrastructure works. Appendix V has details on the indicators, targets, and  results for the latest year where performance data was available for each of  the eight programs in our review. Based on our assessment, on average, the  percentage of targets met declined from 2006 to 2008 across these three  programs.
The longest-running program in our review that is currently active, the  Accelerating Sustainable Agricultural Program, showed improvements in  fiscal year 2008, but declined in fiscal year 2009 in the number of targets  met. For example, the program met 0 percent of its targets in fiscal year  2007, 100 percent of its targets in fiscal year 2008, and 67 percent of its  targets in fiscal year 2009.
As shown in figure 8, trends in targets met could not be determined for the  most recent programs—Afghanistan Water, Agriculture, and Technology  Transfer, ADP-Southwest, Afghanistan Vouchers for Increased Production  in Agriculture, and Incentives Driving Economic Alternatives-North, East  and West—because sufficient data was not available to establish trends. In  addition, as noted earlier, most of these programs failed to establish targets  for all of their indicators and, thus, we could not assess performance for all  indicators. For example, even though recent performance data show that  Afghanistan Water, Agriculture, and Technology Transfer and Afghanistan  Vouchers for Increased Production in Agriculture programs met all targets  for fiscal year 2009, both programs did not set targets for all indicators. In  fiscal year 2009, the Afghanistan Water, Agriculture, and Technology  Transfer program set targets for 3 out of 5 indicators, while Afghanistan  Vouchers for Increased Production in Agriculture set targets for 2 out of 10  indicators. All indicators with targets met or exceeded their annual target.
section_title: USAID Faces Several Challenges in Implementing and Overseeing Agricultural Programs in Afghanistan
paragraphs: The security situation, the Afghan government’s lack of capacity, and  USAID’s difficulties in providing management and staff continuity  challenge the implementation of agricultural programs in Afghanistan. The  security situation hinders USAID’s ability to reach key areas of the country  and monitor programs. Additionally, while the Afghan government’s  capacity to carry out its core functions has improved, key ministries,  including the Ministry of Agriculture, Irrigation, and Livestock—which  works to restore Afghanistan’s licit agricultural economy through  increasing production and productivity, natural resource management,  improved physical infrastructure, and market development—lack the  ability to implement their missions effectively. Finally, USAID’s ability to  maintain institutional knowledge has been hampered by high staff  turnover.
subsections: section_title: Afghanistan’s Security Environment Challenges USAID’s Ability to Oversee and Implement Agricultural Programs
paragraphs: USAID noted difficulties in program oversight and implementation caused  by the challenging security environment in Afghanistan. In November 2009,  we reported that while U.S. and international development projects in  Afghanistan had made some progress, deteriorating security complicated  such efforts to stabilize and rebuild the country. And as we reported in  May 2010, the lack of a secure environment has continued to challenge  reconstruction and development efforts. Specifically, USAID has cited the  security environment in Afghanistan as a severe impediment to its ability to  monitor projects. For example, USAID noted that solely traveling by road  to visit alternative development, food assistance, and environmental  projects in rural areas of northern and eastern Afghanistan is normally not  allowed due to security constraints, and must, consequently, be combined  with some air travel. However, air service in much of the north and east is  limited during the winter months, which has complicated oversight efforts.  Similarly, USAID officials are required to travel with armored vehicles and  armed escorts to visit projects in much of the country. Consequently, as  USAID officials stated, their ability to arrange project visits can become  restricted if military forces cannot provide the necessary vehicles or  escorts because of heightened fighting or other priorities. We experienced  similar restrictions to travel beyond the embassy compound when we  visited Afghanistan in July 2009. For example, we were initially scheduled  to visit agricultural sites in Jalalabad, but could not due to security threats.  Instead, implementing partners traveled to Kabul to meet with us.  According to USAID, limited monitoring has heightened the risk of fraud,  waste, and mismanagement of USAID resources.
In addition to increasing challenges in overseeing programs, the security  environment has also challenged USAID’s ability to implement programs,  increasing implementation times and costs for projects in nonsecure areas.  In particular, U.S. officials cited poor security as having caused delays,  disruptions, and even abandonment of certain reconstruction projects. For  example, according to implementing partner officials, in ADP-Southwest,  some 15 to 20 illegal security checkpoints run by the Taliban and criminals  near major trade centers have increased costs to and endangered the lives  of farmers they support.
USAID predicated the success of its agricultural programs on a stable or  improving security environment. In preparing its 2005-2010 strategic plan,  USAID assumed that security conditions would remain stable enough to  continue reconstruction and development activities. Likewise, several  implementing partner documents included this assumption, and USAID  officials affirmed that this assumption remains true today. Furthermore,  the commander of the North Atlantic Treaty Organization-led International  Security Assistance Force and U.S. forces in Afghanistan testified in his  June 2009 confirmation hearing that improved security was a prerequisite  for the development of local governance and economic growth in  Afghanistan. However, as figure 9 illustrates, while attack levels continue  to fluctuate seasonally, the annual attack “peak” (high point) and “trough”  (low point) for each year since September 2005 have surpassed the peak  and trough, respectively, for the preceding year.
section_title: Limited Afghanistan Ministerial Capacity
paragraphs: USAID has increasingly included and emphasized capacity building among  its programs to address the government of Afghanistan’s lack of capacity to  sustain and maintain many of the programs and projects put in place by  donors. In 2009, USAID rated the capability of 14 of 19 Afghan ministries  and institutions it works with as 1 or 2 on a scale of 5, with 1 representing  the need for substantial assistance across all areas and 5 representing the  ability to perform without assistance. For example, the Ministry of  Agriculture, Irrigation, and Livestock was given a rating of 2—needing  technical assistance to perform all but routine functions—while the  Ministry for Rural Rehabilitation and Development was given a rating of  4—needing little technical assistance. Although USAID has noted overall  improvement among the ministries and institutions in recent years, none  was given a rating of 5. USAID officials noted that a key Afghan official was  recently moved from the Ministry for Rural Rehabilitation and  Development to enhance the Ministry of Agriculture, Irrigation, and  Livestock’s capacity. As a result, USAID officials also said that they have  recently begun to work more closely with Ministry of Agriculture,  Irrigation, and Livestock.
According to the Afghanistan National Development Strategy, Afghanistan’s  capacity problems are exacerbated by government corruption, a significant  and growing problem in the country. Transparency International’s 2009  Corruption Perception Index ranked the country 179 out of 180. Similarly,  in April 2009, USAID published an independent report, Assessment of  Corruption in Afghanistan, that found that corruption, defined as “the  abuse of public position for private gain,” is a significant and growing  problem across Afghanistan that undermines security, development, and  democracy-building objectives. According to USAID’s assessment,  pervasive, entrenched, and systemic corruption is now at an  unprecedented scope in the country’s history. The causes of corruption in  Afghan public administration, according to the Afghanistan National  Development Strategy, can be attributed to a lack of institutional capacity  in public administration, weak legislative and regulatory frameworks,  limited enforcement of laws and regulations, poor and nonmerit-based  qualifications of public officials, low salaries of public servants, and a  dysfunctional justice sector. Furthermore, the sudden influx of donor  money into a system already suffering from poorly regulated procurement  practices increases the risk of corruption and waste of resources. However,  the assessment also noted that Afghanistan has or is developing most of the  institutions needed to combat corruption, but these institutions, like the  rest of the government, are limited by a lack of capacity, rivalries, and poor  integration. The assessment also noted that the Afghan government’s  apparent unwillingness to pursue and prosecute high-level corruption was  particularly problematic.
USAID moved to address this lack of capacity and growing corruption by  including a capacity-building component in its more recent contracts. For  example, the Afghanistan Water, Agriculture, and Technology Transfer  program was designed to, among other things, improve the capabilities of  Afghan ministries and universities by partnering with them on research- based decision making and outreach projects, and to identify water and  land-use policies and institutional frameworks that encourage individuals  and local, provincial, and the national governments to increase sustainable  economic development. Likewise, the Assessment of Corruption in  Afghanistan report noted that “substantial USAID assistance  already  designed to strengthen transparency, accountability, and effectiveness— prime routes to combat corruption—in the most critical functions of  national and subnational government.” For example, the assessment points  to alternative-development and agricultural efforts to create incentives to  not grow poppy, but also notes that these efforts should be coordinated  with enforcement efforts supported by the Departments of Defense,  Justice, and State.
The Administration has further emphasized capacity building by pursuing a  policy of Afghan-led development, or “Afghanization,” to ensure that  Afghans lead efforts to secure and develop their country. At the national  level, the United States plans to channel more of its assistance through the  Afghan government’s core budget. At the field level, the U.S.-assistance  plan is to shift assistance to smaller, more flexible, and faster contract and  grant mechanisms to increase decentralized decision making in the field.  The new U.S. government agricultural strategy, linked to the U.S. effort to  counter insurgency, stresses the importance of increasing the Ministry of  Agriculture, Irrigation, and Livestock’s capacity to deliver services and  promote the private sector and farmers’ associations through direct budget  and technical assistance. However, USAID also recognized that, with the  move toward direct assistance to government ministries, USAID’s  vulnerability to waste and corruption is anticipated to increase. According  to USAID officials, direct budget assistance to the Ministry of Agriculture,  Irrigation, and Livestock is dependent on the ability of the ministry to  demonstrate the capacity to handle the assistance. These officials noted  that an assessment of the Ministry of Agriculture, Irrigation, and Livestock’s ability to manage direct funding was being completed. The  U.S. Embassy has plans under way to establish a unit at the embassy to  receive and program funds on behalf of the Ministry while building the  Ministry’s capacity to manage the funds on its own.
section_title: USAID Faces Challenges to Maintaining Institutional Knowledge
paragraphs: USAID has not taken steps to mitigate challenges to maintaining  institutional knowledge, a problem exacerbated by high staff turnover. As  we noted earlier, USAID did not consistently document decisions and staff  could not always respond to our questions about changes that had taken  place over the life of the programs, often noting that they were not present  at the time of the changes. For example, when we inquired about changes  in the results and performance data reported, USAID officials in  Afghanistan were not able to comment on the performance data or why  changes were made to the data and noted that they were either not present  when the changes took place or were too recently staffed to comment on  performance data reported. Likewise, the Special Representative for  Afghanistan and Pakistan’s staff responsible for drafting the current  agricultural strategy for the United States stated that they could not  effectively discuss USAID program implementation over the last several  years because they were not there and lacked institutional knowledge of  the programs.
We previously reported that USAID and other agencies in Afghanistan lack  enough acquisition and oversight personnel with experience working in  contingency operations. The USAID Mission to Afghanistan has  experienced high staff turnover—USAID personnel are assigned 1-year  assignments with an option to extend their assignment for an additional  year—which USAID acknowledged hampered program design and  implementation. In addition, the State Department Office of Inspector  General noted in its recent inspection of the entire embassy and its staff,  including USAID, that 1-year assignments coupled with multiple rest-and- recuperation breaks limited the development of expertise, contributed to a  lack of continuity, and required a higher number of officers to achieve  strategic goals. For example, the USAID monitoring officials for the eight  programs we examined were in place on average 7.5 months (see table 4).  Moreover, the length of time that a monitoring official is in place has  declined. As of September 2009, the two most recently initiated programs,  the Afghanistan Vouchers for Increased Production in Agriculture program  and the Incentives Driving Economic Alternatives-North, East, and West  program, have had monitoring officials in place for an average of only 3  months each.
USAID has not addressed the need to ensure the preservation of  institutional knowledge. USAID officials noted that the effectiveness of  passing key information from one monitoring official to another, is  dependent on how well the current official has maintained his or her files  and what guidance, if any, is left for their successor. USAID officials noted  that a lack of documentation and knowledge transfer may have contributed  to the loss of institutional knowledge.
section_title: Conclusions
paragraphs: Agricultural development is a key element of U.S. counterinsurgency  efforts in Afghanistan. The United States considers agricultural assistance  a key contribution to Afghanistan’s reconstruction and stabilization. Since  2002, the United States has awarded about $1.4 billion toward agricultural  programs in Afghanistan and plans to invest hundreds of millions of dollars  more.
As such, ensuring sufficient oversight and accountability for development  efforts in Afghanistan takes on particular importance. In addition to relying  on implementing partners to execute its programs, a key part of the U.S.  oversight and accountability efforts involves a reliance on the collection  and analysis of implementing partner data. These implementing partners  are expected to and have reported routinely on the performance of USAID’s  agricultural programs. However, USAID has not always approved  performance indicators established by its implementing partners, allowing  one program to operate for almost 5 years without approved performance  indicators. Additionally, USAID did not ensure that its implementing  partners had established targets for each performance indicator, and  USAID did not consistently analyze and interpret implementing partner  performance data, which is vital to making program adjustments, higher- level decisions, and resource allocation. Without a set of agreed-upon  indicators and targets, and analysis and interpretation of reported  performance data, it becomes more difficult to accurately assess the  performance of USAID agricultural programs. It is also unclear whether or  how USAID has used evaluations to further institutional learning, inform  current programs, and shape future planning. Best management practices  have demonstrated that routine evaluations enable program managers to  identify program vulnerabilities, implement lessons learned, help program  managers understand program weaknesses and make needed  improvements. Moreover, a lack of documentation of key programmatic  decisions and an insufficient method to transfer knowledge to successors  have contributed to the loss of institutional knowledge and the ability of  the U.S. government and others to build on lessons learned. This makes it  more difficult for USAID officials responsible for programmatic decisions,  most of whom are in place for less than a year, to make informed decisions  about shaping current and future programs.
section_title: Recommendations for Executive Action
paragraphs: To enhance the performance management of USAID’s agricultural  programs in Afghanistan, we recommend the Administrator of USAID take  steps to  ensure the approval of implementing partner performance indicators;  ensure that implementing partners establish targets for all performance   consistently analyze and interpret program data, such as determining  the extent to which annual targets are met;   make use of results from evaluations of its agricultural programs; and  address preservation of institutional knowledge.
section_title: Agency Comments and Our Evaluation
paragraphs: USAID provided written comments on a draft of this report. The comments  are reprinted in appendix VI. USAID generally agreed with the report’s  findings, conclusions, and recommendations and described several  initiatives that address elements of the recommendations. In further  discussions with USAID to clarify its response, USAID officials stressed the  challenges involved in working in Afghanistan as a result of the security  environment and working conditions. They submitted additional  documentation, including new guidance on monitoring in high-threat  environments, which was disseminated in December 2009. USAID also  provided technical comments, which we have included throughout this  report as appropriate. We also provided drafts of this report to the  Departments of Agriculture, Defense, and State, all of which declined to  comment.
We are sending copies of this report to interested congressional  committees, USAID, and the Departments of Agriculture, Defense, and  State. In addition, the report will be available at no charge on GAO's Web  site at http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-7331 or johnsoncm@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Key contributors to this report are listed in appendix VII.
section_title: Objectives, Scope, and Methodology
paragraphs: This review assesses (1) how the United States has changed the focus of its  agricultural efforts in Afghanistan since 2002, (2) USAID’s performance  management and evaluation efforts of agricultural programs in  Afghanistan, (3) the extent to which USAID’s agricultural programs in  Afghanistan met targets, and (4) USAID’s efforts to mitigate challenges in  implementing agricultural programs in Afghanistan. In addition, we  analyzed financial information on USAID’s programs in Afghanistan and  reported on the financial status of its agricultural programs.
To assess USAID’s agricultural programs in Afghanistan, including changes  in focus, we met with officials from USAID, the Departments of  Agriculture, Defense, and State—including the Special Representative for  Afghanistan and Pakistan’s office—and implementing partners in  Washington, D.C., and Kabul, Afghanistan. In Kabul, we also met with  officials from the United Nations, the governments of Afghanistan and the  United Kingdom, and a local research group to discuss agricultural efforts.  We traveled to the provinces of Badakhshan and Farah to meet with U.S.  and Afghan officials and discussed various U.S.–funded projects. For  example, in Farah, we met with the local Afghan officials and beneficiaries  of U.S. assistance to discuss the progress of USAID’s agricultural projects,  visited a project site, and met with U.S. contractors implementing the  projects. In Badakhshan, we also met with local officials and beneficiaries  to discuss USAID agricultural efforts and how U.S. assistance was being  used. In addition, we reviewed past GAO work and reports from other  agencies in the U.S. accountability community and nongovernmental  organizations on Afghanistan’s current situation and the challenges it faces.  We reviewed U.S. government documents concerning the U.S. agricultural  strategy and efforts in Afghanistan, as well as USAID funding data.  Beginning in fiscal year 2005, USAID began financing agricultural projects  as part of the U.S. government’s counternarcotics strategy. These were  initially referred to as alternative-livelihood programs, but were later called  alternative-development programs (ADP). To track this strategy over time,  we reported the share of annual obligated and disbursed funds for ADP and  for other agriculture activities. We focused our review on the eight USAID  agricultural programs that were active between 2007 and 2009 and had total  awards of more than $15 million; however, our analysis of financial  information included all USAID agricultural programs. None of the  agricultural programs included in GAO’s review were Office of Transition  Initiatives (OTI) and the Office of U.S. Foreign Disaster Assistance (OFDA)  programs. We did not address agriculture-related projects carried out  independently by other U.S. government agencies, such the Department of  Defense’s Commander’s Emergency Response Program, or those carried  out by multilateral institutions to which the United States contributes, such  as the United Nations Development Programme or World Food Program.
To assess USAID’s performance management and evaluation efforts, we  reviewed the Government Performance and Results Act of 1993 and  pertinent GAO evaluations of performance management practices to  identify best practices. In addition, we examined USAID’s Automated  Directives System and USAID performance management and evaluation  documents to identify the agency’s procedures, requirements, and  guidance. The Government Performance and Results Act and USAID’s  Automated Directives System establish requirements or provide guidance  at a level higher than the program level; the former operates at the agency  level and the latter directs most of its performance management and  evaluations procedures to the bureau or mission level. Nevertheless,  effective planning for results at the agency, mission, and bureau level is a  function of effective planning for results at the program level. We view  results-oriented management tools, including the setting of indicators and  targets, as important at all levels of an agency, including the program level.  Consequently, we determined that the Government Performance and  Results Act criteria, which is operationalized for USAID through its  Automated Directives System, is applicable at this level as well. We  reviewed USAID and implementing partner planning, funding, and  reporting documents for their agricultural programs in Afghanistan, as well  as those addressing evaluations. Our review of these documents provided  us with information regarding the programs’ performance management  structure, goals, objectives, indicators, and targets. We examined these and  other documents to determine the extent to which the Mission followed  requirements, guidance, and best practices.
To assess the extent to which performance was achieved, we reviewed all  quarterly and annual reports, implementing partner PMPs, and annual work  plans for the eight agricultural programs under review. The data were  primarily compiled from implementing partner quarterly reports from April  2005 though September 2009. When data were not available, we used the  PMPs and other documents to fill in gaps. USAID could not provide all the  needed documents; we, therefore, requested missing documents from the  implementing partners. To determine the validity and reliability of the data  reported in the quarterly reports, we requested USAID’s completed data  quality assessments. We received one USAID data quality assessment  completed for all agricultural programs from November 2008, and one  program data quality assessment completed by USAID monitoring officials,  also from November 2008. We also checked the data for inconsistencies  and questioned USAID officials and implementing partners about any  inconsistencies. We found the data in the quarterly reports to be  sufficiently reliable for our purposes.
The data collected were organized in a spreadsheet on a quarterly basis for  the eight programs. To track program performance over time, we collected  all reported quantitative data on indicators, targets, and results. For each  reported indicator, we measured performance achieved as the ratio of  results against established targets. We found that implementing partners  were inconsistent in reporting on targets, by either not setting targets, or in  two cases, retroactively setting or revising targets. In addition, two  programs shifted reporting time frames between fiscal and calendar years.  Some of the changes resulted from implementing partner audits or a USAID  Regional Inspector General audit of the data collected and reported to  USAID. Although USAID encourages and permits changes to targets over  the life of a program in response to new information, these factors  complicated attempts to determine performance. Furthermore, in some  cases, in addition to targets, the results were also updated retroactively. We  captured changes in target levels and reported results by inserting  additional lines in a spreadsheet. This process allowed us to determine  changes in targets, results, and performance achieved over time.
In general, data reported in quarterly reports were presented cumulatively;  however, we found this presentation masked performance achieved in a  specific year. Therefore, once all cumulative data were entered into the  spreadsheet, we calculated the numbers to show annual targets, results,  and performance. The performance data collected were categorized into  eight categories: (1) met or exceeded target, (2) achieved 76 to 99 percent  of target set, (3) achieved 51 to 75 percent of target set, (4) achieved 26 to  50 percent of target set, (5) achieved 1 to 25 percent of target set, (6)  achieved zero progress toward target set, (7) number of indicators used to  assess performance, and (8) no target set. Based on the categorical  assessment, we were able to determine the number of indicators reported  annually and over the life of the program in each of the categories noted  above. We are reporting program performance achievements on the annual  percentage of indicators that met or exceeded the target. For example, if  there were 15 indicators and 9 indicators had met or exceeded the target,  than annual program performance was 60 percent (9/15). We decided this  measure of performance was appropriate because implementing partners  are allowed to adjust and revise target levels to better reflect available  information in the field. Further, we found that the percentage of indicators  meeting their targets could increase or decrease for a variety of reasons,  including changes in measures, the types of measures or the targets set, as  well as changes in actual underlying performance. A review of all those  factors was beyond the scope of this report.
To examine the challenges faced by agricultural efforts in Afghanistan, we  reviewed U.S. strategy documents and USAID documents addressing the  status of and challenges faced by U.S. efforts in Afghanistan, including  security, Afghan capacity and corruption, and USAID staffing and  workspace concerns. We also reviewed Department of Defense documents  on counterinsurgency strategy and joint diplomatic-military plans. We  updated attack data on which we had previously reported. We assessed the  data and found them to be sufficiently reliable for our purposes. We  reviewed the Afghan government and nongovernmental organization  reports regarding capacity and corruption in Afghanistan. To compile a list  of USAID monitoring officials, we reviewed the names listed in data USAID  provided.
To report financial information on USAID agriculture programs and  individual projects, we used financial information from the “Pipeline  Report” generated from USAID’s Phoenix financial management  information system provided to us by the Office of Financial Management  at the USAID Mission to Afghanistan. This report contains cumulative  financial information for individual projects that may be funded through a  contract, cooperative agreement, or grant. We received pipeline reports for  all USAID projects in Afghanistan as of the end of fiscal years 2004-2009  and March 31, 2010, and for selected agriculture projects, we received  quarterly pipeline reports from fiscal years 2005-2009. The March 31, 2010,  pipeline report contains financial information on USAID projects in  Afghanistan from fiscal year 2002 to March 31, 2010, for 254 projects of  which 41 were agriculture projects. We also checked the data for  inconsistencies and questioned USAID officials about any inconsistencies.  To describe the financial status of the USAID agriculture program, we used  three financing concepts: award, unliquidated obligation, and  disbursement. These are related to, but not exactly the same as budget  concepts. Award refers to the dollar amount of the award in a signed  contract, cooperative agreement, or grant. The signed document indicates  the period of time over which the project is expected to be implemented.  The amount of the award, the time frame, and other elements of the  contractual agreement may be changed through a formal amendment  process. Unliquidated obligations represent the current amount of  obligations remaining to be disbursed. Disbursements are those funds that  have been released from the U.S. Treasury. Cumulative obligations is the  total of unliquidated obligations plus disbursements.
The e-supplement presents information on Afghanistan for select  indicators, including poverty rates by province and region, and the number  of households that produced crops during the summer and winter seasons  during 2007/2008. The information comes from supplementary tables to a  Government of Afghanistan household survey, National Risk and  Vulnerability Assessment 2007/8, http://nrva.cso.gov.af/index.html,  released online in January 2010. The survey covered the period from  September 2007 through August 2008, and was conducted with the  financial and technical assistance of the European Commission and several  other organizations.
The report groups contiguous provinces into 8 regions. For each of the 34  provinces and 8 regions we report the population and compute the share of  population that has access to land, has access to safe drinking water, is  literate (ages 15 years and over), is urban, has access to electricity, has  access to public health facilities (1 hour or less by foot), and owns  livestock. The poverty rate is the share of population living on less than a  minimum level of food and nonfood (for example, shelter) consumption. In  the map, provinces are categorized into 5 groups from high to low ranges of  poverty rates. These are from a World Bank analysis of the assessment’s  data. These are ranges of poverty rates and not confidence intervals. The  poverty rate is only presented for the regions, a national average for the  year and for seasons of the year.
For a list of 25 crops, including other crops, the report lists the number of  households that produce each crop during the summer and winter seasons  on irrigated land. Since more than one crop may be grown each season by a  farmer, the assessment reports the number of households that report each  crop as its primary, second, or third crop. For example, during the summer  season, 1,349,200 households cultivate a primary crop, 889,200 households  (66 percent) cultivate a second crop, and 448,700 households (33 percent)  produce a third crop. For each crop, we computed the number of  households that reported producing that crop, regardless whether as a  primary, second, or third crop, for each season. We also provide tables  showing the number of households that report producing crops as the  primary, second, or third crop, for each season. There is no information on  the number of households that produce different crops on rain-fed land.  The assessment reports that 591,000 households have access to rain-fed  land, but some of these households may also have access to irrigated land.
We reviewed the survey methodology of National Risk and Vulnerability  Assessment 2007/8 report and found it to be sufficiently sound,  particularly given the challenging environment in which the data were  collected and the potentially sensitive nature of the questionnaire topics.  That said, we were not able to ascertain information on some aspects of the  survey, which would have helped shed light on its quality. A survey’s design  can be judged by its success or failure in minimizing the following types of  errors.
Sampling error. The report provides no information on the precision of  its estimates. Usually, this is expressed in a confidence interval. We  cannot, therefore, judge the reliability of the point estimates.
Nonresponse error. The report mentions a process of reserve  replacement sampled households in the event of a noncontact.  However, no data is kept on how frequently these were used, so it is not  possible to calculate a response rate. Officials reported that this  replacement rate was low.
Coverage error. Sixty-eight of the 2441 primary sampling units were  replaced, mostly due to security concerns. Such coverage errors could  lead to a coverage bias if those covered are categorically different from  those not covered with respect to variables of interest.
Measurement error. While most of the questionnaire is not of a sensitive  nature, we have to be aware that farmers might not be completely  honest with a government interviewer when it comes to the cultivation  of illicit crops. As such, our assessment was based only on information  that was made available about the survey methodology.
We conducted this performance audit from March 2009 through July 2010  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that the  evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Funding Breakout of USAID Agricultural Programs in Afghanistan, as of the End of Fiscal Year 2009
paragraphs: Summary financial information about the eight agriculture programs  discussed in our report is reported in table 5. Because we selected  relatively large programs, the total amount of funds that USAID planned to  spend on these eight programs over $1 billion or 75 percent of the total  awards for all 41 agriculture programs. One program, the Afghanistan  Vouchers for Increased Production in Agriculture program, accounts for 26  percent of all agricultural awards. As of the end of fiscal year 2009, these  eight programs accounted for 69 percent of total USAID agricultural  assistance disbursements, with one program accounting for 18 percent of  all disbursements.
section_title: USAID Agricultural Program, by Province, and Examples of Activities
paragraphs: The following figure shows the provinces in which the eight agricultural  programs we reviewed were active and an example of the types of activities  they undertook.
section_title: GAO Analysis of Implementing Partner Performance Data
paragraphs: The following tables provide information on the annual number of  indicators that fell into one of the eight percentage categories.  Performance is measured by comparing results against targets. We  assessed annual performance—the number of indicators that met or  exceeded their target—for each of the eight programs to highlight  improvements and declines in performance that took place in a given fiscal  year. As shown below, the data collected were organized into eight  categories (1) met or exceeded target, (2) achieved 76 to 99 percent of  target set, (3) achieved 51 to 75 percent of target set, (4) achieved 26 to 50  percent of target set, (5) achieved 1 to 25 percent of target set, (6) achieved  zero progress toward target set, (7) number of indicators used to assess  performance, and (8) no target set. Based on the categorical assessment,  we were able to determine the number of indicators that fell into one of the  eight categories, the number of indicators with target levels reported, and  the total number of indicators reported. For each fiscal year, the first  column represents the number of indicators whose performance fell within  an indicated range. The second column is a percentage of total number of  indicators (with and without targets) that fell within an indicated range.  Please note this percentage is based on total number of indicators tracked,  and may differ from those in figure 8, which is based on total number of  indicators with targets.
section_title: Eight USAID Agricultural Programs, Annual Targets, Results, and Percentage of Targets Met, for the Latest Year
paragraphs: The tables below provide information on annual targets, results, and  percentage of each target met for the eight programs we reviewed. The  data provided is based on the latest year where performance data was  available.
section_title: Comments from the U.S. Agency for International Development
paragraphs: The following are GAO’s comments on the U.S. Agency for International  Development’s (USAID) letter dated July 1, 2010.
subsections: section_title: GAO Comments
paragraphs: 1. GAO modified the report to reflect the new Automated Directives  System Guidance on Monitoring in High Threat Environments.  However, GAO would note that the Mission to Afghanistan that GAO  was directed to for all inquiries was not aware of the December 2009  guidance until June 2010. In addition our application of the Automated  Directives System criteria was consistent with the new guidance and  required only minor technical revisions.  2. None of the agricultural programs included in GAO’s review were  Office of Transition Initiatives (OTI) or Office of U.S. Foreign Disaster  Assistance (OFDA) programs. 3. USAID’s Automated Directives System generally requires the same  procedures for a conflict zone that it requires elsewhere or else  documentation specifically describing when those procedures are not  followed. The new guidance does not provide any exemptions with  regard to approval of monitoring and evaluation plans and the  establishment of indicators and targets, which USAID did not  consistently approve. 4. Without approved indicator targets, it is unclear how performance can  be reviewed or assessed. 5. GAO acknowledges that USAID is currently developing a PMP on   page 14 of the report. 6. At the time of our review, USAID had completed a one midterm  evaluation covering three of the eight programs we reviewed. USAID  staff, however, were unable to indicate how the findings of the  evaluation were used to inform the design of subsequent programs.  Additionally, the midterm evaluation included recommendations for  improving the three programs, but USAID staff were unable to  comment on how the recommendations of the evaluation were  implemented. 7. GAO removed its mention of the semiannual reviews from the  recommendation based on additional information provided.
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, the following staff contributed to  the report: Hynek Kalkus, Assistant Director; Thomas Costa; Farahnaaz  Khakoo; Bruce Kutnick; Sara Olds; Steven Banovac; Joseph Carney;  Elizabeth Curda; Mark Dowling; Gena Evans; Etana Finkler; Justin Fisher;  Cindy Gilbert; Gloria Mahnad; Kara Marshall; Jackie Nowicki; Sheila  Rajabiun; and Jena Sinkfield.