Foreign Aid: USAID Has Increased Funding to Partner-Country Organizations but Could Better Track Progress
Released: 2014-04-16T00:00:00

Letter

Background

Since announcing the USAID Forward initiative in 2010, USAID has updated policies and procedures to guide the initiative, notably the following: In 2011, USAID established procedures for determining the suitability of using partner-country government systems to implement USAID- funded programs.

In 2011 and 2012, USAID also updated its guidance and procedures for program planning and design, monitoring and evaluation plans, and evaluation standards for all new programs, including those implemented through local organizations.

In March 2013, USAID issued a progress report on USAID Forward that provided information on the status of the Local Solutions initiative, among other things. The report included agency-wide percentages of funding obligated to local organizations for fiscal years 2010 through 2012, as well as disaggregated 2012 figures by region, with a break-out for Afghanistan and Pakistan together, and by sector—for example, health, agriculture, and democracy and governance. Subsequently, in October 2013, USAID released fiscal year 2012 funding data used to report on the principal indicator for tracking and reporting on implementation of the Local Solutions initiative. Just prior to publication of this report, USAID published fiscal year 2013 data for this indicator on its website. USAID also provides updated information on the status of the Local Solutions initiative through its annual performance plans and reports issued jointly with the Department of State, as well as through quarterly and annual reporting to the Office of Management and Budget on agency priority goals.

In 2013, USAID filled the newly created position of Local Solutions Coordinator in the agency’s Counselors office. According to USAID, the Local Solutions Coordinator is responsible for coordinating the activities— such as policy, program, legal, financial, and procurement—of the various headquarters offices and missions involved in carrying out the Local Solutions initiative.

USAID Increased Local Funding, but Extent of Progress toward Its 30 Percent Target Is Not Clear

USAID has reported the funds tracked toward its 30 percent target in different ways, and this inconsistent reporting complicates the assessment of the degree to which USAID is making progress. On the basis of information provided in its 2013 progress report and other data provided by USAID, agency funds obligated to local organizations increased from about 10 to 14 percent of mission program funds in fiscal years 2010 through 2012—a $465 million increase. However, depending on which funds are counted toward the principal Local Solutions indicator, USAID’s progress toward its 30 percent target ranged from the 14 percent reported in its 2013 progress report to as high as 24 percent of mission program funds obligated to local organizations, as noted in data published on the agency’s website. In addition, in fiscal year 2012, including all cash transfers and trust funds, 5 missions, including those in Afghanistan and Pakistan, obligated about 79 percent of all USAID funding to local organizations, while 72 missions obligated the remaining 21 percent.

USAID Funding to Local Organizations Increased in Fiscal Years 2010 through 2012, but Indicator Reporting Lacks Clarity

USAID has reported on its principal Local Solutions indicator with and without cash transfers and before and after the addition of qualifying trust funds in two missions. According to USAID, funds included in the calculation of this indicator include all mission program funds to local organizations that directly implement programs and include loan guarantees to local banks, cash transfers, and contributions to trust funds that are managed by a government department or ministry. USAID’s reporting of the indicator, however, varied in the inclusion of some of these funds. In its 2013 progress report, USAID reported an increase in obligations to local organizations during fiscal years 2010 through 2012— from about 10 to 14 percent of mission program funds. According to data provided by USAID, this represents a $465 million increase, excluding cash transfers and additional qualifying trust funds to two missions that USAID later determined qualify as investments in local organizations. Including cash transfers, the percentage of mission program funds obligated to local organizations increased from about 14 percent to 17 percent from fiscal year 2010 to 2012. Subsequently, in October 2013, USAID published on its website updates showing fiscal year 2012 dollar amounts and percentages of mission program funds obligated to local organizations both including and excluding cash transfers. In this update, USAID noted contributions to certain trust funds that USAID had determined met the agency’s definition of local funding after the release of the 2013 progress report. As shown in figure 1, depending on whether cash transfers and these qualifying trust funds are counted toward the indicator, USAID achieved fiscal year 2012 levels of funds obligated to local organizations that are between 14 percent and 24 percent. However, USAID has not determined which indicator definition it intends to use. Differences in USAID’s reporting on cash transfers and qualifying trust funds make it difficult to compare the indicator from year to year and quantify the progress needed to achieve its 30 percent target.

Cash transfers. In its March 2013 progress report and data released in October 2013, USAID reported fiscal year 2012 dollar amounts and percentages of mission program funds obligated to local organizations, both including and excluding cash transfers. For all 3 fiscal years for which data are available, if cash transfers are included in the indicator definition, the overall percentage of funds obligated to local organizations increases. As of fiscal year 2012, according to USAID data, the agency obligated about 14 percent of mission program funds, or about $1.4 billion, to local organizations, excluding cash transfers. With cash transfers included, the agency obligated more than 17 percent, or about $1.7 billion in fiscal year 2012.

Qualifying trust funds. According to USAID officials, after publication of the March 2013 progress report, USAID determined that contributions to certain trust funds made by two missions met the The addition of these qualifying agency’s definition of local funding. trust funds increases the agency-wide percentage of local funding for fiscal year 2012 from about 14 percent to about 21 percent. According to USAID officials, the agency plans to include these qualifying trust fund contributions in its future tracking and reporting of local mission program funding because they meet the agency’s definition of local funding.

Cash transfers and qualifying trust funds. When both cash transfers and all qualifying trust funds are included in the indicator definition, the agency-wide percentage of local funding rises to about 24 percent in fiscal year 2012.

The October 2013 data published by USAID on its website reflect the agency’s determination that about $29 million and $671 million in trust fund contributions made by USAID missions in Bangladesh and Afghanistan, respectively, met the definition of local funding. meet the Local Solutions definition of local funding and determine whether to include it.

Other Changes in Local Solutions Initiative Tracking

USAID also uses this indicator to report on the agency’s priority goals related to procurement reform that it established in response to requirements under the Government Performance and Results Act Modernization Act of 2010. USAID’s report on its priority goals related to procurement reform defines the indicator as the percentage of program funds implemented through local systems. See http://www.performance.gov for additional information.

Afghanistan and Pakistan because these missions represent key efforts to implement programs through local organizations. If Afghanistan and Pakistan are excluded from the agency-wide indicator, the percentage of mission program funds obligated to local organizations in fiscal year 2012, including qualifying trust funds and cash transfers, drops from about 24 percent to about 14 percent.

USAID’s Local Funding Is Concentrated in a Few Countries

On the basis of our analysis of the fiscal year 2012 data provided by USAID, we found that funds obligated to local organizations are heavily concentrated in a small number of countries—including Afghanistan and Pakistan. With cash transfers and all qualifying trust funds, five USAID missions—Afghanistan, Pakistan, Jordan, South Africa, and Uganda— accounted for 79 percent of all funding obligated to local organizations in fiscal year 2012. The remaining 21 percent of all fiscal year 2012 funding obligated to local organizations was attributed to USAID’s other 72 missions (see fig. 2).

USAID Cannot Demonstrate Progress toward Local Solutions Goals Based on Its Principal Indicator Alone and Without Tracking Relevant Evaluations

USAID’s principal Local Solutions indicator does not fully reflect activities the agency carries out to achieve the initiative’s goals of strengthening partner-country capacity, enhancing and promoting country ownership, and increasing program sustainability. Moreover, USAID does not have a means for determining the extent to which missions are carrying out evaluations that can provide evidence about the long-term effectiveness of the initiative.

Principal Indicator Does Not Fully Capture Efforts to Implement the Local Solutions Initiative

USAID’s principal indicator of progress under the Local Solutions initiative—the percentage of mission program funds obligated to local organizations—does not fully capture the activities the agency undertakes to achieve the goals of the initiative. For example, to achieve USAID’s stated goals under the initiative, according to USAID, the agency has put in place frameworks for all of the following activities, among others: assessing partner-country government and nongovernmental organizations’ strengths and weaknesses, developing implementing mechanisms to mitigate risks and ensure providing technical assistance, and carrying out monitoring and evaluation plans.

In its 2013 progress report, the agency provides information related to the status of some of these various activities, such as numbers of assessments completed and awards made to partner-country government and nongovernmental organizations.report and other reports, the agency emphasizes increases in its Local Solutions funding indicator to demonstrate progress. According to USAID officials, that indicator is likely to remain the primary indicator used for reporting progress on the Local Solutions initiative.

However, in the 2013 progress By tracking obligated funds, the indicator indirectly measures—but to a limited degree—the success with which USAID missions have proceeded through required risk assessment and project design steps. First, in the case of funding to partner-country governments, obligation of funds occurs at the time of agreement—stage 4 under USAID’s five-stage process for using country systems to implement programs. However, according to USAID, because missions move carefully through the project design phase, some missions have not yet obligated funds to partner governments. As a result, the Local Solutions indicator does not capture these missions’ efforts to assess and mitigate risk, including through selection of implementing mechanism, as well as build capacity prior to obligation of funds. Second, the indicator does not reflect the status of reassessment activities in countries where USAID obligated funding to local organizations prior to 2011, particularly in Afghanistan and Pakistan.of mission activities occurring after the obligation of funds, including monitoring and evaluation of program implementation. According to USAID officials, agency headquarters does not require missions to report this information.

Finally, the indicator provides no information about the status Given these limitations, an emphasis on the funding indicator hampers the ability of USAID program managers and external stakeholders to fully understand whether and to what extent progress is being made toward the initiative’s overarching goals. The 2013 progress report states that USAID will revisit its indicators to ensure that they fully reflect USAID’s efforts to achieve the goal of sustainable development.USAID officials have characterized the indicator as a proxy for measuring progress in using the Local Solutions approach to achieve sustainable development goals; officials further stated that the indicator is just one of several approaches to measuring progress toward strengthening partner- country systems and promoting sustainable development. For example, according to USAID, program managers in the field track specific indicator data against outcomes specified in award agreements; in addition, evaluation data provide information on progress toward outcomes. Nevertheless, according to USAID officials, its funding indicator is likely to remain the primary indicator used for reporting progress on the Local Solutions initiative.

Steps Taken by USAID to Evaluate the Local Solutions Initiative Do Not Include the Means to Track Relevant Evaluations

USAID has laid some groundwork for evaluating the Local Solutions initiative but cannot currently determine the extent to which missions are planning or conducting evaluations the agency deems appropriate for understanding the long-term effectiveness of the initiative. USAID has taken the following steps to be able to evaluate the Local Solutions initiative:

Assistance to missions. According to USAID, headquarters staff provide assistance to missions in designing projects and identifying indicators for measuring capacity development and other outcomes, among other things. The missions are responsible for designing projects and carrying out monitoring and evaluation plans.

Evaluations and studies. USAID compiled evaluations of completed capacity-building projects and commissioned studies and background papers. For example, one study commissioned by USAID noted that emphasis on using country systems stems from an experience-based consensus that project-centered approaches have undermined efforts to support partner-government capacity. Furthermore, the study concluded that increasing funding to local organizations is associated with improved capacity of partner governments in some countries, but the study also noted that evidence demonstrating impacts relative to other approaches is limited.

Stakeholder meeting. In November 2012, USAID organized a meeting of international development stakeholders to gather lessons learned from past USAID and other donors’ efforts to build partner- country capacity and ensure sustainability.

Draft framework. The studies and the stakeholder meeting led to publication, in October 2013, of USAID’s draft framework for supporting sustainability; the agency solicited feedback on this draft and plans to issue a final version in the spring of 2014.

According to USAID, the agency has determined that impact evaluations of the Local Solutions initiative are not feasible because of (1) the lack of a valid counterfactual, or control group for the purposes of comparison, and (2) the need for flexibility and adaptability in implementing multiyear programs in complex environments. Senior USAID officials also cited the level of difficulty involved and challenges associated with involving partner-country governments in program and project planning cycles, among other things, as reasons why impact evaluations are not appropriate. Furthermore, these officials told us that performance evaluations of programs or projects using a Local Solutions approach—those involving local organizations as implementers—are both feasible and appropriate. However, headquarters officials were not able to determine the extent to which missions have undertaken performance evaluations of programs or projects using a Local Solutions approach since the launch of the initiative.

USAID’s Evaluation Policy requires missions to provide information about planned and completed evaluations on USAID’s public website. example of evaluations that are relevant for the Local Solutions initiative, USAID officials stated that headquarters will undertake at least one multicountry evaluation of long-term effects of basic education programs in fiscal year 2014, and some of those programs may have used local partners for implementation. As of April 2014, the agency’s website allows users to search for evaluations by document type (e.g., evaluation summary or program evaluation guide), by sector (e.g., health, education, democracy and governance), and by year, among other things. However, the online registry of evaluations does not provide users with a means to search for evaluations of programs or projects implemented by local organizations. Without a means to track evaluations of programs or projects that use a Local Solutions approach, USAID headquarters officials and stakeholders are not able to easily obtain evidence needed to demonstrate Local Solutions’ progress in achieving USAID’s Local Solutions goals related to country capacity, country ownership, and program sustainability.

According to USAID, the agency’s 2013 progress report constitutes a Local Solutions evaluation plan; however, the report does not include plans for evaluating programs or projects implemented through local organizations.

Conclusions

With its Local Solutions initiative, USAID has undertaken an ambitious change in how it administers development assistance, a fundamental shift that is consistent with an international consensus on how best to improve global development effectiveness and accountability. By issuing a progress report on its reform efforts, publishing data on Local Solutions funding, and presenting research on best practices, among other things, USAID has demonstrated a commitment to transparency and public dialogue regarding the initiative. Nevertheless, as USAID expands the Local Solutions initiative, it relies on a single funding indicator to measure progress. The agency has had difficulty deciding how best to define this indicator and report on it consistently; as a consequence, comparing annual results for this indicator and accurately measuring progress is unnecessarily complex. In addition, in relying on this funding indicator to such an extent, USAID does not capture important progress taking place as the agency adopts the Local Solutions funding approach, such as missions’ efforts to assess and mitigate risk and carry out monitoring and evaluation plans, among other things. Finally, though USAID policy states that evaluation is a crucial tool for informing its development efforts, USAID lacks a means to identify evaluations of Local Solutions efforts from among the evaluations that USAID collects and makes available on its website. Without such evaluations, agency officials and stakeholders are prevented from bringing to bear rigorous evidence of the initiative’s long-term effects on local capacity, country ownership, and program sustainability.

Recommendations for Executive Action

We recommend that the USAID Administrator take the following three actions to improve the agency’s tracking and reporting on progress of its Local Solutions initiative:

Clarify in future reporting the types of funding included in the percentage of USAID funds obligated to partner-country local organizations.

Identify additional indicators to better capture Local Solutions progress toward the initiative’s goals.

Provide a means to identify evaluations of programs that are implemented by partner-country local organizations.

Agency Comments and Our Evaluation

We provided a draft of this report to USAID for review and comment. USAID provided technical comments on the draft, which we incorporated as appropriate. USAID also provided written comments, which are reprinted in appendix III. In its written comments, USAID did not explicitly agree or disagree with our recommendations but described efforts underway to consistently report on the Local Solutions initiative, improve project-level monitoring, and track relevant evaluations.

With regard to our first recommendation to clarify the types of funding included in the percentage of USAID funds obligated to partner-country organizations, USAID stated that it has always been its intention to include, in calculating the indicator, every award that meets the definition of “local.” It noted that over time USAID has learned from its experience and has broadened the types of implementing mechanisms that qualify in meeting the definition of “local” used in the indicator. USAID’s explanation is consistent with our finding. While we understand USAID’s position on the circumstances that led to its reporting on the primary Local Solutions indicator in different ways, we believe that further actions are needed. In this context, we note that the fiscal year 2013 dataset USAID published on its website just prior to publication of this report represents a step toward the clarity we recommended the agency provide in its reporting on its primary Local Solutions indicator.

With regard to our second recommendation that USAID identify additional indicators to better capture Local Solutions progress, USAID stated that it does not plan to develop additional agency-wide indicators for Local Solutions. USAID noted that each project involving local development partners varies greatly depending on project objectives, country context, and local partner strengths and challenges; hence, indicators in one country, region, or sector could be irrelevant in another. USAID stated that while it is not pursuing development of common indicators beyond the current agency-wide Local Solutions indicator, it has efforts under way to train USAID staff in effective project design and performance monitoring, including the identification of meaningful and useful indicators to measure project objectives. In addition, it cited a USAID Forward reform that mandates that each mission thoughtfully consider indicators to measure progress toward mission objectives and noted that it is working on providing more guidance to USAID staff on Local Solutions-focused performance monitoring approaches (including indicators) and evaluation questions. We continue to believe that further steps to implement our recommendation are needed. For example, the procedures USAID missions are required to follow to implement programs using a Local Solutions approach—such as those noted in USAID’s response, among others—are examples of concrete activities that could be tracked to reflect agency-wide progress. We maintain that even a relatively modest expansion beyond the one principal Local Solutions indicator would supplement USAID’s reporting on progress in achieving the initiative’s goals.

With regard to our third recommendation that USAID provide a means to identify evaluations of programs implemented by partner-country local organizations, USAID stated that it is introducing a tag in its evaluation registry to track the evaluation of projects that are directly implemented through local partners. In its response to this recommendation, USAID also described plans to start conducting, in the current fiscal year, priority evaluations addressing questions of local systems change, and to commission a multicountry program evaluation in fiscal year 2016 on the effectiveness and efficiency of Local Solutions interventions that use direct funding. We believe that these actions, when implemented, will address the intent of our recommendation.

We are sending copies of this report to appropriate congressional committees and the USAID Administrator. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

The U.S. Agency for International Development’s (USAID) principal indicator for its Local Solutions initiative is the percentage of mission program funds implemented through local organizations, which the agency tracks at the time of obligation. USAID established an agency- wide target of obligating 30 percent of mission program funds to local organizations by fiscal year 2015 (referred to hereafter as the agency’s “30 percent target”). The objectives of this report were to assess the extent to which USAID (1) has demonstrated progress toward achieving its 30 percent target, and (2) is tracking progress in achieving the Local Solutions goals of strengthening partner-country capacity to implement programs, enhancing and promoting country ownership, and increasing sustainability.

To address these objectives, we reviewed USAID Local Solutions initiative documents, including the March 2013 USAID Forward progress report and agency documents published at the launch of the initiative that explain the rationale, baseline figures, and targets for the initiative. We also analyzed fiscal years 2010, 2011, and 2012 data on USAID mission program funds obligated to local organizations, which agency officials had extracted from USAID’s financial system. On the basis of our review of the data and interviews with agency officials, we determined that the data were sufficiently reliable for our reporting purposes. In this report, we note that USAID has reported on the Local Solutions indicator with and without the inclusion of cash transfers and additional qualifying trust funds USAID identified after the release of the 2013 progress report. We chose to present our analyses of the USAID data with the inclusion of cash transfers and qualifying trust funds in our calculations—representing the maximum amounts USAID obligated to local organizations. We note, where appropriate, how these figures change when we exclude these funds—representing the minimum amounts. At the time of our review, fiscal year 2013 data had not yet been completely validated and thus were not available for our analysis. We interviewed USAID officials in Washington, D.C., about the Local Solutions initiative and the supporting data, and in some cases, received written responses to our questions. In assessing the Local Solutions indicator to identify relative strengths and weaknesses, we reviewed a prior GAO report that identified nine attributes of performance goals and measures based on previously established GAO criteria.

In addition, we reviewed aid effectiveness literature on providing assistance directly through local organizations, including studies and other publications collected, commissioned, or conducted by USAID. We reviewed international donor commitments, such as the Paris Declaration and Accra Agenda for Action, which were intended, in part, to promote donor consensus on principles related to recipient country ownership. We reviewed USAID policy and guidance, including the agency’s evaluation policy and relevant sections of the Automated Directives System. We sent USAID officials in Washington, D.C., several sets of questions and either interviewed them in person or received written responses.

We conducted this performance audit from June 2013 to April 2014 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO Analysis of USAID Local Solutions Data for Fiscal Years 2010 through 2012

In fiscal year 2012, USAID obligated about $2.4 billion, including cash transfers and all qualifying trust funds, to local organizations. Of this $2.4 billion, USAID obligated $10 million or less through 56 missions, $10 to $20 million through 10 missions, and over $50 million through 5 missions. See figure 3 for the complete distribution of the funding amounts USAID missions obligated to local organizations in fiscal year 2012.

Of the funds that USAID missions obligated to local organizations in fiscal year 2012, the majority—about 71 percent—were obligated to partner- country governments. The remaining funds were obligated to various local organizations as follows: 20 percent to local nonprofit organizations, 6 percent to local for-profit organizations, and 2 percent to local educational institutions. See figure 4 for a breakdown of USAID funds obligated to local organizations by organization category. For each category, we also list the amount of funds obligated by the five missions obligating the greatest amounts of funds in each local organization category.

In fiscal year 2010, USAID missions obligated 71 percent of local mission program funds to partner-country governments and 29 percent to local nonprofit and for-profit organizations and educational institutions. In fiscal year 2011, USAID mission obligations to these organizations were 63 percent and 37 percent, respectively (see fig. 5). USAID has not fully validated their fiscal year 2010 and 2011 data for accuracy regarding the breakdown of the mission program funds obligated to local nonprofit and for-profit organizations and educational institutions; hence, we did not calculate further breakdowns for these years.

Appendix III: Comments from the U.S. Agency for International Development

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

David Gootnick, (202) 512-3149 or gootnickd@gao.gov.

Staff Acknowledgments

In addition to the contact named above, Jim Michels (Assistant Director), Todd M. Anderson, David Dayton, Martin De Alteriis, Etana Finkler, Grace Lui, Shirley Min, Kathleen Monahan, and John O’Trakoun made key contributions to this report. In addition, Heather Latta provided technical assistance and other support.