Public Financial Management: Improvements Needed in USAID's and Treasury's Monitoring and Evaluation Efforts
Released: 2012-09-13T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: Effective management of public resources can play an important role in a country’s development. In recent years, developing countries committed to strengthen their PFM systems and donors committed to use those systems as much as possible. The United States provides assistance to strengthen PFM systems primarily through USAID and Treasury. USAID conducts capacity building activities to strengthen PFM systems as part of its development programs and has also set a target to obligate 30 percent of its annual assistance through local systems by 2015. Treasury provides technical assistance through advisors who work in country, typically with the finance ministry.

GAO was asked to examine the processes U.S. agencies use to (1) develop programs to strengthen PFM systems and (2) monitor and evaluate those programs. GAO reviewed agency guidance and program documents, interviewed U.S. agency officials, and selected case studies to serve as illustrative examples of PFM-related programs.

section_title: What GAO Found
paragraphs: To develop programs to strengthen developing countries’ Public Financial Management (PFM) systems, the U.S. Agency for International Development (USAID) and the U.S. Department of the Treasury (Treasury) rely on assessments of the host country government’s systems. In 2011, USAID implemented new processes that place a greater emphasis on PFM in its development efforts as the agency aims to increase its use of country systems to deliver assistance. The agency traditionally included PFM capacity-building efforts only as components of broader programs, as it identified relevant weaknesses during the country assessment or program design process. USAID’s new strategy and program development processes include a mandatory assessment of a country’s institutional capacity, including its financial systems, and a requirement to consider the use of country systems to deliver assistance. Most USAID country offices are required to develop a strategy using the new guidance by the end of fiscal year 2013. Treasury’s process for developing programs begins with an initial assessment of the host country’s capabilities. Treasury staff then draft objectives for the program. For example, a Treasury program in Honduras set four objectives, including improving operational efficiency and enhancing accountability by strengthening the organization of the ministry of finance. Once in country, the advisor develops an annual workplan, outlining more specific goals aimed at meeting the overall objectives.

USAID and Treasury use several processes to monitor and evaluate their PFM assistance, but weaknesses exist. USAID uses its regular procedures, which may include performance management plans, periodic progress reporting, site visits, and evaluations, to monitor and evaluate its PFM-related programs. Prior reports by USAID’s Inspector General and GAO have found weaknesses in USAID’s implementation of its monitoring procedures in other programs, including programs from the USAID offices that provide PFM assistance. In addition, USAID is currently unable to monitor overall progress toward its target to obligate 30 percent of its program funds through local systems by 2015. USAID, and GAO in prior reports, have identified a number of weaknesses in evaluation practice. To address weaknesses the agency had identified, USAID adopted a new evaluation policy in January 2011 that states that all large projects are required to have an external evaluation, 3 percent of program budgets should be devoted to external evaluation, and evaluations must use methods that generate the highest quality evidence. Treasury’s processes for monitoring and evaluating its programs include monthly reports, annual quantitative performance measures, voluntary customer feedback surveys, and on-site management reviews, but Treasury does not fully evaluate the performance of its completed technical assistance programs. In addition, Treasury’s quantitative performance measures have been a useful project-level indicator of performance but have not been a useful indicator of overall performance due in part to inherent challenges associated with summarizing program performance and errors introduced when aggregating the performance data. Furthermore, a senior Treasury official reported that Treasury had not yet fully implemented a requirement to conduct independent postproject evaluations of its technical assistance programs.

section_title: What GAO Recommends
paragraphs: USAID should improve its capacity to measure its use of local systems and ensure adequate monitoring of its PFM programs. Treasury should implement additional controls to improve the process for computing program-wide annual performance measures and fully implement its requirement to evaluate the impact of its completed assistance. USAID and Treasury both concurred with GAO’s recommendations.

section_title: Letter
paragraphs: A country’s development is influenced, in part, by how effectively its  government raises, manages, and expends public resources. In 2005, as  part of the Paris Declaration on Aid Effectiveness, developing countries  committed to strengthen their public financial management (PFM)  systems and donors committed to use those systems to the maximum  extent possible. More transparent and effective PFM systems can make  governments more accountable and give donors more assurance that  funds will not be misspent if they deliver more aid through recipient  countries’ budgets and finance systems. Highlighting the need for  stronger PFM, international organizations have raised concerns about low  levels of transparency in countries’ budgets and high levels of corruption.

The United States provides assistance to strengthen other countries’ PFM  systems primarily through the U.S. Agency for International Development  (USAID) and the U.S. Department of the Treasury (Treasury). USAID has  set a target of obligating 30 percent of the agency’s total annual  assistance, or program funds, through local systems, including both  partner country PFM systems and local not-for-profit and for-profit  organizations, by fiscal year 2015. Over the last decade, USAID has  provided most of its assistance through agreements with large  international implementing partners. USAID assesses countries’ PFM  systems to determine their ability to effectively receive and manage U.S.  assistance directly. USAID primarily provides PFM capacity-building  assistance as a component of a broader development program, whereas  the Treasury’s Office of Technical Assistance (OTA) provides PFM- related technical assistance. Since 2007, USAID and Treasury have  provided PFM assistance to 70 countries. The U.S. Department of State  (State) assesses the budget transparency of countries that receive U.S.  assistance and helps to leverage existing PFM assistance, but has not  funded PFM assistance.

In this context, you asked us to examine: (1) the processes U.S. agencies  use to develop programs to strengthen PFM systems in developing  countries and (2) the processes U.S. agencies use to monitor and  evaluate their programs to strengthen PFM systems.

To address these objectives, we reviewed USAID, Treasury, and State  guidance and program documents and interviewed U.S. officials in  Washington, D.C. To address the first objective, we reviewed relevant  documents, including assessment reports, agency guidance, and  assistance agreements. To address the second objective, we reviewed  agency and Organization for Economic Cooperation and Development  (OECD) guidance and program documents, including assistance  agreements, progress reports, evaluations, work plans, and performance  management plans; and quantitative performance data. We spoke with  USAID and Treasury officials about their development, monitoring, and  evaluation of PFM-related programs. We also selected six case study  countries in which Treasury or USAID has relevant ongoing or recently  completed projects focused on strengthening PFM systems. We chose  these case study countries for their geographic and income level  diversity, as illustrative rather than generalizable examples. For Treasury,  we selected Cambodia, Honduras, and South Africa; and for USAID,  Kosovo, Liberia, and Peru. For each of our case study countries, we  interviewed U.S. officials and reviewed program development and  monitoring and evaluation documents related to PFM-focused projects.  Finally, we interviewed officials at the World Bank and International  Monetary Fund to discuss the tools that they use to assess PFM systems.

We conducted this performance audit from October 2011 through  September 2012 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives. See appendix I  for a more detailed discussion of our scope and methodology.
subsections: section_title: Background
subsections: section_title: PFM Focuses on a Country’s Budget Cycle
paragraphs: The broad objectives of PFM are to achieve overall fiscal discipline,  allocation of resources to priority needs, and efficient and effective  allocation of public services, according to OECD. While donors may use  different definitions of PFM, most definitions focus on a country’s budget  cycle, the process used to manage public resources. The budget cycle  centers around four main phases: (1) budget formulation, (2) budget  execution, (3) accounting and reporting, and (4) external oversight (see  fig. 1). OECD states that PFM includes all components of a country’s  budget cycle.

A budget cycle starts with the budget formulation process, in which the  government, often with legislative oversight, plans for the use of the  coming year’s resources in accordance with policy priorities. After the  government approves the budget and the new fiscal year begins,  programming agencies and the ministry of finance, or appropriate entity,  are responsible for executing the budget. They use the resources  allocated to them for items such as salaries for public servants, operating  costs for their offices, and goods and services delivered to their  beneficiaries. The ministry of finance, or equivalent, manages the flow of  funds and monitors and makes in-year adjustments to help ensure  compliance with the budget and PFM rules. Throughout the fiscal year,  each programming agency is to account for and record its expenditures.  The ministry of finance centrally consolidates these accounts. At the end  of the fiscal year, the ministry of finance, or equivalent, issues an  accounting report that demonstrates how the budget was implemented.  External or independent entities, such as a country’s supreme audit  institution, may review this report. The audit institution reviews the  government’s revenue collection and spending and issues its own  statement on the execution of the budget and the strength of the PFM  systems. In many countries, the institution presents this audit report to an  appropriate government entity for further scrutiny and action.

PFM processes involve a number of governmental entities. While the  ministry of finance is generally the focus of a country’s PFM system, PFM  extends to all public ministries that are charged with delivering services or  have spending authority. Civil society, donors, and oversight institutions,  such as a country’s supreme audit institution, also help to ensure the  proper management of public funds through external scrutiny and review.

section_title: Assessment Tools Highlight Both Improvements and Weaknesses in Developing Countries’ PFM Systems
paragraphs: Broad PFM assessments conducted by international organizations report  improvements in countries’ PFM systems, as well as continued  weaknesses. Several international organizations, including the Public  Expenditure and Financial Accountability (PEFA) program, World Bank,  International Budget Partnership, and Transparency International, have  developed assessment tools to assess broad aspects of PFM systems.These assessments have highlighted varying levels of progress in  improving PFM systems around the world, and the last two have identified  relatively low levels of transparency in many countries’ budgets and high  levels of corruption, respectively. More specifically:

PEFA: A 2010 monitoring report on the PEFA program, released by  the PEFA Secretariat, found that while PFM systems were improving  overall, progress varied among elements of countries’ systems. The  PEFA secretariat’s analysis, based on a comparison of 33 repeat  PEFA assessments from 2005 through 2010, showed that more  countries had a higher number of improved versus worsened scores,  indicating a broad trend of PFM improvement across the countries  surveyed. According to the analysis, country actions taken to  strengthen PFM features in the earlier stages of the budget cycle,  such as planning, are more likely to improve or maintain a high score  than actions taken later in the budget cycle, such as control and  oversight of actual spending.

World Bank: According to our analysis of the World Bank’s quality of  budgetary and financial management indicator scores from 2005  through 2010, slightly more than one-third of the countries assessed  showed improvements in the quality of their PFM systems, while one- quarter of the countries showed a worsening in the quality of their  PFM systems.

International Budget Partnership: In its 2010 open budget survey  of 94 countries, the International Budget Partnership concluded that  the state of budget transparency was poor. Only about 21 percent of  the countries surveyed had open budgets while 44 percent of  countries provided limited to no budget information. Nonetheless, the  survey found the trend toward open budgets was favorable based on  substantial improvements in budget transparency, especially among  countries that had provided little information in the past. Some of  these governments achieved improvements by simply making budget  documents available on their websites.

Transparency International: The organization’s Corruption  Perceptions Index showed that nearly three quarters of the 178  countries in the index scored below 5, on a scale of 10 (very clean) to  0 (highly corrupt), indicating what it categorizes as a serious  corruption problem.

For further information on each organization’s assessment tools and  selected results, including the percentile rankings of our six case study  countries in selected PFM diagnostic tools, see appendix II.

section_title: International Attention to Strong and Effective PFM Systems Has Increased since 2003
paragraphs: Donors and recipient governments have increased their attention to  strengthening PFM systems, recognizing that strong and effective PFM  systems underpin fiscal and macroeconomic stability, guide the allocation  of public resources to national priorities, support the efficient delivery of  services for poverty reduction and economic development, and make  possible the transparency and scrutiny of public funds. In 2003, OECD’s  Development Assistance Committee established a Working Party on Aid  Effectiveness, which has played a critical role in establishing an initial  international donor coordination framework and setting goals for donors  and aid recipient countries to strengthen PFM. The working party has  sponsored four global development forums since 2003. The second  forum, held in Paris in 2005, resulted in the Paris Declaration on Aid  Effectiveness which included broad commitments by recipient  governments and donors to strengthen PFM systems and use those  systems as appropriate. More than 100 countries and aid agencies,  including the United States, endorsed the declaration. By signing the  declaration, recipient governments made clear commitments to  strengthen their systems to the maximum extent possible, and donor  governments made clear commitments to use those systems wherever  possible. These commitments were renewed and refined in the  subsequent forum in Accra, Ghana, in 2008.  The most recent forum was  held in Bussan, South Korea, in 2011. (See table 1.)

USAID and Treasury are the two main U.S. agencies providing PFM- related assistance. Combined, the two agencies have PFM-related  projects in 70 countries in all regions of the world, as shown in figure 2.  State also conducts some PFM-related activities, although it has not  funded programs. See appendix III for details on State’s PFM activities.

USAID provides PFM capacity-building activities through its development  programs and is seeking to provide more of its assistance through  recipient countries’ financial systems. Capacity-building activities to  strengthen PFM systems are typically part of broader democracy and  governance (DG) or economic growth (EG) programs. PFM activities  that are included as components of DG programs typically address the  areas of legislative function and processes, public sector executive  function, local government decentralization, and anticorruption reforms.  PFM activities included as components of EG programs typically address  fiscal and monetary policy issues. USAID has identified DG and EG  programs with PFM components in over 60 countries since 2007.  However, according to a USAID official, USAID cannot determine the total  funding for PFM activities because it does not collect data at a sufficiently  detailed level to precisely identify PFM activities. Therefore, the official  reported that USAID is unable to separate PFM assistance from other  assistance in larger DG and EG development programs. In 2011, total  funding for DG programs was $1.7 billion, and for EG programs, $4.2  billion.appendix IV.

For illustrative examples of USAID PFM-related projects, see  In addition to PFM capacity-building programs, USAID is seeking to  increase the use of recipient country PFM systems to deliver assistance.  As part of its implementation and procurement reforms under its  institutional reform agenda, known as USAID Forward, USAID has  announced an agency target of obligating 30 percent of its annual  assistance through local systems, including both partner country PFM  systems and local not-for-profit and for-profit organizations, by fiscal year  2015.

Treasury’s OTA provides technical assistance through advisors who work  in-country within the finance ministry, the central bank, or other  government entities. OTA’s program consists of five core areas, as  follows:

Budget and Financial Accountability helps foreign governments  reform and strengthen their PFM systems in order to promote control,  accountability, and transparency over resources, and to improve a  country’s overall financial condition.

Banking and Financial Services promotes strong financial sectors in  which institutions are well regulated, stable, and accessible; serve as  efficient intermediaries between savers and investors; and are  resistant to criminal activity.

Government Debt Issuance and Management helps host countries  implement sound public debt management practices and develop  markets through which the government can finance itself.

Economic Crimes helps counterpart governments build their capacity  to prevent, detect, investigate, and prosecute complex financial  crimes.

Revenue Policy and Administration provides assistance to  ministries of finance and other relevant organizations that strengthens  their ability to serve the country and its people through the efficient  and responsible collection of revenues.

OTA provides both long-term resident and intermittent advisors. Long- term resident advisors provide advice and training to ministers of finance,  central bank governors, and other government officials. Country  engagement typically lasts between 2 and 10 years, according to an OTA  official. Intermittent advisors provide highly specialized technical  assistance in support of long-term projects, projects requiring several  different specialties, and projects of short duration. According to an OTA  official, they currently have about 70 advisors in roughly 50 countries. In  2011, OTA’s total funding was $44.5 million, with $25.5 million in direct  OTA appropriations and $19 million in transfers from other agencies,  including State, USAID, and the Millennium Challenge Corporation. For  illustrative examples of OTA technical assistance projects, see appendix  IV.

section_title: USAID’s and Treasury’s Processes for Developing PFM Programs Involve Consultation with Stakeholders and Rely on Assessments of Host-Country Systems
subsections: section_title: USAID’s Processes
paragraphs: USAID is implementing new processes for developing programs that  reflect new agency reform priorities to increase the use of country  systems to deliver U.S. assistance. USAID’s work in PFM has traditionally  involved capacity building under broader programs designed to improve  fiscal management and promote good governance. According to agency  officials, USAID has increased its attention to PFM issues. USAID’s new  country strategy development and project design processes include  various analyses and assessments that may identify opportunities to  strengthen and use countries’ PFM systems, as prioritized by USAID’s  reform agenda. In these new processes, USAID may identify the need for  PFM assistance during countrywide development assessments or  through other required assessments. Furthermore, according to USAID  guidance, country offices are to develop these assistance programs in  collaboration with country stakeholders throughout the program  development process. USAID’s new processes are similar in structure  and approach to prior processes, but, according to USAID, aim to  incorporate more analytical rigor at all stages of the strategic planning  framework so that USAID’s efforts are better aligned with the recipient  country’s development efforts.

USAID may identify the need for PFM assistance during the country  strategy development process. According to USAID, most country offices  are required to develop a Country Development Cooperation Strategy, a  5-year strategy document, by the end of fiscal year 2013. USAID’s latest  draft of the guidance for developing the country strategy, released in  September 2011, states that the strategy should demonstrate how it is  integrating the goals of USAID’s assistance reform effort, USAID  Forward, such as working through host-country systems and developing  the capacity of civil society and private sector partners. In developing the  country strategy document, the USAID country office is to consult with  various country stakeholders and conduct several assessments to  understand the development context and develop goals and objectives.  According to the guidance, the country office is required to develop the  strategy document with a focus on maximizing the impact of USAID  resources and build the capacity of specific institutions and related  governance systems at the national, regional, and local levels. For  example, if a USAID country office determines that a nontransparent and  inefficient financial system is a key obstacle to economic growth, the  country office is to work with the host government to improve its capacity  for sound financial management and equitable allocation of resources.  According to USAID officials, one indicator of need for a PFM project  would be an assessment that the country has a major fiscal imbalance  that needs to be corrected by a combination of increased revenue  mobilization or reduction of budget expenditures. Another indicator would  be a determination, arrived at through repeated reporting on corruption in  the media, or internal and external publications, and supported by  stakeholder interviews, that anticorruption programs would improve PFM.

The goals and objectives outlined in the country strategy are to provide  the basis for project design, monitoring, and evaluation. As of June 2012,  USAID stated it had approved 15 country strategies, and 73 USAID  country offices are scheduled to complete a strategy by October 2013.  See figure 3 for highlights of the elements of USAID’s new strategy and  project design processes that could identify the need for PFM capacity  building.

According to the new USAID guidance on project design, after USAID  approves the strategy document and identifies the need for PFM-related  assistance, USAID country offices are to begin the project design phase.  This phase is to begin with the formation of a design team that is  responsible for the project’s development from planning to  implementation. Overall, the project design phase is to comprise three  stages, as follows:

Conceptual stage. During the conceptual stage, the project design  team is to conduct stakeholder outreach, assessments, analysis, and  implementation planning. This stage results in a concept paper, which  provides a summary of a proposed project that country-office  management can review to assess how the project aligns with the  country strategy, the likelihood of success, and the assumptions  underlying project success. After USAID management approves the  concept paper, the design team transitions to the analytical stage.

Analytical stage. During the analytical stage, the design team seeks  to understand the problem or constraints identified during the  conceptual stage, and identify and assess critical assumptions. The  team is to conduct a series of targeted assessments, including  required gender, environmental, and sustainability analyses and other  social, political, and institutional analyses. Of these analyses, we  identified two that pertain directly to PFM systems, as follows:

Sustainability analysis. The sustainability analysis assesses the  partner government’s ability to manage the continuation of the  project after the project has concluded. According to USAID  guidance, to build sustainability into a project, the design should  consider how the country office will increase the skills and  capacity of local stakeholders involved in maintaining  development gains after the project ends—as well as how USAID  will ensure that relevant activities or services are gradually tied to  sustainable financing models through private sector participation  or through sustainable, publicly managed arrangements and  government processes.  Institutional analysis. An institutional analysis is an in-depth  assessment of the local institutions and systems most critical to  the implementation of the project’s development interventions,  including an assessment of the quality of their leadership,  structure, and staff; and identification of their administrative and  financial management strengths and weaknesses. Using the  analysis, USAID is to develop a plan for project activities that are  necessary and sufficient to bring these institutions up to the level  of performance or engagement as partners appropriate for their  roles in the project’s implementation and their eligibility for direct  USAID funding.

According to USAID guidance, the analytical stage results in a project  authorization document that will be the basis for project  implementation, adaptation, and evaluation, which includes a  summary of the analyses that underlie the rationale for the project  design.

Approval stage. A successful project design results in an approved  project authorization, which enables a project to move from the  planning stage to implementation. The project authorization sets out  the purpose and duration of the project, defines fundamental terms  and conditions of the assistance when a partner country agreement is  anticipated, and approves an overall total budget level, subject to the  availability of funds.

In addition to the above analyses, USAID has developed a PFM risk- assessment framework (PFMRAF) to measure the fiduciary risk, or the  risk of funds being misspent or mismanaged, when USAID plans to  provide aid through the country’s finance systems. USAID guidance  commits country offices to offer, if appropriate, a USAID assessment of  partner country PFM systems with the goal of providing funding for project  implementation through the use of those systems. If the offer is accepted,  the assessment must be carried out using the PFMRAF. Whenever  possible, USAID should begin conducting the PFMRAF during the  conceptual stage of the project design process. The PFMRAF consists of  the following five stages:  1.  The Rapid Appraisal provides a high-level snapshot of country-level  fiduciary risks associated with the use of partner country PFM  systems and helps inform decisions on whether to undertake a more  rigorous, formal risk assessment.  2.  The Risk Assessment, Analysis, Mitigation, and the Approval for  Use of Partner Country Systems establishes the baseline level of  risk corresponding to contemplated funding levels and identifies  vulnerabilities of the partner country PFM sector in which USAID is  considering use of the system for project implementation. USAID  determines whether any systemic risk can be reasonably mitigated  and, if so, what kind of mitigating measures might be introduced to  reduce the risk. USAID establishes an accountability framework with a  set of conditions that would, if complied with, constitute formal  approval for the use of a partner country PFM system.  3.  The Project Design, Approval, Designation of Responsibilities,  and Selection of the Funding Mechanism incorporates the approval  of use of country systems into the project design, includes risk  mitigation measures, such as capacity-building technical assistance,  concurrent audits, and disbursements in tranches, as appropriate, and  the appropriate funding mechanism.  4.  Negotiating and Preparing the Bilateral Project Agreement with  the Partner Country Government involves the preparation of the  bilateral agreement in accordance with the risk mitigation measures  outlined in the approval for use of partner country systems and in  collaboration with the partner country government.  5.  Implementation, Monitoring, and Evaluation occurs once both  countries agree to the bilateral agreement and as outlined in the  project design.

According to USAID, the country office is to incorporate recommendations  on actions to mitigate identified risks to providing assistance directly to  the host government into various stages of the project design. According  to one USAID official, the country office’s decision to implement the  recommended steps for mitigating risk may depend on the availability of  funds. According to USAID, if risks cannot be mitigated, USAID is not to  deliver assistance through the partner country’s financial systems.  Mitigation steps may include adding a PFM component to a project, such  as technical assistance to improve an aspect of the financial system. As  of June 2012, USAID had completed the rapid assessment in 20  countries and was in the process of finalizing the report for an additional  12. For 4 of the countries for which USAID had completed the rapid  assessment, USAID had decided to either delay or not to proceed to  stage 2 for various reasons, including funding cuts and the political  situation in the country. For the second stage of the PFMRAF, USAID had  completed the risk assessment for 3 countries and was in the process of  conducting the assessment for 11 others. USAID had completed four  stages of the PFMRAF for 3 countries. Table 2 contains a summary of the  status of countries where USAID has begun the PFMRAF process.

Stage 2 on hold or no plan to continue to stage 2 in four of these countries.

section_title: Treasury’s Processes
paragraphs: Treasury OTA’s processes for developing PFM programs involve  collaboration with country government officials whom Treasury has  determined have a strong commitment to reform and are seeking to  develop strong PFM, as well as OTA’s assessment of the country system.  OTA receives requests for assistance from foreign governments, other  U.S. government agencies, U.S. embassies, international organizations,  and OTA advisors and other donors already working in countries on other  projects. According to OTA officials, OTA receives two to three requests  for assistance per month, but due to the agency’s limited financial  resources and the financial commitment required for each project, it  selectively chooses its projects. OTA’s principles for providing assistance  are as follows: work with partners committed to ownership of their reform; advocate self-reliance by providing countries with knowledge and  skills required to generate and manage their own resources and  reduce dependence on international aid; and work with ministry staff daily to build capacity through mentoring and  on-the-job training on PFM practices.

After receiving the request, OTA begins a preliminary assessment  process to identify weaknesses and areas for assistance, according to  OTA officials. The preliminary assessment is primarily a review of  available information on the country’s PFM system, which may include  information from the U.S. embassy, as well as relevant country reports  and assessments prepared by international financial institutions, other  bilateral assistance providers, and nongovernmental organizations. After  reviewing documents, if it decides to proceed with the assessment, OTA  completes a more in-depth, in-country assessment of PFM, which  consists of meetings with U.S. embassy staff, other donors, and relevant  ministry officials, including the minister of finance or the head of the  central bank, as well as high-level policy staff and mid-level supervisors.  According to OTA officials, the purpose of the in-country assessment is to  review the structural issues related to the country system and determine  whether the country has dedicated staff committed to working toward their  reform efforts. At any point during the assessment process, OTA may  determine that OTA assistance is not suitable for the country’s needs.  OTA officials told us that in one such case, government officials  requested that OTA manage their budget for them, which is against  OTA’s principles because doing so would not promote self-reliance or  help the country develop the capacity to generate and better manage its  own revenues. Finally, OTA officials told us that their decision on whether  to provide assistance depends on whether sufficient funding is available.

The in-country assessment typically results in a draft of official terms of  reference, which identifies weaknesses to be addressed during OTA’s  engagement. According to OTA officials, the terms of reference describes  the broad goals of the project and represents agreement between  Treasury and the host-country counterpart, the ministry staff with whom  the advisor will work on a daily basis. For example, the 2010 terms of  reference for a budget project in Honduras identified the following four  agreed-upon areas for assistance:  improve operational efficiency and enhance accountability by  strengthening the organization of the ministry of finance; move toward compliance with International Public Sector Accounting enhance capacity to conduct fiscal analysis and produce more reliable  macroeconomic estimates and revenue and expenditure projections;  and  conduct workshops on basic finance methods and terminology.

The terms of reference also identifies the advisor’s host-country  counterparts and agreements regarding each party’s responsibilities.  According to OTA officials, after OTA fully vets a project and secures  funding, the parties finalize and sign the terms of reference. However,  OTA and its counterparts may seek to revise the terms of reference if the  central purpose of the technical assistance project changes or during  times of transition, such as when a finance minister changes. In  Honduras, the 2010 terms of reference replaced an existing OTA project  following a gap in OTA assistance during a political crisis in the Honduran  government in 2009.

The OTA advisor assigned to the country uses the broad goals expressed  in the terms of reference to develop a work plan. The work plan contains  specific objectives, milestones, and planned completion dates designed  to work toward goals agreed upon in the terms of reference, and the plan  must be developed within 60 days of the advisor arriving in-country. Each  work plan covers a 1-year period and is the primary basis for regular  monthly progress reports to OTA headquarters. OTA considers work  plans, and the monthly reports on which they are based, to be dynamic  documents that reflect the project’s progress in real time, and advisors  can and should change or modify the work plan during the course of a  project as appropriate and in consultation with the ministry staff and OTA  management. Moreover, according to OTA officials, a 1-week in-country  assessment may not provide all of the information needed and may  require that the advisor rewrite the plan after a few months in the country.  While the work plan is considered a management tool to monitor projects,  OTA officials told us that in recent years it has also become a joint  document between the OTA advisor and ministry staff as a shared tool to  monitor and discuss progress.

section_title: USAID and Treasury Use Certain Processes to Monitor and Evaluate PFM-Related Programs, but Implementation Has Some Weaknesses
subsections: section_title: USAID’s Processes
subsections: section_title: USAID Uses Multiple Tools to Monitor Its PFM Assistance but Lacks a Process for Tracking Use of Country Systems
paragraphs: To monitor PFM-related programs, USAID develops performance  management plans, reviews periodic progress reports, and conducts site  visits, among other activities, but reviews of USAID programs have  identified agency-wide weaknesses in implementation, including using  unreliable baseline data and inaccurate reporting of results. USAID  develops the monitoring and evaluation framework during the project  design, and it generally is to include the following:

A performance management plan: This is a tool to plan and  manage the process of monitoring, evaluating, and reporting progress  toward achieving the project’s development objectives. The  performance management plan includes performance indicators and  targets that link to the project objectives.

A work plan: This specifies activities to be undertaken and the  proposed schedule for these activities during the life of the project.

Periodic progress reports: USAID assistance agreements with  nongovernmental organizations generally also require implementing  partners to submit periodic progress reports, the frequency of which  vary depending on the assistance agreement, but these reports may  not be required more frequently than quarterly or less frequently than  annually. The progress reports should generally contain a comparison  of actual accomplishments with the goals and objectives established  for the period and reasons why established goals were not met, if  appropriate. Implementing partners should immediately notify USAID  of developments that have a significant impact on the award- supported activities.

Site visits: USAID guidance states that site visits are conducted as  needed by the technical officers, who are responsible for monitoring.

USAID works with the implementing partner to produce and approve the  work plan and the performance management plan. USAID’s technical  officer approves the plans. USAID guidance also requires staff to conduct  at least one portfolio review each year that covers all activities included in  their various programs. These reviews determine, among other things,  whether the activities are achieving the desired results.

Prior reviews of USAID programs have identified challenges in the  agency’s implementation of its monitoring processes across many types  of programs, including DG and EG programs. In its fiscal year 2011  memo on management and performance challenges, USAID’s Inspector  General identified specific monitoring-related weaknesses as one of  USAID’s most serious management and performance challenges. The  USAID Inspector General reported problems with assistance planning in  25 out of 80 performance audits conducted in fiscal year 2011.  Assistance planning is important because it provides the means for  program implementers to track progress toward program objectives and  helps to ensure that USAID assistance programs achieve planned results.  In addition, 37 of the 80 audit reports the Office of the Inspector General  issued in fiscal year 2011 identified cases in which USAID operating units  or their implementing partners reported misstated, unsupported, or  unvalidated results. In recent reports, we also identified deficiencies in  USAID’s monitoring practices, including the lack of an integrated plan for  monitoring and evaluating nonemergency food aid, monitoring practices  that do not correspond to agency performance guidelines, difficulties in  developing meaningful outcome indicators related to building trade  capacity, undocumented site visits for assistance programs in Burma,  and lack of performance targets and baseline data for indicators related to  PFM efforts in Afghanistan.

Moreover, in our review of the fiscal year 2011 and 2012 Inspector  General’s audits of USAID’s DG and EG programs, we found monitoring  weaknesses cited in 20 of the 32 audit reports. monitoring weaknesses, including unreliable or nonexistent baseline data;  performance data weaknesses, such as results that were not reported,  lack of data verification, or inaccurate reporting of data; lack of a current  performance management plan; and inadequate monitoring of program  activities, including lack of regular submission of progress reports.

With regard to our three case study countries, we found that the country  offices generally applied USAID’s monitoring processes in all three PFM- related programs; however, we did find some conditions that could make  monitoring more difficult, as described below. Our review was not  intended to be comprehensive or applicable to all USAID PFM-related  programs.

USAID’s DG and EG offices are the primary offices providing PFM-related assistance.

Performance management plans: All three programs were clearly  defined in terms of overall objectives, project objectives, tasks,  expected results or outcomes, and required plans and reports. All  three programs—the Kosovo Growth and Fiscal Stability Initiative, the  Peru ProDecentralization Project, and Liberia’s Governance and  Economic Management Support Program—had performance  management plans containing objectives and indicators. Two of the  three performance management plans also included targets.  However, the targets for Liberia’s program had not been determined 9  months into the program.

Work plans: All three programs also had work plans that specified  tasks and timelines. However, Liberia’s Governance and Economic  Management Support Program initially operated without an approved  work plan, although it did establish an action plan to guide its  activities.

Periodic progress reports: Assistance agreements for all three  USAID programs required quarterly progress reports, and the  implementing partners submitted all reports according to agreed-upon  time frames. In our review of the three programs, we found that the  language used to describe the objectives and results in the quarterly  progress reports did not correspond to the language in the work plans,  which could make it difficult to track progress against the work plans’  objectives. For example, the quarterly reports for the Kosovo Growth  and Fiscal Stability Initiative Program describe progress on five  separate lines of effort under the objective to improve fiscal  stewardship, while the statement of work and work plan discuss only  three lines of effort under the same objective.

Site visits: USAID officials responsible for monitoring progress on all  three projects said that they do not rely solely on progress reports to  track progress, and that additional monitoring tools included weekly  progress meetings, frequent site visits, and telephone and e-mail  communications with implementers. For example, the USAID official  in Kosovo reported that he works on a daily basis with the  implementing partners who produce the quarterly and annual reports  and that they exchange e-mails and phone conversations daily.

See appendix IV for descriptions of USAID PFM-related assistance  programs from our case study countries.

In addition to USAID’s reported program monitoring weaknesses, we  found that the agency does not have a process to fully identify and  measure its use of country systems. USAID has set an agency target of  obligating 30 percent of the agency’s annual assistance through local  systems, including both partner country PFM systems and local not-for- profit and for-profit organizations, by fiscal year 2015. currently cannot track progress toward its 30 percent target because its  accounting system cannot identify the full range of such assistance, which  includes a variety of implementing mechanisms from host-country  contracting to direct cash transfers. The project data in USAID’s  accounting system does not provide sufficiently detailed information, such  as the location of the organization receiving assistance, to identify  qualifying assistance. According to a senior USAID official, the agency is  working on a system whereby it will tag each entity receiving assistance  with an identifier, such as government or not-for-profit organization, as  well as a vendor location signifying U.S.-based or non-U.S. based.  According to the official, this tagging process will facilitate identifying use  of country systems. However, USAID’s financial system currently cannot  distinguish whether non-U.S. based not-for-profit and for-profit  organizations are located in the host country or in a third-party country.

USAID’s process to assess the effectiveness of its PFM-related programs  involves independent evaluations, but weaknesses in the agency’s overall  evaluation practices have been reported. USAID may use independent  external evaluations in the middle or at the end of a program based on  the need to inform program decisions. USAID adopted a new evaluation  policy in January 2011 and updated its guidance in February 2012 to  reflect it. The new evaluation policy requires that all large projects have at  least one evaluation and that evaluations use methods that generate the  highest-quality and most credible evidence, including experimental   The policy also states that 3 percent of program budgets  methods.should be devoted to external evaluation and classifies two types of  evaluations: performance evaluations, which focus on descriptive and normative  questions and other questions pertinent to program design,  management, and operational decision making; and  impact evaluations, which define a counterfactual (what would have  happened had the program not been implemented) to control for  external factors and measure the change in a development outcome  that is attributable to a defined intervention.

USAID reported that experimental methods generate the strongest  evidence for impact evaluations; however, experimental methods and  impact evaluations may be difficult to apply to PFM capacity-building  efforts. For example, USAID evaluations of PFM programs have  highlighted difficulties associated with conducting impact evaluations,  including lack of data or resources, and too short a time period to identify  impact. A February 2012 USAID report on the first year of  implementation of the new evaluation policy noted that USAID had not yet  completed an evaluation under the new policy.

USAID adopted the new policy to address a number of weaknesses it had  identified in its evaluation practices. USAID reported that the number of  evaluations submitted to USAID’s Development Experience  Clearinghouse—its main repository for agency evaluations—decreased  from nearly 500 in 1994 to approximately 170 in 2009 despite an almost  three-fold increase in program dollars managed. Furthermore, according  to USAID, the majority of evaluations in recent years relied heavily on  anecdotal information and expert opinion, rather than systematically  collected evidence. We have also reported weaknesses in USAID  evaluation practices in several areas in recent years, including not  planning for evaluations or not using evaluation results. Furthermore,  the weaknesses in performance indicators that the USAID IG identified in  DG and EG programs could create difficulties for evaluating PFM  programs. For example, weaknesses in baseline data or performance  indicators would make it difficult to use quantitative measures to evaluate  these programs.

These evaluations were not subject to USAID’s new evaluation policy, which was issued  in January 2011. evaluation of the Economic Management for Stability and Growth  program and found that the program improved the government’s  institutional capacity in fiscal and monetary policy.did not document its evaluation methodology.

section_title: Treasury’s Processes
subsections: section_title: Treasury’s OTA Uses Various Processes to Monitor PFM- Related Programs
paragraphs: Treasury’s OTA uses various processes to monitor PFM-related  programs, including narrative monthly and trip reports, annual quantitative  performance measures, and customer feedback surveys, but weaknesses  exist in some of its evaluation processes. The work plan is the key  document OTA uses to monitor a project. OTA uses monthly reports to  document and monitor progress toward the milestones identified in the  annual work plan, including changes to milestones or timelines, and to  facilitate communication between the resident advisors and headquarters.  OTA advisors may also share monthly reports with host-country  counterpart institutions, U.S. embassy staff, other Treasury bureaus and  offices, and other interested partners such as USAID, the Millennium  Challenge Corporation, and relevant international financial institutions to  monitor progress and coordinate donor assistance as relevant. In addition  to updates on work-plan progress, the monthly reports may also include  other information such as activities completed; significant meetings held;  and important political, social, and economic developments. OTA also  uses monthly reports to track changes made to the work plan agreed to  by the advisor and his or her counterpart. For example, an OTA budget  project in Cambodia listed implementation of program budgeting as one  its primary objectives and improving budget classification as one its  secondary objectives. In the monthly reports, the resident advisor  documented the need to refocus the reform efforts on budget  classification to better align with the Cambodian government’s ongoing  PFM reform efforts. Further, in the case of OTA’s 2011 Honduras budget  project to implement international public sector accounting standards, the  objectives remained unchanged, but milestones were changed in the  monthly reports. According to the advisor, some milestones had to be  extended due to overly ambitious initial expectations that were purposely  established to encourage the ministry staff to undertake reforms. See  appendix IV for descriptions of OTA-funded PFM projects in our case  study countries.

OTA headquarters also maintains regular communication with its  advisors. Senior OTA officials monitor project performance through  regular contact with advisors in the field by e-mail, telephone, and site  visits. One advisor told us that their director would already be aware of  any issues raised in the monthly reports, given the frequency of their  communications. Resident advisors also told us that senior Treasury staff  visited Cambodia and Honduras to review OTA projects and talk with  counterparts. Following site visits, OTA officials are required to prepare  trip reports. For example, in a trip report from November 2011, a senior  OTA official reported that he met with host-country counterparts to  discuss the four current OTA engagements in Cambodia, including the  budget project. The visit confirmed the reported difficulties in  implementing the government financial management information system  as a result of a procurement delay, but also affirmed OTA’s role in its  eventual implementation. OTA also uses voluntary customer surveys to  collect feedback on OTA projects from the host-country counterpart. For  example, a Honduran government official who completed the voluntary  customer feedback survey for the budget project indicated that the project  met expectations and made a significant contribution by strengthening the  host-country staff’s technical capacity through the training sessions that  OTA staff offered.

We found that OTA applied its monitoring processes to all three of our  case study country projects, but monthly reports lacked some details on  progress. We reviewed OTA’s monitoring activities in three PFM-related  projects to examine how OTA applied its monitoring process to these  projects. Our review was not intended to be comprehensive or applicable  to all OTA projects. We found the projects to be clearly defined with  specific objectives, milestones, proposed completion dates, and regularly  submitted monthly reports. In addition, in all three cases, the resident  advisor reported frequent communication on project progress with  headquarters. However, in two of the case study countries, Cambodia  and South Africa, work plans had major changes in objectives, but only  Cambodia documented its change in a monthly report. Additionally, our  review found that the level of detail of the monthly report narratives on  progress varied, with a few reports having little detail, which could make  tracking progress for these specific projects difficult without other  communication between the advisor and OTA management. However, as  noted above, OTA headquarters maintained frequent communication with  advisors in the field.

OTA’s quantitative performance measures have been a useful project- level indicator of performance but have not been a useful indicator of  OTA-wide performance because of conceptual problems and errors  introduced by OTA when aggregating the performance data. OTA  advisors and management score technical assistance projects annually  based on project-specific indicators under four main categories. The  projects are scored on a scale of one (lowest) to five (highest) for  “traction,” or the degree of engagement with host-country counterparts,  and “impact,” or the results of project activities that bring about change in  counterpart law, systems, processes, and procedures. OTA officials told  us that the traction and impact measures also reflect the language and  values of OTA, and are useful to management, advisors, and host- country counterparts. The primary purpose of the quantitative  performance scores was to respond to an Office of Management and  Budget reporting requirement for evaluation data. While OTA reported  that it met or exceeded targets for traction and impact set in 2008, in  every year from 2009 through 2011, OTA officials acknowledged that the  aggregate values associated with its annual goals were of limited value  due to lack of comparability across programs and over time. In previous  work on the Office of Management and Budget, we highlighted the  difficulty of representing program performance with a single rating.  Using a single rating can force agencies to simplify more nuanced and  complex performance results, a circumstance similar to OTA’s  aggregation of traction and impact scores. A senior official told us that  OTA, in complying with the OMB requirement, designed the measures to  be as useful as possible at the project level.  For example, OTA uses the  project-level traction and impact measures in setting expectations and  discussing progress with both the host-country counterpart and the  advisor.  In addition, the official reported that OTA is continually looking  for additional ways to use the traction and impact scores.  One example  of how researchers have used PFM performance data is by developing  analytical approaches to identify the determinants of strong PFM systems  and elements of successful PFM reform efforts, which can help to control  for important differences across countries.

Our analysis of OTA performance data found several errors that OTA  introduced when aggregating project-level data; these errors collectively  raised questions about the reliability of the instrument used to aggregate  OTA’s quantitative performance measures and, thus, suggested there  may be limitations in its ability to provide insight into performance across  OTA projects. OTA uses this instrument to calculate the annual  performance averages which are compared against annual targets and  provided to OMB and to Congress in annual reports. Our analysis of the  spreadsheet containing these data, and limited spot checking of some  underlying data for our three case-study countries, identified a number of  errors, including those introduced when transcribing data to the  spreadsheet. In one instance, OTA listed a performance score under the  wrong country, which resulted in inaccurate information for two countries’  projects.

OTA has not yet fully implemented its requirement to conduct  independent evaluations—that is, evaluations conducted by someone  other than the resident advisor—of its completed technical assistance.  OTA guidance for project reporting and documentation includes  requirements for end-of-tour reports when a resident advisor leaves and  end-of-project reports when a technical assistance project is completed.  The end-of-project report is a postproject evaluation whose purpose is to  (1) compare accomplishments with the initial objectives of the project and  (2) improve the planning and execution of future projects. A senior OTA  official said that the requirement for the end-of-tour report was fully  implemented and that OTA intended to fully implement the end-of-project  report requirement by the end of 2012. According to the guidance, the  evaluation must be conducted independently; the report is the  responsibility of the associate director and can be delegated to a senior  advisor, but it should not be prepared by an advisor who prepared an  end-of-tour report. Both OECD guidance and USAID guidance also  highlight the importance of independence in evaluation. In contrast to the  guidance, OTA provided three end-of-project reports that were conducted  by OTA staff who had been involved in providing some of the technical  assistance being evaluated. While no comprehensive and fully  independent end-of-project evaluation has been conducted, OTA officials  described relevant insights from a postproject trip to one country that was  undertaken by a senior OTA official in part to better understand the  longer-term impact of OTA assistance. During the trip, that official  identified some factors that significantly limited the impact of this project,  notably a lack of counterparty commitment at the policy level to  implement and sustain reforms in key areas identified in the terms of  reference.

section_title: Conclusions
paragraphs: As more donors, including the United States, seek to provide additional  assistance through countries’ systems, the need for strong PFM systems  and accountability in recipient countries become even more important to  help lessen corruption and ensure countries effectively manage  resources. Given USAID’s stated goal of obligating 30 percent of annual  assistance through local systems, including partner country PFM  systems, by 2015, it must ensure that those systems are functioning  properly and transparently. Efforts to strengthen PFM systems can lower  the risks that assistance delivered through country systems will be  misspent, while also increasing the capacity of the recipient country to  effectively manage its own resources. Given concerns about  transparency and corruption in many aid-recipient countries, achieving  the expected benefits of using recipient countries’ PFM systems may be  difficult without concerted efforts by the donors and countries to  strengthen these systems. In addition, the risks to using these systems  are increased without efforts to strengthen them. To help strengthen  these systems, USAID provides developing countries with PFM capacity- building assistance. In light of this assistance, and given the difficulties  USAID has experienced in the past with implementing monitoring and  evaluation practices, the importance of developing efforts to strengthen  PFM systems and effectively monitoring and evaluating these efforts has  increased significantly. Previous difficulties USAID has experienced with  regard to monitoring, including poor quality of reported data on projects,  and the lack of reliable baseline data, could affect USAID’s ability to  conduct effective evaluations of these projects, as evaluators will not  have access to reliable data. Also, while USAID’s new evaluation policy  places greater emphasis on impact evaluations and experimental  methods, it may be difficult for USAID to evaluate its PFM capacity- building efforts using these approaches. Finally, for USAID to achieve its  target for use of local systems, by fiscal year 2015, it must be able to  identify and measure the full extent of assistance that qualifies.

Treasury OTA advisors provide technical assistance targeted at  weaknesses in countries’ PFM systems. Although OTA has taken  numerous steps to monitor this assistance, errors in its aggregated  performance data and the lack of comprehensive postproject evaluations  limit OTA’s ability to effectively evaluate its assistance. OTA has adapted  a challenging mandate from OMB to create a useful measure of its efforts’  performance for management, advisors, and host-country counterparts;  however, conceptual problems with and errors in the aggregated  measures undermine the measures’ reliability. With greater confidence in  the quality of the data, opportunities exist to better identify patterns in  performance across OTA programs, such as economic or institutional  factors that influence program performance. These patterns could help  OTA better understand the strengths and weaknesses of its assistance  programs and make appropriate changes. OTA could also use the  quantitative performance measurement system it has developed to  experiment and document the results of new approaches. Finally,  although OTA guidance recognizes the importance of postproject  evaluations by including in its guidance the requirements for an end-of- project evaluation, the agency has yet to enforce this requirement.  Without a postproject evaluation OTA may not fully understand results of  its technical assistance or be able to apply lessons learned to new  projects.

section_title: Recommendations for Executive Action
paragraphs: To monitor progress toward USAID’s target to obligate 30 percent of its  annual assistance through local systems by 2015, we recommend that  the Administrator of USAID direct the appropriate offices to develop a  process to reliably identify and track the agency’s use of local systems in  all countries receiving assistance.

To help ensure that USAID conducts effective evaluations of PFM-related  programs under its new policy, we recommend that the Administrator of  USAID direct the appropriate offices to ensure that they are establishing  adequate monitoring practices for PFM-related programs. Such practices  may include selecting proper indicators, collecting reliable baseline data,  and ensuring the reliability of reported results.

To improve the effectiveness of OTA’s technical assistance, we  recommend that the Secretary of the Treasury direct OTA to take the  following two actions to improve monitoring and evaluation:  implement additional controls to improve the process for computing  OTA-wide annual performance measures, and  fully implement OTA’s existing requirement for end-of-project  evaluations and, consistent with its existing guidance, have an  independent party conduct the evaluations.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to USAID, Treasury, and State for their  review and comment.  Both USAID and Treasury concurred with our  recommendations in their written comments, which are reproduced in  appendix V and appendix VI respectively. These agencies along with  State provided technical comments and updated information, which we  have incorporated throughout this report, as appropriate.

In concurring with our two recommendations, USAID reported that it is in  the process of refining definitions that will identify and help measure the  assistance that qualifies to meet the agency’s target of obligating 30  percent of its annual assistance through local systems by 2015.  USAID  also reported that it has implemented two complementary reforms that will  help ensure effective evaluations and adequate monitoring of its PFM  assistance.  The first reform involves USAID planning for the monitoring  and evaluation of assistance during the early stages of project design,  including defining indicators and collecting baseline data.  The second  reform requires USAID to plan final monitoring and evaluation schedules  during project design.

In concurring with our two recommendations, Treasury reported that OTA  has corrected several errors in the 2011 annual performance measures,  and has taken steps to strengthen data controls, including conducting  additional reviews and increasing staff resources dedicated to computing  the performance measures.  In addition, OTA has begun to implement its  requirement for independent end-of-project evaluations of its technical  assistance and intends to fully implement the requirement by the end of  2012.

We are sending copies of this report to appropriate congressional  committees. We are also sending copies of this report to the  Administrator of USAID and the Secretaries of the Treasury and State. In  addition, this report is available at no charge on the GAO Web site at  http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-9601 or melitot@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made key contributions to this reported  are listed in appendix VII.

section_title: Appendix I: Scope and Methodology
paragraphs: To examine the process the U.S. Agency for International Development  (USAID) uses to develop programs to strengthen public financial  management (PFM) systems, we reviewed USAID’s official policy and  procedures, as contained in the Automated Directives System, as well as  USAID’s new guidance documents for developing country strategies and  designing projects, including the Country Development Cooperation  Strategy and Project Design Guidance. We also interviewed USAID  officials, including officials from the democracy and governance (DG) and  economic growth (EG) offices. Most USAID country offices are required to  have an approved country strategy by the end of fiscal year 2013 and to  follow the new project design guidance starting July 2012. Because  USAID was still in the process of transitioning to these new processes  during the course of our audit, we did not review USAID’s implementation  of these processes.

To examine the process the U.S. Department of Treasury’s Office of  Technical Assistance (OTA) uses to develop programs to strengthen PFM  systems, we interviewed senior OTA officials regarding OTA’s policies  and procedures. We interviewed resident advisors regarding how OTA  assessed and developed projects in our three OTA case study countries.  We also reviewed OTA’s official policy guidance document and project- specific documents, including assessment reports, signed terms of  reference, work plans, and monthly reports.

To examine and assess the processes USAID uses to monitor and  evaluate its PFM-related programs, we reviewed USAID program  development guidance; monitoring and evaluation policies and  procedures; USAID reports; and program documents, including  assistance agreements, monitoring and evaluation plans, and progress  reports; USAID external project evaluations; past GAO reports; and  USAID Inspector General reports. We used the review of the documents  for our three USAID country case studies to illustrate the implementation  of USAID’s monitoring processes for its PFM-related programs. We also  interviewed agency officials in Washington, D.C., and conducted  telephone interviews and had e-mail communications with key country- office staff for each of our case study countries.

To examine and assess OTA’s processes for monitoring and evaluating  PFM-related programs, we reviewed OTA project reporting and  documentation instructions, project work plans, and monthly reports, end- of-tour and end-of-project reports, annual quantitative performance data,  Organization for Economic Cooperation and Development guidance on  evaluation, and conducted interviews with OTA headquarters staff. We  used the review of the documents for our three OTA case studies to  illustrate the implementation of OTA’s monitoring processes. We  supplemented the document review with interviews with current or former  advisors for each of the case study countries. In addition, we assessed  the reliability of the instrument OTA uses to aggregate quantitative  performance measures across projects. We examined spreadsheets  provided by OTA for consistency, examined data for outliers and missing  values, and spot-checked the transcription of data to the spreadsheet for  our case study countries. Due to a number of errors in the OTA data, we  could not determine if the aggregated performance data were sufficiently  reliable for identifying patterns in performance across projects or over  time.

In selecting our case study countries, we focused on countries in which  OTA or USAID had relevant ongoing or recently completed projects  designed to strengthen PFM systems. We selected six countries:  Cambodia, Honduras, and South Africa for OTA, and Kosovo, Liberia,  and Peru for USAID. In selecting these countries, we considered the  following two factors:

Geographic diversity: For each agency, we selected countries from  three different geographical regions.

Country income group diversity: For each agency, we chose a country  that the World Bank has listed as (1) lower income, (2) lower-middle  income, and (3) upper-middle income in order to report examples from  different income levels, which may also be associated with different  institutional characteristics.

In cases where more than one country would be acceptable under our  decision criteria, we considered additional criteria, such as the availability  of other broad-based indicators. For OTA, we focused our selection on  countries receiving technical assistance from OTA’s Budget and Financial  Accountability team, given its focus on traditional PFM aspects. For  USAID, we selected our countries from a list of countries with significant  PFM-related programs that USAID provided. USAID excluded some  countries with PFM-related programs from the list because staff were not  available to discuss their programs with us. Our review of USAID and  OTA case study countries was not intended to be comprehensive or  applicable to all their respective programs and projects or generalizable to  all countries.

To describe recent trends in country PFM systems, we reviewed the data  and publications of five international organizations that conduct broad  assessments of country PFM systems. These broad assessment tools  include the Public Expenditure and Financial Accountability Program, the  World Bank’s Country Policy and Institutional Assessment’s quality of  budgetary and financial management indicator, International Budget  Partnership’s Open Budget Survey, Transparency International’s  Corruption Perceptions Index, and the International Monetary Fund’s  fiscal transparency Reports on the Observance of Standards and Codes.  We converted our six case study country scores for the three PFM  diagnostic tools for which scores were available (Open Budget Survey,  Corruption Perceptions Index, and quality of budget indicator) into  percentile rankings to illustrate each country’s performance as measured  by the three PFM diagnostic tools. We also interviewed officials at the  World Bank and International Monetary Fund to discuss their PFM-related  diagnostic tools.

To describe the Department of State’s PFM-related efforts, we conducted  interviews with agency officials in Washington, D.C. We reviewed State  documents, including agency guidance, waiver packages, and program  documents. We also reviewed relevant appropriations laws to identify the  requirements for State’s fiscal transparency reviews.

We conducted this performance audit from October 2011 through  September 2012 in accordance with generally accepted government  auditing standards. Those standards require that we plan and perform the  audit to obtain sufficient, appropriate evidence to provide a reasonable  basis for our findings and conclusions based on our audit objectives. We  believe that the evidence obtained provides a reasonable basis for our  findings and conclusions based on our audit objectives.

section_title: Appendix II: Several International Organizations Reported Results of Their Public Financial Management Diagnostic Tools
paragraphs: Five international organizations — the Public Expenditure and Financial  Accountability Program, the World Bank, International Budget  Partnership, Transparency International, and the International Monetary  fund — have developed tools to assess various aspects of countries’  public financial management (PFM) systems, and some have published  recent findings or results. We illustrate the percentile ranking for each of  our case study countries in three broad PFM diagnostic tools – the Open  Budget Survey, the Corruption Perceptions Index, and the World Bank’s  quality of budgetary and financial management indicator.
subsections: section_title: Public Expenditure and Financial Accountability Program’s PFM Performance Measurement Framework
paragraphs: In December 2001, a multiagency partnership founded the Public  Expenditure and Financial Accountability Program (PEFA) to strengthen  the ability of aid recipients and donors to assess and improve country  public expenditure, procurement, and financial accountability systems.   In 2005, the program developed the PFM Performance Measurement  Framework, known as the PEFA Framework, to provide a measure of the  strengths and weaknesses of a country’s PFM system. The PEFA  Framework incorporates a PFM performance report, which includes an  assessment of the evaluated country’s PFM performance along six core  dimensions of PFM. The six dimensions are the following:

Credibility of the budget: The budget is realistic and is implemented  as intended.

Comprehensiveness and transparency: The budget and the fiscal  risk oversight are comprehensive, and fiscal and budget information is  accessible to the public.

Policy-based budgeting: The budget is prepared with due regard to  government policy.

Predictability and control in budget execution: The budget is  implemented in an orderly and predictable manner, and arrangements  for the exercise of control and stewardship in the use of public funds  exist.

Accounting, recording, and reporting: Adequate records and  information are produced, maintained, and disseminated to meet  decision-making control, management, and reporting purposes.

External scrutiny and audit: Arrangements for scrutiny of public  finances and follow up by executive are operating.

The six core dimensions include 28 high-level indicators, each of which is  assigned a letter score from A to D. The initial assessment helps  establish performance baselines, while repeat assessments help in  monitoring performance progress over time.

In 2011, the PEFA secretariat released the 2010 PEFA Monitoring  Report. Based on a comparison of 33 repeat PEFA assessments over the  2005-2010 period, the analysis shows that more countries had a higher  number of highest or improved scores (23 countries) than lowest or  worsened scores (8 countries), indicating a broad trend of PFM  improvement across the countries surveyed. According to the analysis,  while PFM systems are improving overall, systems features vary  significantly. Formal PFM features where progress can be achieved  through adopting a new law, regulation, or technical tool, or focusing on  no more than a few agencies, or at an early stage in the budget cycle are  more likely to improve or maintain a high score than functional PFM  features where progress requires actually implementing a new law or  regulation, or coordinating the work of many agencies, or working later in  the budget cycle.

Starting July 2012, the PEFA partners are extending the PEFA program  by 5 years and conducting a comprehensive review of the PEFA  Framework, the first since it was launched in 2005. One of the objectives  is to improve confidence in PEFA assessments through an endorsement  process that provides an incentive to ensure adherence to PEFA good  practices in undertaking an assessment.

section_title: World Bank’s Quality of Budgetary and Financial Management Indicator
paragraphs: The World Bank undertakes annual Country Policy and Institutional  Assessments to assess the quality of a country’s present policy and  institutional framework. One of the 16 indicators the World Bank uses to  assess the performance of a country’s current policies and institutions is  the quality of budgetary and financial management. This criterion  assesses the extent to which the country has a comprehensive and  credible budget linked to policy priorities; effective financial management  systems to ensure that the budget is implemented as intended in a  controlled and predictable way; and timely and accurate accounting and  fiscal reporting. Over the 5-year period from 2005 through 2010, 26 out of  73 countries, or slightly more than one-third, showed improvements in the  quality of their PFM systems, while 19 countries’ scores worsened. Most  countries, 62 percent in 2010, are clustered in the mid-range.

In addition to the quality of budgetary and financial management  indicator, the World Bank uses other broad PFM diagnostic tools,  including the Country Financial Accountability Assessments and the  Public Expenditure Reviews. The objective of the Country Financial  Accountability Assessments is to support the World Bank’s development  objectives by identifying strengths and weaknesses in country PFM  systems. The assessments are intended to help identify priorities and  inform the design and implementation of capacity-building programs. The  assessments also describe and analyze financial management and  expenditure controls, including expenditure monitoring, accounting and  financial reporting, internal controls, internal and external auditing, and  legislative review. Information obtained from the assessments, taken  together with that obtained from other World Bank diagnostic products  and other sources, supports the preparation of an integrated fiduciary  assessment. The results of these assessments inform the preparation of  the World Bank’s Country Assistance Strategy, particularly the sections  dealing with the size of the support program, the sectors to be supported,  selection of lending instruments, and approaches to risk management.  The assessments are particularly important where World Bank resources  are managed by the country’s own PFM system, as in the case of budget  support.

The Public Expenditure Review’s objectives are to strengthen budget  analysis and processes to achieve a better focus on growth and poverty  reduction, and to assess public expenditure policies and programs to  provide governments with an external review of their policies. Public  Expenditure Reviews may also address the incentives and institutions  needed to improve the efficacy of public spending in major sectors such  as health and education.

section_title: International Budget Partnership’s Open Budget Survey
paragraphs: The International Budget Partnership’s Open Budget Survey assesses  the availability in each country of eight key budget documents, as well as  the comprehensiveness of the data contained in these documents. The  survey also examines the extent of effective oversight provided by  legislatures and supreme audit institutions, as well as the opportunities  available to the public to participate in national budget decision-making  processes.

section_title: Appendix III: State Department Conducts Fiscal Transparency Reviews and Administers the Domestic Finance for Development Initiative
subsections: section_title: State Department Conducts Two Public Financial Management- Related Activities
paragraphs: The State Department (State) does not directly fund public financial  management (PFM) programs, but its Office of Monetary Affairs (OMA) is  responsible for two PFM-related activities. First, since 2008, U.S.  appropriations laws have required State to evaluate the fiscal  transparency of foreign governments receiving U.S. assistance. When  State determines it is important to the United States’ national interest, it  may approve waivers for countries that are deemed to be nontransparent  that allow U.S. agencies to provide assistance to these countries in  accordance with the legal prohibition against providing such assistance  without waivers. State processed 28 such waivers in 2011.

Second, State supports the presidential initiative called Domestic Finance  for Development (DF4D), which President Obama announced in 2011.  Under this initiative, State is to help countries use their own resources  and leverage other donor resources to meet development goals. State is  piloting this initiative in five countries: El Salvador, Honduras, Kyrgyzstan,  Tunisia, and Zambia. The goal of DF4D is to strengthen the commitment  to reform within partner countries; provide technical assistance in partner  countries, such as taxation expertise, including through innovative public- private partnerships; and elevate the importance and interrelation of  domestic resource mobilization, fiscal transparency, and anticorruption  efforts in public finance as key components for sustainable economic  development. Because State has not funded programs in the past for this  initiative, State has reached out to other organizations operating in these  countries, including international and bilateral organizations, to leverage  their programs and resources.

section_title: State Processes for PFM Activities Derive from Legal Requirements and Presidential Initiatives
paragraphs: State has developed processes for carrying out its PFM-related activities  required by legislative mandate and presidential initiative. Under the  Fiscal Transparency Review Process (FTRP), OMA reviews central  governments expected to receive bilateral economic assistance and  international security assistance for several dimensions of transparency:  publicly disclosed budget, budget breakdown by ministry, standards for  awarding natural resource contracts, and timely and accurate  documents. Each year, OMA reviews all countries it deemed  nontransparent in the prior year, and countries where recent events may  have affected fiscal transparency to evaluate whether they meet the  threshold of fiscal transparency. State uses the IMF’s definition of fiscal  transparency as a guideline for the FTRP. During this process, OMA  officials said they obtain information about the level of transparency of  each country by collecting information from U.S. embassy staff working in  those countries. In addition, OMA staff review publicly available reports  published by international organizations and civil society representatives,  such as the World Bank, International Monetary Fund, International  Budget Partnership, and the World Economic Forum. Because these  organizations do not always prepare country reports on an annual basis,  officials told us they use these reports as a check on their internally  generated information, rather than relying on them as primary sources of  information. For countries that State finds to be nontransparent, State can  issue a waiver that allows the country to receive otherwise restricted  assistance. As part of the process of requesting a waiver, embassy staff  in country develop action plans to assist the country in improving the level  of transparency in its budget process. The action plan should address  specific fiscal transparency issues identified in the transparency review  process and should include recommendations of short- and long-term  steps that the countries can take to improve budget transparency.  Embassies work with host governments to encourage implementation of  action plans, which may include activities such as continued daily  engagement with country officials working on budget reforms, providing  training and coordinating training on issues related to the budget process,  and funding local nongovernmental organizations to perform budget  oversight functions.

For OMA’s DF4D initiative, State officials, working with other agencies  and organizations, helped develop programs based on needs expressed  by country government officials that built on existing reform efforts and  development priorities. As an interagency effort, OMA helps identify and  leverage existing programs and resources, such as those of multilateral  organizations and other donors, such as USAID’s Innovation Fund and  the Global Health Initiative. For example, USAID is implementing a $7.6  million program to advance El Salvador’s fiscal reform agenda by building  capacity and improving systems for public expenditure and management  and tax revenue mobilization, promoting private sector engagement, and  creating a $2 million Revenue Challenge Fund to support improved tax  collection at the municipal level. With five countries piloting DF4D, State  plans to proceed by selecting additional partner countries based on  performance against quantitative DF4D-related measures, consultations  with posts, and expressions of interest from ministers of finance and other  economic leaders. Moreover, State plans to use action plans developed  for FTRP for countries to be considered for DF4D. State encourages  posts to report on PFM issues and opportunities to mobilize domestic  resources, raise the issues with relevant stakeholders in the public and  private sectors, and work with OMA staff to further the objectives of the  DF4D initiative.

section_title: State Is Taking Steps to Monitor Progress of Nontransparent Aid Recipients
paragraphs: State is beginning to conduct additional monitoring as part of its fiscal  transparency evaluations. Starting in 2012, State is requiring benchmarks  in its action plans for nontransparent countries so that it can compare  progress annually. According to State officials, some country action plans  had benchmarks but the quality of the benchmarks varied. The action  plan attempts to capture all the steps necessary to improve a country’s  fiscal transparency and includes more than just State actions. Lastly,  appropriations law for 2012 requires State to evaluate whether or not the  country has made progress toward improved fiscal transparency included  in any waiver request submitted. State guidance reports that progress  toward implementation of embassy action plans will factor into its decision  of whether to renew waivers.

section_title: Appendix IV: Examples of Public Financial Management-related Projects in Six Countries
paragraphs: To provide illustrative examples of U.S. projects to strengthen public  financial management (PFM) systems, we chose six case study  countries. For the U.S. Agency for International Development (USAID),  we selected projects in Kosovo, Liberia, and Peru and for the U.S.  Department of the Treasury (Treasury), we selected projects in  Cambodia, Honduras, and South Africa. We used the following criteria to  select the three countries for each agency:

The agency has relevant ongoing or recently completed projects  focused on strengthening PFM systems.

The countries represent different geographic regions.

The countries have different income levels, which may be associated  with different levels of government capabilities. For each U.S. agency,  we selected a country that the World Bank has classified as lower  income (Cambodia and Liberia), lower-middle income (Honduras and  Kosovo), and upper-middle income (Peru and South Africa).

For one PFM-related project in each country we summarized the project’s  background, and, for selected objectives, we summarized some of  USAID’s expected results and some of Treasury’s activities. These  examples are meant to be illustrative and not generalizable.
subsections: section_title: Selected USAID PFM– Related Projects
paragraphs: Tables 3 through 5 summarize selected USAID PFM-related projects in  Kosovo, Liberia, and Peru.

Kosovo Growth and Fiscal Stability Initiative  Time period: July 2010-July 2013  Award amount: $1,051,208 Description: Since 1999, USAID’s Kosovo economic policy and  institutional strengthening programs have focused on establishing key  central economic institutions and an enabling environment for private  sector growth. USAID is to adjust the focus of technical assistance as the  Growth and Fiscal Stability Initiative builds upon the experience and  lessons learned from the creation of reliable financial institutions in the  central government to address the fiscal stewardship challenges faced by  subnational governments. The initiative is to work with municipalities in  areas that are directly linked with the Ministry of Finance and Economy,  such as budget, treasury, property tax, and public private partnerships.

Liberia: Governance and Economic Management Support Program  (GEMS)

Time period: July 2011-June 2016  Contract amount: $44,902,679  Description: USAID-GEMS is to address significant governance  challenges remaining after USAID completed its previous capacity- building program in 2010. The program is to strengthen human and  institutional capacity within selected ministries, agencies, and  commissions. USAID-GEMS is to develop and maintain systems that  increase transparency and accountability, increase efficiency, reduce  expenditures, increase revenue, and limit corruption.

Peru: USAID ProDecentralization 2  Time period: August 2008-July 2012  Award amount: $10,644,432 Description: The USAID/Peru ProDecentralization project is in the  second phase. The first phase targeted national, regional, and municipal  institutions responsible for implementing the decentralization process.  The second phase seeks to continue to improve the policy framework at  the national level and strengthen the institutional capabilities of regional  and municipal governments. The national project aims to improve the  legal and policy framework for decentralization, including fiscal policies  that support a more equitable distribution of public resources. Regional  and local-level activities are to target technical assistance and training to  the diverse needs of Peru’s regional and municipal governments in  effectively administering resources and responding to citizens’ increasing  expectations. Under this project, existing services offered to subnational  governments in planning, budgeting, accountability, and institutional  strengthening are supplemented by new training and technical assistance  programs.

Tables 6 through 8 summarize selected Treasury Office of Technology  Assessment (OTA) PFM-related projects in Cambodia, Honduras, and  South Africa.

Cambodia: Technical Advisory Services on Budgeting  Time period: October 2009–September 2010  Description: The Cambodian government has been implementing a plan  to create a credible budget and improve accountability. In the current  phase of the plan, OTA and Cambodian government actions are focused  on decentralizing the reform effort to the line ministries and subnational  levels. These actions include expanding the use of strategic and program  budgeting, implementing a PFM information system, improving  macroeconomic forecasting, enhancing the linkage between the capital  and operating budgets, and improving overall financial accountability. The  overall aim is to improve the ability of the budget to be an instrument for  policy delivery and to support effective and efficient service delivery.

Honduras: Technical Advisory Services on Budgeting  Time Period: January 2011-December 2013  Description: OTA’s technical assistance focuses on implementing  international public sector accounting standards.

South Africa: Technical Advisory Services on Budgeting  Time Period: February 2007—January 2009  Description: Treasury OTA has conducted technical assistance projects  in South Africa in the areas of budget formulation, intergovernmental  finance, infrastructure budgeting, public finance training, and others since  1997. In 2006, the National Treasury of South Africa requested a new  OTA budget project to focus on the organization of an expenditure  program performance evaluation unit, performance information and cost  analysis, and support for the Collaborative Africa Budget Reform  Initiative. The central mission of the resident advisor was to establish an  expenditure program performance review system.

section_title: Appendix V: Comments from the U.S. Agency for International Development

section_title: Appendix VI: Comments from the Department of the Treasury

section_title: Appendix VII: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: Cheryl Goodman (Assistant Director), Michael Maslowski, Shirley Min,  RG Steinman, Michael Hoffman, Debbie Chung, Grace Lui, David  Dayton, and Etana Finkler made key contributions to this report.


Question-summary annotations: questions: question: What data do Treasury and USAID use in strengthening country government systems?
answer: To develop programs to strengthen developing countries’ Public Financial Management (PFM) systems, the U.S. Agency for International Development (USAID) and the U.S. Department of the Treasury (Treasury) rely on assessments of the host country government’s systems.

question: What new processes did USAID implement in 2011?
answer: In 2011, USAID implemented new processes that place a greater emphasis on PFM in its development efforts as the agency aims to increase its use of country systems to deliver assistance.
child_questions: question: How did these new processes differ from what the agency traditionally did?
answer: The agency traditionally included PFM capacity-building efforts only as components of broader programs, as it identified relevant weaknesses during the country assessment or program design process.

question: What do these new strategy and program development processes include?
answer: USAID’s new strategy and program development processes include a mandatory assessment of a country’s institutional capacity, including its financial systems, and a requirement to consider the use of country systems to deliver assistance. Most USAID country offices are required to develop a strategy using the new guidance by the end of fiscal year 2013.
child_questions: question: What are the steps involved in the process?
answer: Treasury’s process for developing programs begins with an initial assessment of the host country’s capabilities. Treasury staff then draft objectives for the program. For example, a Treasury program in Honduras set four objectives, including improving operational efficiency and enhancing accountability by strengthening the organization of the ministry of finance. Once in country, the advisor develops an annual workplan, outlining more specific goals aimed at meeting the overall objectives.

questions: question: What is the current state of USAID and Treasury's processes for evaluating PFM assistance?
answer: USAID and Treasury use several processes to monitor and evaluate their PFM assistance, but weaknesses exist.
child_questions: question: What do USAID's procedures include?
answer: USAID uses its regular procedures, which may include performance management plans, periodic progress reporting, site visits, and evaluations, to monitor and evaluate its PFM-related programs.
child_questions: question: What weaknesses are there in this monitoring system?
answer: Prior reports by USAID’s Inspector General and GAO have found weaknesses in USAID’s implementation of its monitoring procedures in other programs, including programs from the USAID offices that provide PFM assistance. In addition, USAID is currently unable to monitor overall progress toward its target to obligate 30 percent of its program funds through local systems by 2015. USAID, and GAO in prior reports, have identified a number of weaknesses in evaluation practice.
child_questions: question: How has USAID attempted to address these weaknesses?
answer: To address weaknesses the agency had identified, USAID adopted a new evaluation policy in January 2011 that states that all large projects are required to have an external evaluation, 3 percent of program budgets should be devoted to external evaluation, and evaluations must use methods that generate the highest quality evidence.

question: What do the Treasury's processes for monitoring and evaluating programs include?
answer: Treasury’s processes for monitoring and evaluating its programs include monthly reports, annual quantitative performance measures, voluntary customer feedback surveys, and on-site management reviews, but Treasury does not fully evaluate the performance of its completed technical assistance programs.
child_questions: question: What weaknesses have been identified with Treasury's quantitative performance measures?
answer: In addition, Treasury’s quantitative performance measures have been a useful project-level indicator of performance but have not been a useful indicator of overall performance due in part to inherent challenges associated with summarizing program performance and errors introduced when aggregating the performance data. Furthermore, a senior Treasury official reported that Treasury had not yet fully implemented a requirement to conduct independent postproject evaluations of its technical assistance programs.

questions: question: Why might it be in the best interest of developing countries to strengthen their PFM systems?
answer: Effective management of public resources can play an important role in a country’s development. In recent years, developing countries committed to strengthen their PFM systems and donors committed to use those systems as much as possible.

question: How does the United States provide assistance to strengthen PFM systems?
answer: The United States provides assistance to strengthen PFM systems primarily through USAID and Treasury.
child_questions: question: How does USAID strengthen PFM systems?
answer: USAID conducts capacity building activities to strengthen PFM systems as part of its development programs and has also set a target to obligate 30 percent of its annual assistance through local systems by 2015.

question: How does Treasury provide assistance?
answer: Treasury provides technical assistance through advisors who work in country, typically with the finance ministry.