Global Health: Global Fund to Fight AIDS, TB and Malaria Has Improved Its Documentation of Funding Decisions but Needs Standardized Oversight Expectations and Assessments
Released: 2007-05-07T00:00:00

Letter

Background

The Global Fund, a private foundation in Switzerland, was established in 2002 as a funding mechanism to collect public and private contributions to finance grants for the prevention and treatment of HIV/AIDS, tuberculosis (TB), and malaria. About 96 percent of the contributions to the Global Fund since its inception have come from governments. Private sector support increased in 2006, notably owing to a pledge of $500 million by the Bill and Melinda Gates Foundation and receipt of almost $19 million through the (RED)™ initiative.

By December 2006, the Global Fund had completed six proposal rounds, approved about $7 billion in grants, signed grant agreements worth over $5 billion and disbursed more than $3 billion to grant recipients in 132 countries. The Global Fund also reports that by December 2006, its grants enabled recipients to supply antiretroviral drugs to 770,000 people infected with HIV/AIDS, treat 2 million people infected with tuberculosis, and distribute 18 million insecticide-treated nets to combat malaria.

Global Fund Portfolio Distribution

The Global Fund’s underlying principles require, among other things, that it maintain a balanced grant portfolio in terms of region, disease, and type of intervention; operate transparently and accountably; and use a simple, rapid, and innovative grant-making process. As figure 1 shows, between 2002 and 2006, the Global Fund distributed 55 percent of grant funds to sub-Saharan Africa, with the remainder of the funding divided among other regions of the world. Also during this time period, the Global Fund distributed 57 percent, 15 percent, and 27 percent of grant funds, respectively, for HIV/AIDS, tuberculosis, and malaria programs. The remaining 1 percent funded projects to strengthen health systems.

In 2005, the Global Fund began tracking the allocation of grant funds in grant proposals according to the disease intervention categories of prevention, care, and treatment. Figure 2 shows the Global Fund’s estimated distribution by disease and intervention category.

Key Elements of the Global Fund’s Structure

The Global Fund’s principles also require it to focus on performance by linking its resources to the achievement of clear, measurable, and sustainable results. To achieve this focus on performance-based funding, the Global Fund relies primarily on oversight by, and interaction among, Global Fund secretariat personnel, grant recipients, and LFAs. (See fig. 3.)

Global Fund secretariat personnel. Within the secretariat, the fund portfolio managers are primarily responsible for reviewing grant progress and making performance-based funding recommendations. Other secretariat personnel are consulted during key stages in the grant life cycle. Fund portfolio managers interact with the LFA country teams providing oversight services for their grant, as well as with principal recipients, subrecipients, technical partners, and other stakeholders in the recipient countries. In addition to the country teams, staff from the contracts department and the Global Fund’s LFA manager interact with LFA headquarters teams, known as the central teams, regarding contracting and administrative matters and any problems related to LFA country teams.

Grant recipients. At the country level, the principal grant recipient is responsible for receiving and implementing the grant. During implementation, the principal recipient is responsible for monitoring and evaluating the grant’s effectiveness in accordance with grant objectives and making sure that funds are properly accounted for. The principal recipient is also responsible for overseeing the activities of any subrecipients implementing grant activities.

Local fund agents. The Global Fund currently contracts with LFA headquarters central teams at six organizations to oversee grants in 120 countries and multicountry programs. In the majority of cases, the central team establishes interfirm agreements with its affiliated offices within each country to perform LFA services. The LFA assesses recipient capacity prior to grant signing and performs verification and oversight duties throughout the life of the grant. The LFA central teams typically contract with external specialists for work outside the expertise of their organization, especially during the initial assessments of recipient capacity. For example, during the initial assessments, some organizations contract with a procurement expert to review recipients’ capacity for procurement and supply chain management. The Global Fund budgeted approximately $22 million—26 percent of the secretariat’s operating budget—to support the LFA structure for 2006 and spent about an additional $320,000 on LFA-related functions, including personnel costs related to LFA management, studies of the LFA model, and travel to meet with LFAs.

Performance-Based Funding Process

The Global Fund’s performance-based funding process involves an initial assessment of recipients’ capacity to manage and implement grants, periodic disbursement decisions based largely on reports by the grant recipient and the LFA, and a phase 2 grant renewal review after about 2 years.

Assessment of grant recipients’ capacity. Prior to funding a grant, the Global Fund considers, among other things, a grant recipient’s ability to (1) manage, evaluate, and report on grant-funded activities; (2) manage and account for funds; and (3) procure goods and services. LFAs assess prospective recipients’ capacity to perform these activities. Funding is conditioned on recipients’ meeting performance targets mutually agreed with Global Fund technical experts.

Periodic disbursements. After the initial screening and approval process, the grant recipient must produce quarterly or semi-annual reports describing financial and program progress as well as a funding request for the next disbursement period. The grant recipient sends these reports to the LFA, which prepares a report verifying the accuracy of the information, identifying potential grant problems, rating grant performance, and recommending a disbursement amount. The LFA then forwards the grant recipient’s report and its own report to the designated Global Fund portfolio manager. The fund portfolio manager compares the financial information and data on program progress with the grant’s performance targets, taking into account information such as the status of procurements and political or economic constraints. The fund portfolio manager’s team leader makes a funding decision if the disbursement is less than $2 million. Figure 4 illustrates the Global Fund’s periodic disbursement decision process.

Grant renewal process. A key milestone in the performance-based funding approach is the phase 2 grant renewal review—a detailed analysis of grant progress at about 2 years, required to continue funding for up to the next 3 years. Similar to its process for disbursement decisions, the Global Fund’s grant renewal process requires grant recipients to send documentation of financial and program progress to the LFA, which verifies the information and analyzes the phase 2 proposal. In a report to the Global Fund, the LFA identifies potential grant problems, rates the performance of the grant, and recommends the amount needed to fund up to the next 3 years of the grant. Because phase 2 is the decision point for continuing the grant to its 5-year completion, the process also includes an extensive and detailed review of cumulative information by fund portfolio managers, their supervisors, and evaluation and finance staff. The phase 2 renewal process culminates in a panel discussion and final recommendation to the board. Figure 5 illustrates key elements of the Global Fund’s phase 2 grant renewal process.

Table 1 outlines the key source documents related to disbursement decisions and phase 2 grant renewals from electronic “grant files” that we reviewed in conducting this study.

In preparation for its retender of LFA contracts worldwide in mid to late 2007, the secretariat will continue discussions on the key principles that will potentially shape the design of the next LFA model. According to a Global Fund official, they will take into consideration findings from this review and two internal Global Fund studies. They expect to issue the request for applications in late June.

Global Fund Has Improved Documentation of Its Basis for Funding Decisions

The Global Fund has improved its documentation of information supporting its performance-based funding decisions. For example, the Global Fund’s system now requires that portfolio managers consistently document factors that may affect the decision to continue funding a grant, especially when the grant falls short of its performance targets. In addition, the Global Fund grant files that we reviewed consistently contained explanations of funding decisions and contextual data informing the decisions. Although we found many reports submitted by grant recipients to the Global Fund do not provide explanations for all unmet grant performance targets, according to Global Fund officials, fund portfolio managers are able to obtain the needed information directly from the grant recipients or other stakeholders.

Global Fund Has Improved Its Processes for Documenting the Basis for Funding Decisions

The Global Fund has developed a more systematic process for documenting the basis for its funding decisions. In June 2005, we noted that the Global Fund did not consistently document the reasons for its determinations that recipients’ performance warranted additional funding for periodic disbursements or its recommendations regarding grant renewals. We recommended that the Global Fund continue efforts to clearly document the Global Fund’s reasons for periodically disbursing funds and renewing grant agreements. Subsequently, to increase the transparency of its performance-based funding system, the organization made several changes to its documentation process.

In late 2005, the Global Fund developed a disbursement decision-making form to ensure that fund portfolio managers consistently show the basis for each disbursement decision. The form requires, among other things, an explanation of the performance rating assigned to the grant, addressing any discrepancy between ratings by the fund portfolio manager and the LFA and verifying that the manager has considered any problems raised by the LFA regarding data quality and reporting, program progress, and expenditures; and contextual information critical to the disbursement decision, such as information demonstrating governance problems, grant management weaknesses, currency fluctuations, or natural disasters in the country or region where a funded project is being implemented.

The Global Fund also revised other forms to provide more systematic reporting of grant program progress. These revisions included improvements to the grant progress update and disbursement request, the LFA progress review and recommendation report, and the phase 2 grant scorecard. For example, the updated progress update form contains a new requirement that LFAs comment on the reliability of grant recipient data on target achievement. The updated grant scorecard includes requests for data showing the timeliness and effectiveness of grant recipients’ use of the funds during phase 1 of grant implementation. It also includes a new section where the secretariat can record plans to materially change the grant in phase 2—for example, adding or dropping goals or objectives because of lessons learned during phase 1.

According to Global Fund officials, the changes to their system of recording information about grant performance and problems have made the decision process more systematic and transparent. For example, Global Fund officials, including some fund portfolio managers, said the new disbursement decision-making form provides managers a place to systematically explain their ratings. Officials from OGAC and the World Bank with whom we met said that the Global Fund’s more consistent documentation of the basis of its decisions has made the decisions easier to understand.

Grant Files Demonstrate Improved Documentation Supporting Funding Decisions

Our review of random samples of disbursement and phase 2 grant files indicated that most files systematically explain a number of key factors used to support disbursement and grant renewal decisions. In our 2005 report, we observed that the Global Fund’s grant files provided little explanation of disbursement decisions, including decisions to disburse funds to recipients who reported not meeting their targets. In addition, our 2005 report noted that, according to Global Fund officials, many grant renewal decisions were not supported by the information needed to assess whether funding decisions were consistently based on grant performance.

In our current review, we found that Global Fund grant files consistently contained explanations of ratings of grant performance that were important to funding decisions, even when key stakeholders’ ratings and recommendations did not concur. Based on our samples, 100 percent of both disbursement and phase 2 files contained narratives explaining the manager’s rating of the grant’s performance. Most grant files also contained explanations for lack of concurrence in ratings or recommendations by the various officials charged with assessing grant performance. For example: In all 8 cases in which fund portfolio managers’ phase 2 grant ratings differed from those of the evaluation staff, the scorecards in the grant files contained an explanation for the lack of concurrence.

In all 5 cases in which fund portfolio managers’ phase 2 grant renewal recommendations differed from those of the LFAs’, the scorecards in the grant files contained an explanation for the lack of concurrence.

In cases in which fund portfolio managers’ assessments of grant performance led them to disagree with the disbursement amount recommended by the LFA or the amount requested by the grant recipient, 31 of 32 of those grant files included a narrative explaining the discrepancy.

According to Global Fund officials, such explanations provide valuable information that helps decision-makers better understand and consider grant performance.

The Global Fund files that we reviewed also consistently explained contextual information and other issues that may affect the grant recipient’s ability to meet agreed-upon targets. We found that an estimated 96 percent of disbursement files in which portfolio managers had specified contextual issues that negatively affected grant performance included narratives explaining each issue. According to Global Fund officials, such information is crucial to grant disbursement and renewal decisions, especially if recipients have not met many of the targets set at the grant’s initiation. In addition, in reviewing 43 disbursement files for grants whose recipients had not yet met remedial requirements imposed by the Global Fund as conditions for grant renewal, we found that 42 (98 percent) contained an explanation of recipients’ progress toward meeting the requirements.

Global Fund Informally Gathers Missing Data on Grant Recipients’ Performance

Although recipients’ formal documentation of grant performance remains incomplete, Global Fund officials informally gather the missing information that they consider necessary for making and documenting the basis for funding decisions. In June 2005, we reported that recipient progress reports varied in quality and that the limited monitoring and evaluation capabilities of many recipients raised questions about the accuracy of their reporting. Our current analysis found that an estimated 39 percent of phase 2 grant files contained evidence, identified by fund portfolio managers, of weaknesses in the reliability of the grant recipient’s data. In contrast to the consistent documentation we found on forms completed by Global Fund staff, many periodic performance reports completed by grant recipients lacked explanations either for not meeting or for exceeding the grant’s targets. In those cases, according to LFAs, fund portfolio managers, and other Global Fund officials, fund portfolio managers relied on regular phone and e-mail contact with grant recipients or other stakeholders to obtain the missing information needed for performance-based funding decisions. Global Fund officials said that information crucial to funding decisions is documented on the disbursement decision-making form or phase 2 grant scorecard, even if it was not recorded by grant recipients or LFAs in their formally required reports.

Global Fund portfolio managers said that recipients’ explanations for not meeting targets may influence the portfolio manager’s recommendation to withhold funds or, alternately, to continue to fund the grant despite performance problems. According to Global Fund officials and public health experts, improving grant recipients’ monitoring and evaluation capacity, as well as their ability to report accurately and completely, will require long-term investment on the part of the Global Fund and its partners.

Global Fund Did Not Implement Earlier Risk Model but Is Developing Comprehensive Risk Framework That Includes Early Alert and Response System

The Global Fund did not implement the risk assessment model that it was developing at the time of our 2005 report, because it determined that the model did not accurately identify grant risk. The organization relies on elements of its structures and processes, including its initial technical review, disbursement decision-making form, periodic grant ratings, CCM oversight, information from technical partners, and LFA oversight, to identify potential risks. Recognizing the need to establish a more comprehensive approach to risk management, the Global Fund has begun developing a risk assessment framework for the organization that includes an early warning and response system to address poorly performing grants.

Global Fund Did Not Implement the Grant Risk Assessment Model Described in the 2005 GAO Report

The Global Fund did not implement the risk-assessment framework to identify poorly performing grants that it was developing at the time of our 2005 report. We reported that the organization had devised a risk assessment model and an early warning system to identify poorly performing grants and more systematically alert portfolio managers when they needed to intervene. According to the Global Fund, because it disburses grants in countries with varying levels of economic development and health systems capacity, the risk model was to incorporate quantitative data, such as grant size and performance as well as country development and corruption data. Using these quantitative indicators, the model would generate reports that indicated potential problems, and portfolio managers would evaluate the reports in light of contextual information. According to an official who participated in developing the framework, it was not implemented because it relied too heavily on quantitative indicators and did not adequately account for contextual factors, leading to risk ratings that were too inaccurate to be useful. For example, according to Global Fund officials, in a test of the model, a particular grant was rated as low risk based on the host country’s development and corruption indicators. However, the grant ultimately failed and was not renewed at the phase 2 grant renewal review.

Global Fund Applies Elements of Risk Management but Lack of Risk Assessment Model Limits its Ability to Respond to Poorly Performing Grants

Although the Global Fund’s grant oversight incorporates several elements of risk management, the absence of a risk assessment model limits its ability to respond methodically to poorly performing grants and use available resources to increase oversight as needed. According to Global Fund officials, the organization currently relies on existing oversight mechanisms such as its initial technical review of grant proposals, standardized decision forms, periodic grant ratings, CCM oversight, technical partners’ assistance, and LFAs’ reviews to identify and assess potential mismanagement, irregularities, or other risks to successful grant implementation.

Technical review. The technical review panel raises important technical issues that grantees must address when presenting their proposals to the panel. Proposals with a very high technical risk profile generally are not recommended for funding.

Disbursement decision-making form. Grant risk is mitigated by the fund portfolio manager’s use of the standard disbursement decision-making form, which includes financial and program data from the recipient that has been verified by the LFA.

Rating grant progress for each disbursement request. Since 2006, according to Global Fund officials, every progress and disbursement request, after the first disbursement, has been graded on a scale of A, B1, B2, or C. This information is posted on the Global Fund Website’s financial data base and included in grant performance reports that are updated on the Global Fund website at each disbursement. This rating system is a key tool for identifying grants at risk and for sharing information with technical partners.

Country coordinating mechanism (CCM). A key responsibility of the CCM is to continue to monitor grant progress during implementation and identify problems that may affect progress. In many countries, Global Fund technical partners such as UNAIDS, WHO, and USAID are members of CCMs.

Partner assistance. Global Fund officials rely on technical partners with staff based in recipient countries to provide contextual information on grant performance, identify risks, and help respond to problems that affect grant implementation.

LFA reviews. Global Fund officials emphasized that LFAs identify potential grant risks in their periodic verification of grant progress and during the cumulative review and performance assessment for the phase 2 renewal process.

The absence of a model to assess grant risk has limited the Global Fund’s ability to methodically identify poorly performing grants and use available resources to increase oversight as needed. According to a senior manager from an LFA with one of the largest grant portfolios, because the Global Fund has not assessed grants as high, medium, or low risk, LFA oversight has not differed substantially by grant risk level and LFA oversight resources have not been used to optimal effect. For example, although some grants may require more oversight because of potential misuse of grant money or limited recipient capacity, LFA task orders and fees are generally no different for potentially problematic grants than for grants with few expected problems. Similarly, the frequency of LFA visits to project sites for monitoring and data verification has not been based on a systematic estimate of risk.

Global Fund Is Developing a Comprehensive Framework for Risk Management and Has Established First Components

Global Fund officials informed us in April 2007 that they are developing a top-level framework for risk management that will cover all aspects of the organization’s strategic direction, including addressing resources and skills at the secretariat level, developing a model for systematically assessing grant risk, and fully implementing an early alert and response system. Global Fund officials have acknowledged the need for a quantitative model for ranking grants by level of risk, supplemented by additional qualitative factors that may change a grant’s rating or ranking, to identify and mitigate grant risk in a timely manner. The Global Fund recently organized a risk seminar for directors and senior managers and engaged an international firm to help them achieve consensus on a shared approach to comprehensive risk management. We reviewed a draft outline of the guiding principles for developing a risk management framework that the Global Fund developed in February 2007. As of April 2007, the Global Fund is in varying stages of progress in developing and implementing detailed risk strategies.

According to Global Fund officials, the secretariat has put in place the first components of its early alert and response system intended to identify as early as possible grants that are performing poorly and facing implementation difficulties. These first components include posting the grant disbursement and renewal ratings on the Global Fund Website and efforts to engage more technical assistance from partners for poorly performing grants. Global Fund officials recognize that for the system to work effectively principal recipients, CCMs, stakeholders, and partners must share responsibility for tracking the progress of grant implementation and take appropriate action when progress is impeded. Global Fund officials stated that the early alert and response system will eventually include documenting and disseminating best practices, formalizing the system within the secretariat and among country-level stakeholders and partners, and increasing the engagement of partners in grant oversight.

Global Fund Has Limited Ability to Manage and Oversee Local Fund Agents’ Performance

The Global Fund lacks the information it needs to manage and oversee the LFAs because of a lack of systematic assessments of LFAs’ performance. Good practice suggests that organizations monitor their contractors’ performance during the course of interaction and periodically evaluate performance based on clear expectations. Assessment results should include consistent patterns of rewards, penalties, or both. (See app. II for sources of these principles.) Global Fund portfolio managers set expectations for LFA performance informally and individually, frequently leading to inconsistent expectations and resulting in challenges for both the LFAs and their managers. Moreover, the Global Fund does not require formal assessments or reports of LFA performance; instead, LFAs are assessed informally and irregularly, and assessment results are not systematically documented. As a result, the Global Fund has limited records of, and data on, LFA performance. Although the Global Fund made a previous attempt to assess LFAs more systematically, this effort was unsuccessful. According to secretariat staff, key partners, and outside experts, LFAs’ performance and capacity vary.

Global Fund Has Not Standardized Its Expectations for Local Fund Agent Performance

Rather than establish standardized performance expectations for all LFAs, the Global Fund portfolio managers establish expectations informally for the quality of each LFA’s performance. The Global Fund’s contracts with LFAs, as well as the guidelines that it provides, outline the tasks that all LFAs must complete, such as considering a principal recipient’s programmatic and financial implementation progress in relation to its plans. The contracts also contain general statements about LFA performance; for example, a sample LFA contract specifies that all tasks should be carried out to the satisfaction of the Global Fund. However, Global Fund policy states—and secretariat staff and 11 of the 12 LFA country teams that participated in our structured interview confirmed— that the fund portfolio manager is primarily responsible for defining satisfactory performance for his or her designated LFAs. As of March 2007, the Global Fund had not provided fund portfolio managers any specific training or written guidance defining acceptable or good LFA performance, including the quality of their reporting.

The lack of standardized performance expectations results in challenges both for the LFAs and for Global Fund management.

LFA challenges. Because fund portfolio managers set expectations informally and individually, LFA country teams frequently encounter different expectations from successive portfolio managers. For example, among the LFA country teams who participated in our structured interviews, 6 of 10 teams who had interacted with more than one fund portfolio manager said that the managers’ criteria for a high-quality report had varied. According to the respondents, variations in fund portfolio managers’ expectations often involved differences in the scope of work that they expected LFAs to complete. For instance, members of one country team said that their previous fund portfolio manager had wanted the team to focus primarily on financial data in its reports, while the current manager wanted a focus on programmatic issues. In another country, one fund portfolio manager wanted the team to conduct on-site verification of grant subrecipients, while a subsequent manager did not consider site visits a priority. In the remaining four cases, country teams did not report substantial variation between different fund portfolio managers. In one of these cases, the team said that a previous manager established expectations that subsequent managers used in turn, therefore providing continuity.

Management challenges. The lack of standardized LFA performance expectations also presents challenges among those responsible for managing the LFAs. According to the manager of one LFA central team with a large grant portfolio, variations in Global Fund expectations made it difficult to establish best practices for its LFA country teams. Likewise, several Global Fund officials noted that secretariat staff have conflicting ideas about LFA performance criteria. In one instance, according to Global Fund officials, a group of fund portfolio managers were unable to agree on what constituted a high-quality report. Fund portfolio managers and other key officials observed that more standardization of expectations would be beneficial, although some noted that this standardization should be balanced to allow them to maintain flexibility to account for country context.

Global Fund Does Not Assess LFAs Systematically and Lacks Formal Records of LFA Performance

The Global Fund does not require formal, regular assessments of LFA performance. As a result, it lacks LFA performance records and cannot systematically track performance.

LFA Performance Assessment and Reporting Are Informal and Irregular

Although Global Fund officials regularly review the reports they receive from the LFAs and provide informal feedback to country teams, they assess LFA performance informally and at irregular intervals. All fund portfolio managers that we interviewed, as well as 11 of the 12 LFA country teams that participated in our structured interviews, said that fund portfolio managers interact informally and frequently with country team members, often orally or by e-mail. In addition, all but one country team said that fund portfolio managers provide informal feedback on their performance. For example, members of one team noted that they received feedback from their fund portfolio manager for each report and requested more information as necessary. In another case, the manager provided feedback to the LFA central team, which passed the feedback to the country team. A majority of the country teams reported receiving feedback from their fund portfolio manager when he or she traveled to the recipient country, at least once a year. Fund portfolio managers told us that if they continually encounter problems with a country team, they inform the Global Fund’s LFA manager, who discusses the issue with the central team; if this does not resolve the problem, the contract may be terminated. A key Global Fund official stated that two to three LFA country team subcontracts are terminated each year based on information from fund portfolio managers on their poor performance.

Although LFA central teams provide some performance feedback to their country teams, the scope and frequency of that feedback varies depending on the LFA’s structure. Ten of the LFA country teams that participated in our structured interviews told us that LFA central teams provide informal feedback about their performance. The central team at two of the six LFA organizations reviews all products before they reach the secretariat. At two other LFA organizations, reports are reviewed only if it is deemed necessary, and in those cases, regional coordinators or mentors may review the reports. At the remaining two LFA organizations, the distinctions between the country and central teams are less clear, because the reports are at least partially written by central team members.

Lack of Assessment Records Limits Available Information on LFA Performance

The lack of formal assessments limits the Global Fund’s access to the information needed to manage LFAs’ performance. The fund portfolio managers we interviewed said that their frequent interaction with country teams generally allows them to know which teams are performing well and which teams need improvement. However, Global Fund officials with LFA management responsibility stated that formalizing the process would allow the agency to establish LFA performance records and identify situations in which more oversight of LFAs by the Global Fund may be required. A working group of Global Fund experts echoed these concerns, reporting that despite extensive debate about the LFA model, the Global Fund had conducted little systematic assessment of the performance of individual LFA country teams.

Global Fund Previously Attempted to Implement Systematic LFA Assessments

A prior attempt to introduce a tool to make LFA assessments more systematic was not successful, because use of the tool was not mandatory and fund portfolio managers did not see it as a priority. According to the Global Fund, the LFA manager conducted a pilot of an evaluation checklist as a tool for fund portfolio managers to systematically assess LFA performance. Secretariat staff stated that, because Global Fund management did not require fund portfolio managers to complete the checklist, very few portfolio managers used it. As of early 2007, the LFA manager was developing a revised tool to routinely collect staff input on LFAs. However, it is too early to determine whether this tool, when finalized and made available to staff, will capture the information needed to assess LFA performance. According to the LFA manager, integrating this tool into existing tasks and making it an explicit component of fund portfolio managers’ job descriptions would increase its use.

LFA Performance and Capacity Reportedly Vary

Although Global Fund officials cited instances in which LFA country teams performed adequately, secretariat staff, key partners, and outside experts expressed concerns about some LFAs’ capacity to oversee grant recipients’ programmatic and procurement activities. To illustrate adequate LFA capacity and performance, secretariat staff identified cases where LFA country teams have quickly identified problems with grants. For example, the majority of fund portfolio managers we interviewed noted that most LFA country teams performed financial oversight competently. Similarly, a procurement officer described a country team that focused on supply management issues throughout the life of the grant and proved instrumental in resolving a large corruption problem. One fund portfolio manager described a country team that had enough knowledge of monitoring, evaluation, and procurement to understand an LFA subcontractor’s initial assessment of recipient capacity and, therefore, could use this information in subsequent reporting about grantee progress.

Conversely, secretariat staff also cited multiple instances in which LFAs demonstrated lack of capacity that affected the conduct of their oversight duties. In a recent Global Fund survey, secretariat employees who identified themselves as having sufficient exposure to LFAs to form an opinion reported mixed experiences regarding LFAs’ technical program and monitoring expertise. About half of the respondents disagreed with the statement that LFAs with whom they had worked provided “adequate” expertise on health and program issues. The remainder was about equally divided between those who agreed that health and program expertise was adequate and those who neither agreed nor disagreed. Global Fund employees’ assessments of the adequacy of LFAs’ expertise for monitoring and evaluation were similarly mixed. The survey also provided evidence that the service quality varies greatly among countries: about three- quarters of the staff agreed with the statement that “within LFA firms there are substantial differences in service quality across countries.”

In addition, Global Fund partners and outside experts have raised concerns about LFAs’ oversight capacity. UNAIDS officials said that they have serious doubts about the capacity of LFAs to monitor program progress, specifically noting that LFAs have little understanding of how to weigh and balance various performance targets that are not of equal importance. For example, according to these officials, an LFA that lacks strong technical program capacity might not recognize that a recipient exceeding a simple performance target, such as training staff, may be trying to compensate for poor performance in more difficult target areas, such as administering drugs to treat AIDS, TB, or malaria. Several other sources—a report commissioned by the Global Fund, our 2005 report, and several reports by an acknowledged expert on the Global Fund—also note problems with LFA capacity. (See app. II for a list of sources.)

Conclusions

Since our report in June 2005, the Global Fund has taken steps to improve operations at its secretariat in Geneva while it continues to confront complex issues facing grant implementation at the country level. The Global Fund will continue to face important challenges, including evaluating the effectiveness of its investment of the billions of dollars it receives annually from the United States and other donors and the impact of its programs over time. The Global Fund has improved the transparency of its periodic disbursement and grant renewal decisions by providing better documentation in its grant files. Delays in developing a systematic approach to assessing grant risk have left the Global Fund vulnerable to some of the risks associated with implementing grants in developing countries. However, the Global Fund board recognizes the importance of developing a more comprehensive approach to risk management, and the secretariat has recently drafted principles to guide its development of a risk framework for the organization. In addition, the Global Fund’s lack of a formal process for setting expectations and assessing and analyzing LFA performance constrains its ability to determine the quality of the LFAs’ services, track LFA performance over time, and address ongoing capacity issues. The lack of a formal assessment process also limits the information available to the Global Fund during its upcoming global retender of LFA contracts.

Recommendations for Executive Action

To enhance Global Fund operations, we recommend that the Secretaries of State and HHS work with the Global Fund’s Board Chair and Executive Director to take the following two actions: establish standardized expectations for LFA performance, and require systematic assessments of LFA performance and the collection and analysis of LFA performance data to improve the management and oversight of LFAs.

Agency Comments

Written comments on a draft of this report from the Departments of State and Health and Human Services and the U.S. Agency for International Development are represented in the unified response received from the Office of the U.S. Global AIDS Coordinator. The Global Fund also provided written comments on the draft. Both entities agreed with the report’s findings, conclusions, and recommendations. The Office of the U.S. Global AIDS Coordinator stated that the report accurately reflected areas in which the Global Fund has made progress as well as areas in which more work remains to be done. The Office’s response noted that the United States will urge the Board to adopt concrete recommendations from several studies to improve the accountability and transparency of Global Fund activities and will work with the Global Fund to improve its oversight and risk assessment mechanisms. The Global Fund stated that the report reinforced the importance of measures it has begun taking to better measure local fund agent performance. Regarding risk assessment, the Global Fund emphasized the role of its country-based and global partners in helping to track grant implementation and provide technical support as needed. (See apps. III and IV for their comments.)

We are sending copies of this report to interested congressional committees and will make copies available to others on request. In addition, this report is available on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

To assess the Global Fund’s efforts to improve documentation of performance-based-funding decisions, we analyzed the forms used to collect data on grant performance and discussed with Global Fund officials how the forms have changed since GAO’s 2005 report. We met with officials from all of the entities that have current contracts to serve as local fund agents and with officials from the Office of the Global AIDS Coordinator, UNAIDS, and the World Bank—-the Global Fund’s key technical partners—to discuss how documentation of the Global Fund’s decisions has changed and whether it has improved.

To review information in the grant files used to support performance- based funding decisions, we analyzed two separate simple random probability samples of grant funding decisions using a separate data collection instrument for each sample. The first sample consisted of 80 of the 348 grants that the Global Fund assessed for performance-based funding disbursements between January 1 and October 1, 2006. The second sample consisted of 45 of the 65 grants that the Global Fund assessed for phase 2 grant renewal during the same time period. We chose January 2006 as the starting point for our sample because Global Fund officials told us that their new disbursement decision worksheet—created to address GAO’s recommendation that they better document their decisions—was fully implemented by the end of 2005. With our two probability samples, each member of the study population had a nonzero probability of being included, and that probability could be computed for any member. Each sample case file was subsequently weighted in the analysis to account statistically for all the case files in the population, including the files that were not selected.

Because we followed a probability procedure based on random selections, our samples are only one of a large number of samples that we might have drawn (for both the grant disbursement and phase 2 file reviews). Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as 95 percent confidence intervals. These are intervals that would contain the actual population values for 95 percent of the samples we could have drawn. All percentage estimates provided in this report based on our samples have 95 percent confidence intervals of within plus or minus 13 percentage points.

For each grant in our disbursement sample, we requested the grant recipient’s Progress Update and Disbursement Request report, the Local Fund Agent Recommendation report, and the Global Fund’s Disbursement Decision-Making Worksheet. For the phase 2 decision sample, we requested the following documents for each grant: Phase 2 Grant Scorecard (the in-house version that goes to the panel)

Phase 2 Grant Scorecard (the panel’s version that goes to the board)

Documentation of the board’s decision with any explanation or justification of the decision (whether this is in the form of a scorecard or some other format)

Annex A Attachments 2 and 3 Local fund agent Phase 2 Assessment Report We gathered the information from the documents for our data collection instruments and verified the data that we used to support our findings.

Our file review consisted of a number of checks on the presence or absence of an array of qualitative and contextual information that the Global Fund forms specify as important for grant decision-making and that therefore should be provided by those responsible for filling them out. We did not evaluate the appropriateness or accuracy of the judgments made on the basis of the narrative or contextual information provided in the documents; we focused on the issue of whether the contextual or narrative information provided was relevant to particular judgments or decisions.

In our analysis of the disbursement files, while we examined information contained in other documents, we focused mostly on the Disbursement Decision-Making Form, due to the fact that this was the key form that the Global Fund instituted for disbursement-related decision-making since our last report. In this form, we checked for the presence or absence of the following pieces of information: Fund portfolio manager/cluster explanation for differences in fund portfolio manager/cluster and LFA ratings Fund portfolio manager’s explanation for performance rating Fund portfolio manager’s explanation for differences in funding decision with the principal recipient request and LFA recommendation Fund portfolio manager’s check-offs on an array of grant decision-making issues such as performance evaluation, financial considerations, future and outstanding requirements, etc.

Fund portfolio manager’s explanation for array of contextual issues (e.g., governance, general management, and external factors)

Fund portfolio manager’s explanation for required documents and In the phase 2 files we focused on the Grant Scorecard, as this is the key Global Fund decision-making document for this phase, although we performed checks on several other documents as well. We checked for the presence or absence of the following pieces of information in the following phase 2-related documents; all items are for the Grant Scorecard unless otherwise noted: Evaluation unit’s explanation for nonconcurrence with fund portfolio General explanation for difference between fund portfolio manager/cluster recommendation and the LFA’s recommendation Evaluation unit’s explanation for difference between fund portfolio manager/cluster recommendation and the LFA’s recommendation General explanation for difference in the recommended amounts between Global Fund financial review and cluster leader phase 2 General explanation for difference in the recommended amounts between LFA and fund portfolio manager/cluster leader (i.e. the Global Fund) phase 2 Fund portfolio manager/cluster identification of material issues with reliability of principal recipient’s data Fund portfolio manager/cluster explanation for phase 2 rating Fund portfolio manager/cluster explanation for phase 2 recommendation (LFA phase 2 assessment) report-specific conditions identified by LFA for ‘Conditional Go’ recommendations Time-bound actions identified by portfolio manager/cluster for ‘Conditional Go’ recommendations To assess the Global Fund’s progress in implementing a risk assessment model and early warning systems for its grants, we followed up on the risk assessment model and early warning system described in our 2005 report with Global Fund officials. In addition, we reviewed international standards for risk that establish the criteria for assessing risk developed by the Committee on Sponsoring Organizations of the Treadway Commission. We also discussed the need for a risk assessment model and early warning system with key Global Fund staff, key Global Fund partners, UNAIDS, World Bank, and OGAC, and with the Global Fund Inspector General and LFA officials. In addition, we reviewed Global Fund board notes and the Global Fund’s draft outline of Guiding Principles for a Risk Management Framework. In their technical review of our draft, Global Fund officials clarified the distinction between the risk assessment model that was not implemented and the current early alert and response system, and they provided additional information on the current status of the early alert and response system. We modified the risk assessment section of the draft based on our analysis of this information.

To assess how the Global Fund oversees and measures the capacity and performance of local fund agents, we established criteria for good practices in performance assessment, consulted key reports and documents, met with relevant personnel, conducted structured interviews with LFA country teams, and analyzed the structured interview data using a content analysis framework. We consulted GAO experts in performance assessment and contracting, good practice documents from the Government Performance and Results Act, performance-based contracting literature, internal control literature, and training development literature. (See app. II) To understand Global Fund policies and requirements regarding LFA performance, we reviewed Global Fund documentation, including guidelines for LFA work, the Global Fund’s operational policy manual, LFA manager terms of work, and LFA framework contract forms, as well as relevant evaluative reports published by the Global Fund and development experts. We meet with key Global Fund officials in Geneva, Switzerland, including the fund portfolio managers and members of the Portfolio Services and Projects Team, representatives from all LFA central teams, selected LFA subcontractors, and recognized experts on the Global Fund to discuss the current practices and potential difficulties of the LFA system. We also reviewed the methodology and results of a survey of Geneva-based Global Fund secretariat staff who are users of LFAs conducted by a Global Fund contractor in November of 2006.

Additionally, to support our review of issues related to LFA capacity and performance, we conducted structured telephone interviews with a nonprobability sample of 12 LFA country teams out of a total of 102 LFA country teams worldwide. We selected country teams for this sample based on criteria such as the size and amount of the grants that the LFA oversaw, whether the LFA was located within the country, and whether the LFA acted as a mentor to other LFA country teams. The purpose of these 12 interviews was to obtain information on the views of these important participants in the Global Fund process; we did not obtain a representative perspective of the views of all 102 country teams. We pre- tested our questions with two of our initial respondents and refined our questions based on their input. We followed up via e-mail with our respondents to obtain supplementary information. To summarize the open-ended responses and developed categories for the analysis, we first grouped interview responses by general categories, based on our objectives, criteria, and inductive observation. Within each of these general categories, we then further grouped the data into subcategories based on the emerging themes. To ensure the validity and reliability of our analysis, these subcategories were reviewed by a methodologist, who proposed modifications. After discussion of these suggestions, we determined a final set of subcategories. An analyst and a methodologist independently tallied the number of respondents providing information in each subcategory, compared their tallies, and reconciled any variation.

Appendix II: Other Sources Cited

GAO, Strategic Issues Web pages on the Government Performance and Results Act (GPRA)

GAO, Internal Control Management and Evaluation Tool. GAO-01- 1008G. Washington D.C.: Aug. 2001.

Office of Federal Procurement Policy, Seven Steps to Performance-Based Services Acquisition, see: www.acqnet.gov/Library/OFPP/BestPractices/pbsc GAO, Contract Management: Guidance Needed for Using Performance- based Service Contracting. GAO-02-1049. Washington, D.C.: Sept. 23, 2002.

Analyses of LFA Capacity and Performance

GAO, Contract Management: Guidance Needed for Using Performance- Based Service Contracting. GAO-02-1049. Washington, D.C.: 2002.

GAO, Internal Control Management and Evaluation Tool. GAO-01-1008G.

Washington, D.C.: 2001.

Kruse, Stein-Erik, and Jens Claussen. Review of the Roles, Functions and Performance of Local Fund Agents. Centre for Health and Social Development and Nordic Consulting Group; 2004.

Office of Federal Procurement Policy, Seven Steps to Performance-Based Services Acquisition. See: www.acqnet.gov/Library/OFPP/BestPractices/pbsc Radelet, Steven et al. Challenges and Opportunities for the New Executive Director of the Global Fund: Seven Essential Tasks. Global Fund Working Group, Center for Global Development, Washington D.C.: October 2006.

Radelet, Steven. The Global Fund to Fight AIDS, TB, and Malaria: Progress, Potential, and Challenges for the Future. Center for Global Development, Washington D.C.: 2004.

Appendix III: Comments from the Global Fund

Appendix IV: Comments from Departments of State and Health and Human Services and U.S. Agency for International Development

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the person named above, Audrey Solis, Assistant Director; Tom Zingale; Jeffrey Baldwin-Bott; Bruce Kutnick; Barbara Steel-Lowney; David Dornisch; and Reid Lowe made key contributions to this report. Kathryn Bernet, Antoine Clark, Etana Finkler, Hynek Kalkus, and Tracy Guerrero provided technical support.