Global Health: Global Fund to Fight AIDS, TB and Malaria Has Improved Its Documentation of Funding Decisions but Needs Standardized Oversight Expectations and Assessments
Published: 2007-05-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Global Fund to Fight AIDS, Tuberculosis and Malaria has approved about $7 billion in grants to developing countries; the U.S. has contributed $1.9 billion. The State Department's Office of the Global AIDS Coordinator (OGAC) coordinates the U.S. government's overseas AIDS programs, with participation from the Department of Health and Human Services (HHS) and the U.S. Agency for International Development (USAID). In 2003, Congress directed GAO to report on the Global Fund every 2 years. This report assesses the Global Fund's (1) documentation of information used to support performance-based funding decisions, (2) progress in implementing a risk assessment model and early warning system, and (3) oversight of the performance of "local fund agents" (LFAs), which monitor grant progress in recipient countries. GAO reviewed the documentation for funding decisions and interviewed key officials.
section_title: What GAO Found
paragraphs: Since our 2005 review, the Global Fund has improved its documentation for decisions to disburse funds and renew grants. The Global Fund now requires that fund portfolio managers more consistently document factors, such as grant ratings and contextual information that support disbursement and grant renewal decisions. Our current review of 80 grant disbursements and 45 grant renewal decisions confirmed that Global Fund grant files consistently contained explanations of the information used in its funding decisions. For example, all grant disbursement files in our sample contained a written narrative explaining the ratings that portfolio managers gave the grants, based in part on reports completed by grant recipients. Although we noted that many of the grant recipients' reports lacked some information needed for disbursement and renewal decisions, Global Fund officials said that portfolio managers obtain this information informally from the recipients or other stakeholders. The Global Fund did not implement the risk assessment model that it was developing at the time of our 2005 report, because it determined that the model did not accurately identify grant risk. To identify risks that may affect grant implementation, the organization currently relies on elements of its structures and processes, including its initial technical review, disbursement decision-making form, periodic grant ratings, oversight by country stakeholders, information from technical partners, and LFA oversight. Recognizing the need for a more comprehensive approach to risk management, the Global Fund has begun developing a risk assessment framework for the organization that includes an early alert and response system to address poorly performing grants. The Global Fund has limited access to the information it needs to manage and oversee LFAs because it does not require systematic assessments of LFAs' performance. As a result, the Global Fund has limited ability to determine the quality of LFAs' monitoring and reporting and to identify situations in which more oversight of LFAs' performance may be required. Previously, the Global Fund introduced a tool to assess LFA performance more systematically; however, this effort was unsuccessful, because use of the tool was not mandatory. Numerous sources raise concerns about the quality of grant monitoring and reporting provided by LFAs, particularly their ability to assess and verify recipients' procurement capacity and program implementation.

section_title: Letter
paragraphs: The Global Fund to Fight AIDS, Tuberculosis and Malaria (the Global  Fund) was created to help finance the fight against three infectious  diseases that kill more than 6 million people each year. Established in  2002, the Global Fund is a private foundation intended as a partnership  among governments, civil society, the private sector, and affected  communities. From 2002 through February 2007, the Global Fund  approved grant proposals totaling about $7 billion to governmental and  nongovernmental entities to carry out disease intervention activities in  more than 100 countries. During this period, the U.S. government  contributed $1.9 billion—the largest amount provided to the Global Fund  by a single donor. The State Department’s Office of the Global AIDS  Coordinator (OGAC) coordinates the U.S. government’s overseas AIDS  programs, with participation from the Department of Health and Human  Services (HHS) and the U.S. Agency for International Development  (USAID).
The Global Fund uses a performance-based funding system designed to  encourage financial accountability and program progress and to link its  continued grant disbursements to achievement of these results. The Global  Fund contracts with “local fund agents” (LFA) to assess recipients’  capacity for grant implementation and verify recipients’ financial and  program performance data. In June 2007, it plans to issue applications for  new contracts for LFA oversight services.
In 2003, Congress directed the Comptroller General to monitor and  evaluate projects supported by the Global Fund and to submit a report  every 2 years. In June 2005, we reported that the Global Fund’s grant  documentation did not always show clear grounds for performance-based  funding decisions.  These decisions can include periodic disbursements as  well as decisions to renew, suspend, or terminate grants. We further noted  that some of the grant recipients’ self-reported performance data were of  poor quality or incomplete. We also reported that the Global Fund was  developing a risk assessment model and early warning system to alert its  fund portfolio managers to the need to respond to poorly performing  grants. In addition, we noted that LFAs’ assessments of recipient  performance contained inconsistencies and that LFAs often lacked the  knowledge and experience needed for these assessments. In its response  to our 2005 report, the Global Fund generally concurred with our findings  and indicated that it was taking steps to improve its structures and  systems.
Following up on our 2005 review, this report assesses the Global Fund’s  (1) documentation of the information used to support performance-based  funding decisions, (2) progress in implementing a risk assessment model  and early warning system, and (3) oversight of local fund agents’  performance.
To address these objectives, we met with key officials from the Global  Fund and reviewed Global Fund documents, including key source  documents from grant files. To assess the documentation of the  information used to support performance-based funding decisions, we  analyzed two separate random samples of grant decisions from the Global  Fund. The first sample consisted of 80 grants that were assessed for  performance-based funding disbursements between January 1 and  October 1, 2006. The second sample consisted of 45 grants that were  assessed for phase 2 grant renewal. We projected our findings to the files  of all grants that received a disbursement or were assessed for grant  renewal during that period. To assess Global Fund progress in  implementing a risk assessment model and early warning system, we  reviewed international standards on risk management, met with Global  Fund secretariat officials, and reviewed Global Fund documents. We also  held discussions with the Global Fund’s Inspector General and three of its  key technical partners. To assess the Global Fund’s oversight and  measurement of LFAs’ capacity and performance, we met with each of the  six LFA headquarters offices and relevant secretariat personnel. In  addition, we held structured interviews with LFA teams in 12 countries via  telephone. We also obtained perspectives on the progress of the Global  Fund from its partners and outside experts. We conducted our work  between May 2006 and March 2007 in accordance with generally accepted  government auditing standards. (See app. I for a detailed description of  our scope and methodology.)
subsections: section_title: Background
paragraphs: The Global Fund, a private foundation in Switzerland, was established in  2002 as a funding mechanism to collect public and private contributions to  finance grants for the prevention and treatment of HIV/AIDS, tuberculosis  (TB), and malaria. About 96 percent of the contributions to the Global  Fund since its inception have come from governments. Private sector  support increased in 2006, notably owing to a pledge of $500 million by the  Bill and Melinda Gates Foundation and receipt of almost $19 million  through the (RED)™ initiative.
By December 2006, the Global Fund had completed six proposal rounds,  approved about $7 billion in grants, signed grant agreements worth over $5  billion and disbursed more than $3 billion to grant recipients in 132  countries. The Global Fund also reports that by December 2006, its grants  enabled recipients to supply antiretroviral drugs to 770,000 people infected  with HIV/AIDS, treat 2 million people infected with tuberculosis, and  distribute 18 million insecticide-treated nets to combat malaria.
subsections: section_title: Global Fund Portfolio Distribution
paragraphs: The Global Fund’s underlying principles require, among other things, that  it maintain a balanced grant portfolio in terms of region, disease, and type  of intervention; operate transparently and accountably; and use a simple,  rapid, and innovative grant-making process. As figure 1 shows, between  2002 and 2006, the Global Fund distributed 55 percent of grant funds to  sub-Saharan Africa, with the remainder of the funding divided among  other regions of the world. Also during this time period, the Global Fund  distributed 57 percent, 15 percent, and 27 percent of grant funds,  respectively, for HIV/AIDS, tuberculosis, and malaria programs. The  remaining 1 percent funded projects to strengthen health systems.
In 2005, the Global Fund began tracking the allocation of grant funds in  grant proposals according to the disease intervention categories of  prevention, care, and treatment. Figure 2 shows the Global Fund’s  estimated distribution by disease and intervention category.
section_title: Key Elements of the Global Fund’s Structure
paragraphs: The Global Fund’s principles also require it to focus on performance by  linking its resources to the achievement of clear, measurable, and  sustainable results. To achieve this focus on performance-based funding,  the Global Fund relies primarily on oversight by, and interaction among,  Global Fund secretariat personnel, grant recipients, and LFAs.   (See fig. 3.)
Global Fund secretariat personnel. Within the secretariat, the fund  portfolio managers are primarily responsible for reviewing grant progress  and making performance-based funding recommendations. Other  secretariat personnel are consulted during key stages in the grant life  cycle. Fund portfolio managers interact with the LFA country teams  providing oversight services for their grant, as well as with principal  recipients, subrecipients, technical partners, and other stakeholders in the  recipient countries. In addition to the country teams, staff from the  contracts department and the Global Fund’s LFA manager interact with  LFA headquarters teams, known as the central teams, regarding contracting and administrative matters and any problems related to LFA  country teams.
Grant recipients. At the country level, the principal grant recipient is  responsible for receiving and implementing the grant. During  implementation, the principal recipient is responsible for monitoring and  evaluating the grant’s effectiveness in accordance with grant objectives  and making sure that funds are properly accounted for. The principal  recipient is also responsible for overseeing the activities of any  subrecipients implementing grant activities.
Local fund agents. The Global Fund currently contracts with LFA  headquarters central teams at six organizations to oversee grants in 120  countries and multicountry programs. In the majority of cases, the central  team establishes interfirm agreements with its affiliated offices within  each country to perform LFA services. The LFA assesses recipient  capacity prior to grant signing and performs verification and oversight  duties throughout the life of the grant. The LFA central teams typically  contract with external specialists for work outside the expertise of their  organization, especially during the initial assessments of recipient  capacity. For example, during the initial assessments, some organizations  contract with a procurement expert to review recipients’ capacity for  procurement and supply chain management. The Global Fund budgeted  approximately $22 million—26 percent of the secretariat’s operating  budget—to support the LFA structure for 2006 and spent about an  additional $320,000 on LFA-related functions, including personnel costs  related to LFA management, studies of the LFA model, and travel to meet  with LFAs.
section_title: Performance-Based Funding Process
paragraphs: The Global Fund’s performance-based funding process involves an initial  assessment of recipients’ capacity to manage and implement grants,  periodic disbursement decisions based largely on reports by the grant  recipient and the LFA, and a phase 2 grant renewal review after about 2  years.
Assessment of grant recipients’ capacity. Prior to funding a grant, the  Global Fund considers, among other things, a grant recipient’s ability to  (1) manage, evaluate, and report on grant-funded activities; (2) manage  and account for funds; and (3) procure goods and services. LFAs assess  prospective recipients’ capacity to perform these activities. Funding is  conditioned on recipients’ meeting performance targets mutually agreed  with Global Fund technical experts.
Periodic disbursements. After the initial screening and approval process,  the grant recipient must produce quarterly or semi-annual reports  describing financial and program progress as well as a funding request for  the next disbursement period. The grant recipient sends these reports to  the LFA, which prepares a report verifying the accuracy of the  information, identifying potential grant problems, rating grant  performance, and recommending a disbursement amount. The LFA then  forwards the grant recipient’s report and its own report to the designated  Global Fund portfolio manager. The fund portfolio manager compares the  financial information and data on program progress with the grant’s  performance targets, taking into account information such as the status of  procurements and political or economic constraints. The fund portfolio  manager’s team leader makes a funding decision if the disbursement is  less than $2 million. Figure 4 illustrates the Global Fund’s periodic  disbursement decision process.
Grant renewal process. A key milestone in the performance-based funding  approach is the phase 2 grant renewal review—a detailed analysis of grant  progress at about 2 years, required to continue funding for up to the next 3  years. Similar to its process for disbursement decisions, the Global Fund’s  grant renewal process requires grant recipients to send documentation of  financial and program progress to the LFA, which verifies the information  and analyzes the phase 2 proposal. In a report to the Global Fund, the LFA  identifies potential grant problems, rates the performance of the grant, and  recommends the amount needed to fund up to the next 3 years of the  grant. Because phase 2 is the decision point for continuing the grant to its  5-year completion, the process also includes an extensive and detailed  review of cumulative information by fund portfolio managers, their  supervisors, and evaluation and finance staff. The phase 2 renewal process  culminates in a panel discussion and final recommendation to the board.  Figure 5 illustrates key elements of the Global Fund’s phase 2 grant  renewal process.
Table 1 outlines the key source documents related to disbursement  decisions and phase 2 grant renewals from electronic “grant files” that we  reviewed in conducting this study.
In preparation for its retender of LFA contracts worldwide in mid to late  2007, the secretariat will continue discussions on the key principles that  will potentially shape the design of the next LFA model. According to a  Global Fund official, they will take into consideration findings from this  review and two internal Global Fund studies. They expect to issue the  request for applications in late June.
section_title: Global Fund Has Improved Documentation of Its Basis for Funding Decisions
paragraphs: The Global Fund has improved its documentation of information  supporting its performance-based funding decisions. For example, the  Global Fund’s system now requires that portfolio managers consistently  document factors that may affect the decision to continue funding a grant,  especially when the grant falls short of its performance targets. In  addition, the Global Fund grant files that we reviewed consistently  contained explanations of funding decisions and contextual data  informing the decisions. Although we found many reports submitted by  grant recipients to the Global Fund do not provide explanations for all  unmet grant performance targets, according to Global Fund officials, fund  portfolio managers are able to obtain the needed information directly from  the grant recipients or other stakeholders.
subsections: section_title: Global Fund Has Improved Its Processes for Documenting the Basis for Funding Decisions
paragraphs: The Global Fund has developed a more systematic process for  documenting the basis for its funding decisions. In June 2005, we noted  that the Global Fund did not consistently document the reasons for its  determinations that recipients’ performance warranted additional funding  for periodic disbursements or its recommendations regarding grant  renewals. We recommended that the Global Fund continue efforts to  clearly document the Global Fund’s reasons for periodically disbursing  funds and renewing grant agreements. Subsequently, to increase the  transparency of its performance-based funding system, the organization  made several changes to its documentation process.
In late 2005, the Global Fund developed a disbursement decision-making  form to ensure that fund portfolio managers consistently show the basis  for each disbursement decision. The form requires, among other things,    an explanation of the performance rating assigned to the grant, addressing  any discrepancy between ratings by the fund portfolio manager and the  LFA and verifying that the manager has considered any problems raised by  the LFA regarding data quality and reporting, program progress, and  expenditures; and   contextual information critical to the disbursement decision, such as  information demonstrating governance problems, grant management  weaknesses, currency fluctuations, or natural disasters in the country or  region where a funded project is being implemented.
The Global Fund also revised other forms to provide more systematic  reporting of grant program progress. These revisions included  improvements to the grant progress update and disbursement request, the  LFA progress review and recommendation report, and the phase 2 grant  scorecard. For example, the updated progress update form contains a  new requirement that LFAs comment on the reliability of grant recipient  data on target achievement. The updated grant scorecard includes  requests for data showing the timeliness and effectiveness of grant  recipients’ use of the funds during phase 1 of grant implementation. It also  includes a new section where the secretariat can record plans to  materially change the grant in phase 2—for example, adding or dropping  goals or objectives because of lessons learned during phase 1.
According to Global Fund officials, the changes to their system of  recording information about grant performance and problems have made  the decision process more systematic and transparent. For example,  Global Fund officials, including some fund portfolio managers, said the  new disbursement decision-making form provides managers a place to  systematically explain their ratings. Officials from OGAC and the World  Bank with whom we met said that the Global Fund’s more consistent  documentation of the basis of its decisions has made the decisions easier  to understand.
section_title: Grant Files Demonstrate Improved Documentation Supporting Funding Decisions
paragraphs: Our review of random samples of disbursement and phase 2 grant files  indicated that most files systematically explain a number of key factors  used to support disbursement and grant renewal decisions. In our 2005  report, we observed that the Global Fund’s grant files provided little  explanation of disbursement decisions, including decisions to disburse  funds to recipients who reported not meeting their targets. In addition, our  2005 report noted that, according to Global Fund officials, many grant  renewal decisions were not supported by the information needed to assess  whether funding decisions were consistently based on grant performance.
In our current review, we found that Global Fund grant files consistently  contained explanations of ratings of grant performance that were  important to funding decisions, even when key stakeholders’ ratings and  recommendations did not concur. Based on our samples, 100 percent of  both disbursement and phase 2 files contained narratives explaining the  manager’s rating of the grant’s performance. Most grant files also  contained explanations for lack of concurrence in ratings or  recommendations by the various officials charged with assessing grant  performance. For example:  In all 8 cases in which fund portfolio managers’ phase 2 grant ratings  differed from those of the evaluation staff, the scorecards in the grant files  contained an explanation for the lack of concurrence.
In all 5 cases in which fund portfolio managers’ phase 2 grant renewal  recommendations differed from those of the LFAs’, the scorecards in the  grant files contained an explanation for the lack of concurrence.
In cases in which fund portfolio managers’ assessments of grant  performance led them to disagree with the disbursement amount  recommended by the LFA or the amount requested by the grant recipient,  31 of 32 of those grant files included a narrative explaining the  discrepancy.
According to Global Fund officials, such explanations provide valuable  information that helps decision-makers better understand and consider  grant performance.
The Global Fund files that we reviewed also consistently explained  contextual information and other issues that may affect the grant  recipient’s ability to meet agreed-upon targets. We found that an estimated  96 percent of disbursement files in which portfolio managers had specified  contextual issues that negatively affected grant performance included  narratives explaining each issue. According to Global Fund officials, such  information is crucial to grant disbursement and renewal decisions,  especially if recipients have not met many of the targets set at the grant’s  initiation. In addition, in reviewing 43 disbursement files for grants whose  recipients had not yet met remedial requirements imposed by the Global  Fund as conditions for grant renewal, we found that 42 (98 percent)  contained an explanation of recipients’ progress toward meeting the  requirements.
section_title: Global Fund Informally Gathers Missing Data on Grant Recipients’ Performance
paragraphs: Although recipients’ formal documentation of grant performance remains  incomplete, Global Fund officials informally gather the missing  information that they consider necessary for making and documenting the  basis for funding decisions. In June 2005, we reported that recipient  progress reports varied in quality and that the limited monitoring and  evaluation capabilities of many recipients raised questions about the  accuracy of their reporting. Our current analysis found that an estimated  39 percent of phase 2 grant files contained evidence, identified by fund  portfolio managers, of weaknesses in the reliability of the grant recipient’s  data. In contrast to the consistent documentation we found on forms  completed by Global Fund staff, many periodic performance reports  completed by grant recipients lacked explanations either for not meeting  or for exceeding the grant’s targets. In those cases, according to LFAs,  fund portfolio managers, and other Global Fund officials, fund portfolio  managers relied on regular phone and e-mail contact with grant recipients  or other stakeholders to obtain the missing information needed for  performance-based funding decisions. Global Fund officials said that  information crucial to funding decisions is documented on the  disbursement decision-making form or phase 2 grant scorecard, even if it  was not recorded by grant recipients or LFAs in their formally required  reports.
Global Fund portfolio managers said that recipients’ explanations for not  meeting targets may influence the portfolio manager’s recommendation to  withhold funds or, alternately, to continue to fund the grant despite  performance problems. According to Global Fund officials and public  health experts, improving grant recipients’ monitoring and evaluation  capacity, as well as their ability to report accurately and completely, will  require long-term investment on the part of the Global Fund and its  partners.
section_title: Global Fund Did Not Implement Earlier Risk Model but Is Developing Comprehensive Risk Framework That Includes Early Alert and Response System
paragraphs: The Global Fund did not implement the risk assessment model that it was  developing at the time of our 2005 report, because it determined that the  model did not accurately identify grant risk. The organization relies on  elements of its structures and processes, including its initial technical  review, disbursement decision-making form, periodic grant ratings, CCM  oversight, information from technical partners, and LFA oversight, to  identify potential risks. Recognizing the need to establish a more  comprehensive approach to risk management, the Global Fund has begun  developing a risk assessment framework for the organization that includes  an early warning and response system to address poorly performing  grants.
subsections: section_title: Global Fund Did Not Implement the Grant Risk Assessment Model Described in the 2005 GAO Report
paragraphs: The Global Fund did not implement the risk-assessment framework to  identify poorly performing grants that it was developing at the time of our  2005 report. We reported that the organization had devised a risk  assessment model and an early warning system to identify poorly  performing grants and more systematically alert portfolio managers when  they needed to intervene. According to the Global Fund, because it  disburses grants in countries with varying levels of economic development  and health systems capacity, the risk model was to incorporate  quantitative data, such as grant size and performance as well as country  development and corruption data. Using these quantitative indicators, the  model would generate reports that indicated potential problems, and  portfolio managers would evaluate the reports in light of contextual  information. According to an official who participated in developing the  framework, it was not implemented because it relied too heavily on  quantitative indicators and did not adequately account for contextual  factors, leading to risk ratings that were too inaccurate to be useful. For  example, according to Global Fund officials, in a test of the model, a  particular grant was rated as low risk based on the host country’s  development and corruption indicators. However, the grant ultimately  failed and was not renewed at the phase 2 grant renewal review.
section_title: Global Fund Applies Elements of Risk Management but Lack of Risk Assessment Model Limits its Ability to Respond to Poorly Performing Grants
paragraphs: Although the Global Fund’s grant oversight incorporates several elements  of risk management, the absence of a risk assessment model limits its  ability to respond methodically to poorly performing grants and use  available resources to increase oversight as needed. According to Global  Fund officials, the organization currently relies on existing oversight  mechanisms such as its initial technical review of grant proposals,  standardized decision forms, periodic grant ratings, CCM oversight,  technical partners’ assistance, and LFAs’ reviews to identify and assess  potential mismanagement, irregularities, or other risks to successful grant  implementation.
Technical review. The technical review panel raises important technical  issues that grantees must address when presenting their proposals to the  panel. Proposals with a very high technical risk profile generally are not  recommended for funding.
Disbursement decision-making form. Grant risk is mitigated by the fund  portfolio manager’s use of the standard disbursement decision-making  form, which includes financial and program data from the recipient that  has been verified by the LFA.
Rating grant progress for each disbursement request. Since 2006,  according to Global Fund officials, every progress and disbursement  request, after the first disbursement, has been graded on a scale of A, B1,  B2, or C. This information is posted on the Global Fund Website’s financial  data base and included in grant performance reports that are updated on  the Global Fund website at each disbursement. This rating system is a key  tool for identifying grants at risk and for sharing information with  technical partners.
Country coordinating mechanism (CCM). A key responsibility of the  CCM is to continue to monitor grant progress during implementation and  identify problems that may affect progress. In many countries, Global  Fund technical partners such as UNAIDS, WHO, and USAID are members  of CCMs.
Partner assistance. Global Fund officials rely on technical partners with  staff based in recipient countries to provide contextual information on  grant performance, identify risks, and help respond to problems that affect  grant implementation.
LFA reviews. Global Fund officials emphasized that LFAs identify  potential grant risks in their periodic verification of grant progress and  during the cumulative review and performance assessment for the phase 2  renewal process.
The absence of a model to assess grant risk has limited the Global Fund’s  ability to methodically identify poorly performing grants and use available  resources to increase oversight as needed. According to a senior manager  from an LFA with one of the largest grant portfolios, because the Global  Fund has not assessed grants as high, medium, or low risk, LFA oversight  has not differed substantially by grant risk level and LFA oversight  resources have not been used to optimal effect. For example, although  some grants may require more oversight because of potential misuse of  grant money or limited recipient capacity, LFA task orders and fees are  generally no different for potentially problematic grants than for grants  with few expected problems. Similarly, the frequency of LFA visits to  project sites for monitoring and data verification has not been based on a  systematic estimate of risk.
section_title: Global Fund Is Developing a Comprehensive Framework for Risk Management and Has Established First Components
paragraphs: Global Fund officials informed us in April 2007 that they are developing a  top-level framework for risk management that will cover all aspects of the  organization’s strategic direction, including addressing resources and  skills at the secretariat level, developing a model for systematically  assessing grant risk, and fully implementing an early alert and response  system. Global Fund officials have acknowledged the need for a  quantitative model for ranking grants by level of risk, supplemented by  additional qualitative factors that may change a grant’s rating or ranking,  to identify and mitigate grant risk in a timely manner. The Global Fund  recently organized a risk seminar for directors and senior managers and  engaged an international firm to help them achieve consensus on a shared  approach to comprehensive risk management. We reviewed a draft outline  of the guiding principles for developing a risk management framework  that the Global Fund developed in February 2007. As of April 2007, the  Global Fund is in varying stages of progress in developing and  implementing detailed risk strategies.
According to Global Fund officials, the secretariat has put in place the first  components of its early alert and response system intended to identify as  early as possible grants that are performing poorly and facing  implementation difficulties. These first components include posting the  grant disbursement and renewal ratings on the Global Fund Website and  efforts to engage more technical assistance from partners for poorly  performing grants. Global Fund officials recognize that for the system to  work effectively principal recipients, CCMs, stakeholders, and partners  must share responsibility for tracking the progress of grant  implementation and take appropriate action when progress is impeded.  Global Fund officials stated that the early alert and response system will  eventually include documenting and disseminating best practices,  formalizing the system within the secretariat and among country-level  stakeholders and partners, and increasing the engagement of partners in  grant oversight.
section_title: Global Fund Has Limited Ability to Manage and Oversee Local Fund Agents’ Performance
paragraphs: The Global Fund lacks the information it needs to manage and oversee the  LFAs because of a lack of systematic assessments of LFAs’ performance.  Good practice suggests that organizations monitor their contractors’  performance during the course of interaction and periodically evaluate  performance based on clear expectations. Assessment results should  include consistent patterns of rewards, penalties, or both. (See app. II for  sources of these principles.) Global Fund portfolio managers set  expectations for LFA performance informally and individually, frequently  leading to inconsistent expectations and resulting in challenges for both  the LFAs and their managers. Moreover, the Global Fund does not require  formal assessments or reports of LFA performance; instead, LFAs are  assessed informally and irregularly, and assessment results are not  systematically documented. As a result, the Global Fund has limited  records of, and data on, LFA performance. Although the Global Fund  made a previous attempt to assess LFAs more systematically, this effort  was unsuccessful. According to secretariat staff, key partners, and outside  experts, LFAs’ performance and capacity vary.
subsections: section_title: Global Fund Has Not Standardized Its Expectations for Local Fund Agent Performance
paragraphs: Rather than establish standardized performance expectations for all LFAs,  the Global Fund portfolio managers establish expectations informally for  the quality of each LFA’s performance. The Global Fund’s contracts with  LFAs, as well as the guidelines that it provides, outline the tasks that all  LFAs must complete, such as considering a principal recipient’s  programmatic and financial implementation progress in relation to its  plans. The contracts also contain general statements about LFA  performance; for example, a sample LFA contract specifies that all tasks  should be carried out to the satisfaction of the Global Fund. However,  Global Fund policy states—and secretariat staff and 11 of the 12 LFA  country teams that participated in our structured interview confirmed— that the fund portfolio manager is primarily responsible for defining  satisfactory performance for his or her designated LFAs. As of March 2007,  the Global Fund had not provided fund portfolio managers any specific  training or written guidance defining acceptable or good LFA  performance, including the quality of their reporting.
The lack of standardized performance expectations results in challenges  both for the LFAs and for Global Fund management.
LFA challenges. Because fund portfolio managers set expectations  informally and individually, LFA country teams frequently encounter  different expectations from successive portfolio managers. For example,  among the LFA country teams who participated in our structured  interviews, 6 of 10 teams who had interacted with more than one fund  portfolio manager said that the managers’ criteria for a high-quality report  had varied. According to the respondents, variations in fund portfolio  managers’ expectations often involved differences in the scope of work  that they expected LFAs to complete. For instance, members of one  country team said that their previous fund portfolio manager had wanted  the team to focus primarily on financial data in its reports, while the  current manager wanted a focus on programmatic issues. In another  country, one fund portfolio manager wanted the team to conduct on-site  verification of grant subrecipients, while a subsequent manager did not  consider site visits a priority. In the remaining four cases, country teams  did not report substantial variation between different fund portfolio  managers. In one of these cases, the team said that a previous manager  established expectations that subsequent managers used in turn, therefore  providing continuity.
Management challenges. The lack of standardized LFA performance  expectations also presents challenges among those responsible for  managing the LFAs. According to the manager of one LFA central team  with a large grant portfolio, variations in Global Fund expectations made it  difficult to establish best practices for its LFA country teams. Likewise,  several Global Fund officials noted that secretariat staff have conflicting  ideas about LFA performance criteria. In one instance, according to Global  Fund officials, a group of fund portfolio managers were unable to agree on  what constituted a high-quality report. Fund portfolio managers and other  key officials observed that more standardization of expectations would be  beneficial, although some noted that this standardization should be  balanced to allow them to maintain flexibility to account for country  context.
section_title: Global Fund Does Not Assess LFAs Systematically and Lacks Formal Records of LFA Performance
paragraphs: The Global Fund does not require formal, regular assessments of LFA  performance. As a result, it lacks LFA performance records and cannot  systematically track performance.
subsections: section_title: LFA Performance Assessment and Reporting Are Informal and Irregular
paragraphs: Although Global Fund officials regularly review the reports they receive  from the LFAs and provide informal feedback to country teams, they  assess LFA performance informally and at irregular intervals. All fund  portfolio managers that we interviewed, as well as 11 of the 12 LFA  country teams that participated in our structured interviews, said that fund  portfolio managers interact informally and frequently with country team  members, often orally or by e-mail. In addition, all but one country team  said that fund portfolio managers provide informal feedback on their  performance. For example, members of one team noted that they received  feedback from their fund portfolio manager for each report and requested  more information as necessary. In another case, the manager provided  feedback to the LFA central team, which passed the feedback to the  country team. A majority of the country teams reported receiving feedback  from their fund portfolio manager when he or she traveled to the recipient  country, at least once a year. Fund portfolio managers told us that if they  continually encounter problems with a country team, they inform the  Global Fund’s LFA manager, who discusses the issue with the central  team; if this does not resolve the problem, the contract may be terminated.  A key Global Fund official stated that two to three LFA country team  subcontracts are terminated each year based on information from fund  portfolio managers on their poor performance.
Although LFA central teams provide some performance feedback to their  country teams, the scope and frequency of that feedback varies depending  on the LFA’s structure. Ten of the LFA country teams that participated in  our structured interviews told us that LFA central teams provide informal  feedback about their performance. The central team at two of the six LFA  organizations reviews all products before they reach the secretariat. At  two other LFA organizations, reports are reviewed only if it is deemed  necessary, and in those cases, regional coordinators or mentors may  review the reports. At the remaining two LFA organizations, the  distinctions between the country and central teams are less clear, because  the reports are at least partially written by central team members.
section_title: Lack of Assessment Records Limits Available Information on LFA Performance
paragraphs: The lack of formal assessments limits the Global Fund’s access to the  information needed to manage LFAs’ performance. The fund portfolio  managers we interviewed said that their frequent interaction with country  teams generally allows them to know which teams are performing well  and which teams need improvement. However, Global Fund officials with  LFA management responsibility stated that formalizing the process would  allow the agency to establish LFA performance records and identify  situations in which more oversight of LFAs by the Global Fund may be  required. A working group of Global Fund experts echoed these concerns,  reporting that despite extensive debate about the LFA model, the Global  Fund had conducted little systematic assessment of the performance of  individual LFA country teams.
section_title: Global Fund Previously Attempted to Implement Systematic LFA Assessments
paragraphs: A prior attempt to introduce a tool to make LFA assessments more  systematic was not successful, because use of the tool was not mandatory  and fund portfolio managers did not see it as a priority. According to the  Global Fund, the LFA manager conducted a pilot of an evaluation  checklist as a tool for fund portfolio managers to systematically assess  LFA performance. Secretariat staff stated that, because Global Fund  management did not require fund portfolio managers to complete the  checklist, very few portfolio managers used it. As of early 2007, the LFA  manager was developing a revised tool to routinely collect staff input on  LFAs. However, it is too early to determine whether this tool, when  finalized and made available to staff, will capture the information needed  to assess LFA performance. According to the LFA manager, integrating  this tool into existing tasks and making it an explicit component of fund  portfolio managers’ job descriptions would increase its use.
section_title: LFA Performance and Capacity Reportedly Vary
paragraphs: Although Global Fund officials cited instances in which LFA country  teams performed adequately, secretariat staff, key partners, and outside  experts expressed concerns about some LFAs’ capacity to oversee grant  recipients’ programmatic and procurement activities. To illustrate  adequate LFA capacity and performance, secretariat staff identified cases  where LFA country teams have quickly identified problems with grants.  For example, the majority of fund portfolio managers we interviewed  noted that most LFA country teams performed financial oversight  competently. Similarly, a procurement officer described a country team  that focused on supply management issues throughout the life of the grant  and proved instrumental in resolving a large corruption problem. One fund  portfolio manager described a country team that had enough knowledge of  monitoring, evaluation, and procurement to understand an LFA  subcontractor’s initial assessment of recipient capacity and, therefore,  could use this information in subsequent reporting about grantee progress.
Conversely, secretariat staff also cited multiple instances in which LFAs  demonstrated lack of capacity that affected the conduct of their oversight  duties. In a recent Global Fund survey, secretariat employees who  identified themselves as having sufficient exposure to LFAs to form an  opinion reported mixed experiences regarding LFAs’ technical program  and monitoring expertise. About half of the respondents disagreed with  the statement that LFAs with whom they had worked provided “adequate”  expertise on health and program issues. The remainder was about equally  divided between those who agreed that health and program expertise was  adequate and those who neither agreed nor disagreed. Global Fund  employees’ assessments of the adequacy of LFAs’ expertise for monitoring  and evaluation were similarly mixed. The survey also provided evidence  that the service quality varies greatly among countries: about three- quarters of the staff agreed with the statement that “within LFA firms there  are substantial differences in service quality across countries.”
In addition, Global Fund partners and outside experts have raised  concerns about LFAs’ oversight capacity. UNAIDS officials said that they  have serious doubts about the capacity of LFAs to monitor program  progress, specifically noting that LFAs have little understanding of how to  weigh and balance various performance targets that are not of equal  importance. For example, according to these officials, an LFA that lacks  strong technical program capacity might not recognize that a recipient  exceeding a simple performance target, such as training staff, may be  trying to compensate for poor performance in more difficult target areas,  such as administering drugs to treat AIDS, TB, or malaria. Several other  sources—a report commissioned by the Global Fund, our 2005 report, and  several reports by an acknowledged expert on the Global Fund—also note  problems with LFA capacity. (See app. II for a list of sources.)
section_title: Conclusions
paragraphs: Since our report in June 2005, the Global Fund has taken steps to improve  operations at its secretariat in Geneva while it continues to confront  complex issues facing grant implementation at the country level. The  Global Fund will continue to face important challenges, including  evaluating the effectiveness of its investment of the billions of dollars it  receives annually from the United States and other donors and the impact  of its programs over time. The Global Fund has improved the transparency  of its periodic disbursement and grant renewal decisions by providing  better documentation in its grant files. Delays in developing a systematic  approach to assessing grant risk have left the Global Fund vulnerable to  some of the risks associated with implementing grants in developing  countries. However, the Global Fund board recognizes the importance of  developing a more comprehensive approach to risk management, and the  secretariat has recently drafted principles to guide its development of a  risk framework for the organization. In addition, the Global Fund’s lack of  a formal process for setting expectations and assessing and analyzing LFA  performance constrains its ability to determine the quality of the LFAs’  services, track LFA performance over time, and address ongoing capacity  issues. The lack of a formal assessment process also limits the information  available to the Global Fund during its upcoming global retender of LFA  contracts.
section_title: Recommendations for Executive Action
paragraphs: To enhance Global Fund operations, we recommend that the Secretaries  of State and HHS work with the Global Fund’s Board Chair and Executive  Director to take the following two actions:    establish standardized expectations for LFA performance, and  require systematic assessments of LFA performance and the collection  and analysis of LFA performance data to improve the management and  oversight of LFAs.
section_title: Agency Comments
paragraphs: Written comments on a draft of this report from the Departments of State  and Health and Human Services and the U.S. Agency for International  Development are represented in the unified response received from the  Office of the U.S. Global AIDS Coordinator. The Global Fund also  provided written comments on the draft. Both entities agreed with the  report’s findings, conclusions, and recommendations. The Office of the  U.S. Global AIDS Coordinator stated that the report accurately reflected  areas in which the Global Fund has made progress as well as areas in  which more work remains to be done. The Office’s response noted that the  United States will urge the Board to adopt concrete recommendations  from several studies to improve the accountability and transparency of  Global Fund activities and will work with the Global Fund to improve its  oversight and risk assessment mechanisms. The Global Fund stated that  the report reinforced the importance of measures it has begun taking to  better measure local fund agent performance. Regarding risk assessment,  the Global Fund emphasized the role of its country-based and global  partners in helping to track grant implementation and provide technical  support as needed. (See apps. III and IV for their comments.)
We are sending copies of this report to interested congressional  committees and will make copies available to others on request. In  addition, this report is available on GAO’s Web site at http://www.gao.gov.
If you or your staff have any questions, please contact me at (202) 512-3149  or gootnickd@gao.gov. Contact points for Offices of Congressional  Relations and Public Affairs may be found on the last page of this report.  Key contributors to this report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: To assess the Global Fund’s efforts to improve documentation of  performance-based-funding decisions, we analyzed the forms used to  collect data on grant performance and discussed with Global Fund  officials how the forms have changed since GAO’s 2005 report. We met  with officials from all of the entities that have current contracts to serve as  local fund agents and with officials from the Office of the Global AIDS  Coordinator, UNAIDS, and the World Bank—-the Global Fund’s key  technical partners—to discuss how documentation of the Global Fund’s  decisions has changed and whether it has improved.
To review information in the grant files used to support performance- based funding decisions, we analyzed two separate simple random  probability samples of grant funding decisions using a separate data  collection instrument for each sample. The first sample consisted of 80 of  the 348 grants that the Global Fund assessed for performance-based  funding disbursements between January 1 and October 1, 2006. The  second sample consisted of 45 of the 65 grants that the Global Fund  assessed for phase 2 grant renewal during the same time period. We chose  January 2006 as the starting point for our sample because Global Fund  officials told us that their new disbursement decision worksheet—created  to address GAO’s recommendation that they better document their  decisions—was fully implemented by the end of 2005. With our two  probability samples, each member of the study population had a nonzero  probability of being included, and that probability could be computed for  any member. Each sample case file was subsequently weighted in the  analysis to account statistically for all the case files in the population,  including the files that were not selected.
Because we followed a probability procedure based on random selections,  our samples are only one of a large number of samples that we might have  drawn (for both the grant disbursement and phase 2 file reviews). Since  each sample could have provided different estimates, we express our  confidence in the precision of our particular sample’s results as 95 percent  confidence intervals. These are intervals that would contain the actual  population values for 95 percent of the samples we could have drawn. All  percentage estimates provided in this report based on our samples have 95  percent confidence intervals of within plus or minus 13 percentage points.
For each grant in our disbursement sample, we requested the grant  recipient’s Progress Update and Disbursement Request report, the Local  Fund Agent Recommendation report, and the Global Fund’s Disbursement  Decision-Making Worksheet. For the phase 2 decision sample, we  requested the following documents for each grant:    Phase 2 Grant Scorecard (the in-house version that goes to the panel)
Phase 2 Grant Scorecard (the panel’s version that goes to the board)
Documentation of the board’s decision with any explanation or  justification of the decision (whether this is in the form of a scorecard or  some other format)
Annex A Attachments 2 and 3    Local fund agent Phase 2 Assessment Report  We gathered the information from the documents for our data collection  instruments and verified the data that we used to support our findings.
Our file review consisted of a number of checks on the presence or  absence of an array of qualitative and contextual information that the  Global Fund forms specify as important for grant decision-making and that  therefore should be provided by those responsible for filling them out. We  did not evaluate the appropriateness or accuracy of the judgments made  on the basis of the narrative or contextual information provided in the  documents; we focused on the issue of whether the contextual or  narrative information provided was relevant to particular judgments or  decisions.
In our analysis of the disbursement files, while we examined information  contained in other documents, we focused mostly on the Disbursement  Decision-Making Form, due to the fact that this was the key form that the  Global Fund instituted for disbursement-related decision-making since our  last report. In this form, we checked for the presence or absence of the  following pieces of information:    Fund portfolio manager/cluster explanation for differences in fund  portfolio manager/cluster and LFA ratings    Fund portfolio manager’s explanation for performance rating    Fund portfolio manager’s explanation for differences in funding decision  with the principal recipient request and LFA recommendation   Fund portfolio manager’s check-offs on an array of grant decision-making  issues such as performance evaluation, financial considerations, future  and outstanding requirements, etc.
Fund portfolio manager’s explanation for array of contextual issues (e.g.,  governance, general management, and external factors)
Fund portfolio manager’s explanation for required documents and  In the phase 2 files we focused on the Grant Scorecard, as this is the key  Global Fund decision-making document for this phase, although we  performed checks on several other documents as well. We checked for the  presence or absence of the following pieces of information in the  following phase 2-related documents; all items are for the Grant Scorecard  unless otherwise noted:    Evaluation unit’s explanation for nonconcurrence with fund portfolio    General explanation for difference between fund portfolio manager/cluster  recommendation and the LFA’s recommendation    Evaluation unit’s explanation for difference between fund portfolio  manager/cluster recommendation and the LFA’s recommendation   General explanation for difference in the recommended amounts between  Global Fund financial review and cluster leader phase 2   General explanation for difference in the recommended amounts between  LFA and fund portfolio manager/cluster leader (i.e. the Global Fund)  phase 2   Fund portfolio manager/cluster identification of material issues with  reliability of principal recipient’s data   Fund portfolio manager/cluster explanation for phase 2 rating   Fund portfolio manager/cluster explanation for phase 2 recommendation  (LFA phase 2 assessment) report-specific conditions identified by LFA for  ‘Conditional Go’ recommendations   Time-bound actions identified by portfolio manager/cluster for  ‘Conditional Go’ recommendations To assess the Global Fund’s progress in implementing a risk assessment  model and early warning systems for its grants, we followed up on the risk  assessment model and early warning system described in our 2005 report  with Global Fund officials. In addition, we reviewed international  standards for risk that establish the criteria for assessing risk developed  by the Committee on Sponsoring Organizations of the Treadway  Commission. We also discussed the need for a risk assessment model and  early warning system with key Global Fund staff, key Global Fund  partners, UNAIDS, World Bank, and OGAC, and with the Global Fund  Inspector General and LFA officials. In addition, we reviewed Global Fund  board notes and the Global Fund’s draft outline of Guiding Principles for a  Risk Management Framework. In their technical review of our draft,  Global Fund officials clarified the distinction between the risk assessment  model that was not implemented and the current early alert and response  system, and they provided additional information on the current status of  the early alert and response system. We modified the risk assessment  section of the draft based on our analysis of this information.
To assess how the Global Fund oversees and measures the capacity and  performance of local fund agents, we established criteria for good  practices in performance assessment, consulted key reports and  documents, met with relevant personnel, conducted structured interviews  with LFA country teams, and analyzed the structured interview data using  a content analysis framework. We consulted GAO experts in performance  assessment and contracting, good practice documents from the  Government Performance and Results Act, performance-based  contracting literature, internal control literature, and training development  literature. (See app. II) To understand Global Fund policies and  requirements regarding LFA performance, we reviewed Global Fund  documentation, including guidelines for LFA work, the Global Fund’s  operational policy manual, LFA manager terms of work, and LFA  framework contract forms, as well as relevant evaluative reports published  by the Global Fund and development experts. We meet with key Global  Fund officials in Geneva, Switzerland, including the fund portfolio  managers and members of the Portfolio Services and Projects Team,  representatives from all LFA central teams, selected LFA subcontractors,  and recognized experts on the Global Fund to discuss the current  practices and potential difficulties of the LFA system. We also reviewed  the methodology and results of a survey of Geneva-based Global Fund  secretariat staff who are users of LFAs conducted by a Global Fund  contractor in November of 2006.
Additionally, to support our review of issues related to LFA capacity and  performance, we conducted structured telephone interviews with a  nonprobability sample of 12 LFA country teams out of a total of 102 LFA  country teams worldwide. We selected country teams for this sample  based on criteria such as the size and amount of the grants that the LFA  oversaw, whether the LFA was located within the country, and whether  the LFA acted as a mentor to other LFA country teams. The purpose of  these 12 interviews was to obtain information on the views of these  important participants in the Global Fund process; we did not obtain a  representative perspective of the views of all 102 country teams. We pre- tested our questions with two of our initial respondents and refined our  questions based on their input. We followed up via e-mail with our  respondents to obtain supplementary information. To summarize the  open-ended responses and developed categories for the analysis, we first  grouped interview responses by general categories, based on our  objectives, criteria, and inductive observation. Within each of these  general categories, we then further grouped the data into subcategories  based on the emerging themes. To ensure the validity and reliability of our  analysis, these subcategories were reviewed by a methodologist, who  proposed modifications. After discussion of these suggestions, we  determined a final set of subcategories. An analyst and a methodologist  independently tallied the number of respondents providing information in  each subcategory, compared their tallies, and reconciled any variation.
section_title: Appendix II: Other Sources Cited
paragraphs: GAO, Strategic Issues Web pages on the Government Performance and  Results Act (GPRA)
GAO, Internal Control Management and Evaluation  Tool. GAO-01- 1008G. Washington D.C.: Aug. 2001.
Office of Federal Procurement Policy, Seven Steps to Performance-Based  Services Acquisition, see:  www.acqnet.gov/Library/OFPP/BestPractices/pbsc    GAO, Contract Management: Guidance Needed for Using Performance- based Service Contracting. GAO-02-1049. Washington, D.C.: Sept. 23, 2002.
subsections: section_title: Analyses of LFA Capacity and Performance
paragraphs: GAO, Contract Management: Guidance Needed for Using Performance- Based Service Contracting. GAO-02-1049. Washington, D.C.: 2002.
GAO, Internal Control Management and Evaluation Tool. GAO-01-1008G.
Washington, D.C.: 2001.
Kruse, Stein-Erik, and Jens Claussen. Review of the Roles, Functions and  Performance of Local Fund Agents. Centre for Health and Social  Development and Nordic Consulting Group; 2004.
Office of Federal Procurement Policy, Seven Steps to Performance-Based  Services Acquisition. See:  www.acqnet.gov/Library/OFPP/BestPractices/pbsc   Radelet, Steven et al. Challenges and Opportunities for the New  Executive Director of the Global Fund: Seven Essential Tasks. Global  Fund Working Group, Center for Global Development, Washington D.C.:  October 2006.
Radelet, Steven. The Global Fund to Fight AIDS, TB, and Malaria:  Progress, Potential, and Challenges for the Future. Center for Global  Development, Washington D.C.: 2004.
section_title: Appendix III: Comments from the Global Fund
section_title: Appendix IV: Comments from Departments of State and Health and Human Services and U.S. Agency for International Development
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, Audrey Solis, Assistant Director;  Tom Zingale; Jeffrey Baldwin-Bott; Bruce Kutnick; Barbara Steel-Lowney;  David Dornisch; and Reid Lowe made key contributions to this report.  Kathryn Bernet, Antoine Clark, Etana Finkler, Hynek Kalkus, and Tracy             Guerrero provided technical support.