President's Emergency Plan for AIDS Relief: Shift toward Partner- Country Treatment Programs Will Require Better Information on Results
Released: 2013-05-13T00:00:00

Letter

Background

OGAC is responsible for establishing overall PEPFAR policy and program strategies and allocating funds from the Global Health and Child Survival account to PEPFAR implementing agencies, primarily CDC and USAID.These agencies execute PEPFAR program activities through agency headquarters offices and in-country interagency teams (PEPFAR country teams) and their implementing partners in the 33 countries and three regions with PEPFAR-funded programs as of fiscal year 2012. OGAC coordinates these activities through its approval of operational plans, which document work plans, budgets, and the anticipated results of HIV/AIDS-related programs. OGAC also provides annual guidance to PEPFAR country teams on how to develop and submit operational plans. For fiscal years 2009 through 2012, OGAC approved country operational plan budgets totaling over $16 billion. Country operational plan activities fit broadly in three areas: prevention, treatment, and care. Other program budget areas are laboratory infrastructure, strategic information, and health systems strengthening.

To promote a more sustainable approach to combating HIV/AIDS, characterized by PEPFAR countries’ strengthened capacity, ownership, and leadership, the 2008 Leadership Act authorized the U.S. government to establish partnership frameworks with partner countries. These frameworks are 5-year joint strategic agreements for cooperation between the U.S. government and partner governments to combat HIV/AIDS in the partner country through technical assistance and support for service delivery, policy reform, and coordinated funding commitments. As of February 2013, the U.S. government had signed 22 PEPFAR partnership frameworks. According to OGAC guidance, a key expectation of the frameworks is that partner-country governments will become better prepared to assume primary responsibility for their responses to HIV/AIDS. Moreover, PEPFAR’s 2012 “blueprint” defines country ownership as the end state in which partner countries lead, manage, and coordinate the efforts needed to ensure that the AIDS response is effective, efficient, and durable.

This is typically measured by CD4 (cluster of differentiation antigen 4) count in a sample of blood. CD4 cells are a type of white blood cell that fights infection. Along with other tests, the CD4 count helps determine the strength of the immune system, indicates the stage of the HIV disease, guides treatment, and predicts the disease’s progress. threshold in its laboratory criteria and by recommending treatment for all people coinfected with HIV and tuberculosis, thereby expanding the number of people eligible for treatment. Based on WHO guidelines, each country is expected to establish country-specific guidelines on when to initiate treatment for these groups. UNAIDS estimated at the end of 2011 that, on the basis of WHO’s 2010 guidelines, 15 million people in low- and middle-income countries needed treatment; of these, an estimated 8 million people are on treatment. People who are HIV positive but not yet eligible for treatment generally may seek access to care and support services as well as regular checkups and laboratory monitoring. People eligible for treatment should receive antiretroviral (ARV) drugs as well as checkups and monitoring to assess the effectiveness of treatment.People on treatment also receive various care and support services such as treatment of opportunistic infections including tuberculosis coinfection, nutritional support, and programs to promote adherence to treatment and remaining on treatment (patient retention). People on treatment are expected to take ARV drugs on a continuing, lifelong basis. . Prior to 2010, WHO’s guidelines recommended treatment for all people with CD4 counts of less than 200 cells/mm. reporting on PEPFAR results. According to the guidance, these indicators are intended to demonstrate progress in the fight against HIV/AIDS while also promoting responsible program management. In addition, among other things, the guidance establishes a distinction between national results and PEPFAR direct results. The guidance defines national results as achievements of all contributors to a partner country’s HIV/AIDS program and defines PEPFAR direct results as achievements of the PEPFAR program through its funded activities. (See app. II for a summary of OGAC criteria for assessing PEPFAR direct support.) With regard to treatment programs, the guidance instructs PEPFAR country teams providing direct support for treatment services to report to OGAC using the PEPFAR direct indicators. In addition, the guidance directed these country teams, as well as PEPFAR country teams providing technical assistance and other support to build partner- country capacity for managing treatment programs, to report on one national indicator. Table 1 summarizes these indicators.

Data Indicate Progress in Achieving Treatment Program Results, but Some Indicators Have Limitations

From fiscal year 2010 through 2012, OGAC reported PEPFAR results in terms of three primary indicators: (1) the number of people currently on treatment directly supported by PEPFAR (PEPFAR direct number of people on treatment), (2) the percentages of eligible people receiving treatment in partner countries (national treatment coverage rates), and (3) the percentage of adults and children known to be alive and on treatment 12 months after starting treatment (PEPFAR direct treatment retention rates). However, two of these indicators have limitations that could affect their usefulness. Regarding the first indicator, although the number of people on treatment directly supported by PEPFAR has increased significantly, this indicator alone does not provide complete information needed for assessing PEPFAR’s contributions to partner countries’ treatment programs. Regarding the third indicator, 10 PEPFAR country teams reported percentages of adults and children known to be alive and on treatment 12 months after starting treatment that exceeded 80 percent. However, the treatment retention data are not always complete and have other limitations, which OGAC acknowledged and is taking steps to address. In addition to these limitations, OGAC lacks a common set of indicators for monitoring quality assurance efforts. Although OGAC indicated in 2010 that it would establish a common set of indicators to monitor the results of PEPFAR’s efforts to improve the quality of treatment programs, it has not yet done so.

PEPFAR Exceeded Its Fiscal Year 2012 Direct Treatment Target, but This Indicator Does Not Fully Reflect PEPFAR Contributions to Partner- Country Treatment Programs

Responding to treatment-related requirements in the 2008 Leadership Act, OGAC reports on the number of people currently on treatment directly supported by PEPFAR as a key indicator of program results. This number is calculated by determining the number of people who ever started treatment at facilities where PEPFAR directly supports treatment services, minus patients who died, stopped treatment, transferred out, or have unknown treatment outcomes. (See app. II for a summary of OGAC guidance on determining whether people can be counted as receiving direct services through PEPFAR.) PEPFAR met or exceeded annual targets for this indicator in fiscal years 2004 through 2012. Currently, this indicator is used to track treatment program expansion and to assess progress toward PEPFAR’s target of providing direct support for treatment for 6 million people by the end of fiscal year 2013. For fiscal year 2012, PEPFAR’s target for this indicator was 5 million people. According to data provided by OGAC, the number of people currently on treatment directly supported by PEPFAR has steadily increased from about 67,000 people in 11 countries in fiscal year 2004 to more than 5.1 million in 23 countries in fiscal year 2012. (See table 2.)

Furthermore, the number of people on treatment directly supported by PEPFAR was about half of the total number of people on treatment in all low- and middle-income countries, which UNAIDS estimated at 8 million in 2011 (see fig. 1).

As PEPFAR has begun to shift resources toward providing technical assistance and other support to help partner countries build their capacity to manage treatment programs, PEPFAR’s direct treatment indicator has increasingly fallen short of reflecting results of PEPFAR’s contributions to partner countries’ treatment programs. Specifically, this indicator does not reflect the expansion of partner countries’ treatment programs that PEPFAR technical assistance and other support for building treatment program management capacity have made possible. These efforts include, among others, activities such as implementing revised treatment guidelines, assisting partner-country district and national health officials with treatment facility oversight, and training and mentoring treatment facility staff. In several PEPFAR partner countries, for example, PEPFAR implementing partners providing direct treatment services have begun transferring stable patients to other treatment providers, including an often expanding number of local public and private health clinics, many of which receive PEPFAR-funded technical assistance and other support. In part because of this PEPFAR assistance, these providers also have begun increasing the number of people they enroll in treatment. In such cases, the PEPFAR direct treatment indicator may not account for these people. In the past, PEPFAR’s direct results often were equivalent to the national number of people receiving services, including treatment, according to PEPFAR’s 2010 Next Generation Indicators Reference Guide. However, as PEPFAR increases its efforts to build partner-country capacity to manage treatment programs through technical assistance and other support, PEPFAR’s direct treatment indicator alone does not provide complete information for assessing PEPFAR’s contributions to partner countries’ treatment programs.

PEPFAR’s 2010 Next Generation Indicators Reference Guide noted that OGAC was working on a method for deriving PEPFAR direct results from partner-country national-level indicators but had not yet devised one. The guidance stated that the new method would take into account the percentage of PEPFAR funding that contributes to partner-country programs. In addition, OGAC and PEPFAR country teams have considered other factors to determine PEPFAR’s contribution to partner- country treatment programs. For example, in their fiscal year 2011 annual reports to OGAC, seven PEPFAR country teams reported the proportion of all treatment facilities receiving PEPFAR support or the percentage of all patients on treatment directly supported by PEPFAR. Some country teams noted that neither method fully accounted for PEPFAR’s contributions in these countries. As of February 2013, according to a senior OGAC official, OGAC had drafted a method for representing PEPFAR contributions based on proportional financial support to partner- country program results but had not finalized the method or revised its guidance to PEPFAR country teams.

PEPFAR Has Helped Increase Partner Countries’ National Treatment Coverage Rates

Increases in the number of people on treatment have helped improve partner countries’ national treatment coverage rates—generally defined as the percentage of eligible people receiving treatment. According to the most current UNAIDS and PEPFAR data, 8 of the 23 countries where PEPFAR directly supported treatment services in 2011 achieved estimated treatment coverage rates of 80 percent or more (see table 3).Although the remaining 15 countries fell short of this target, almost all of these countries have increased their estimated treatment coverage rates since 2009, according to our analysis of UNAIDS data.

Since fiscal year 2010, OGAC has required PEPFAR country teams providing direct support for treatment services to track treatment retention rates, an indicator defined as the percentage of adults and children known to be alive and on treatment 12 months after starting treatment. In addition to being an essential indicator of treatment program outcomes— a higher retention rate indicates that more people on treatment are surviving—facilities’ retention rates are used by OGAC and PEPFAR country teams as a proxy indicator of treatment program quality. Of the 23 PEPFAR country teams directly providing treatment services, 20 provided data on this indicator in their fiscal year 2012 reports to OGAC. Ten of the 20 teams reported retention rates at or above 80 percent for facilities where PEPFAR implementing partners support direct treatment services. (See table 4.)

However, PEPFAR patient retention data have several key limitations.

The data are not always complete. PEPFAR’s reported retention rates reflect only the rates at facilities where PEPFAR directly supported treatment services and that were able to properly collect and report retention data. In addition, in their fiscal year 2012 reports to OGAC, three PEPFAR country teams reported that data for this indicator were not available. Several country teams noted problems in obtaining data from partner-country systems or from all sites where PEPFAR directly supports treatment services. Several country teams also reported concerns about data quality, including limited understanding of how to collect these data.

Methods and definitions vary. For example, PEPFAR country teams accounted for patients transferring to or from treatment facilities differently. In addition, country teams used different definitions to count numbers of patients lost to follow-up (i.e., those with unknown outcomes, including possible death, treatment cessation, or self- transfer to another treatment facility). Under the current WHO definition, a patient may be considered lost to follow-up 90 days after the last scheduled appointment, but the definition may be adjusted depending on the stage of a patient’s treatment.

Data on treatment retention are rarely available for key populations, including children and adolescents, injecting drug users, men who have sex with men, and sex workers. These populations are at higher risk for HIV infection and may face specific challenges that make it more difficult to retain them in treatment programs.

Few data on long-term retention (after 24 months from the start of treatment) are available. Although OGAC guidance encourages PEPFAR country teams to use data for cohorts of patients to track retention and survival at 24, 36, and 48 months, PEPFAR does not have a retention indicator that extends beyond 12 months.

OGAC officials stated that OGAC has taken several steps to improve the fiscal year 2012 PEPFAR treatment retention data. First, OGAC clarified guidance to PEPFAR country teams regarding how to calculate and report on this indicator. Second, PEPFAR implementing agencies conducted data quality assessments in three PEPFAR countries. As a result, according to OGAC officials, three more PEPFAR country teams were able to report on the treatment retention indicator in fiscal year 2012 than in fiscal year 2011. Furthermore, OGAC officials stated that data completeness is a priority for the current fiscal year 2013 and that they will help PEPFAR country teams with reporting retention data.

In addition to routinely reported information on treatment program results, various studies of treatment programs, although they may not represent national treatment program conditions and may be limited by incomplete data, provide information that can be useful for improving the results of treatment programs. Appendix III contains examples of information provided by studies we identified.

PEPFAR Supports Efforts to Improve Treatment Program Quality, but OGAC Has Not Established Indicators to Assess Results

PEPFAR country teams engage in a number of activities, often characterized as technical assistance or support, whose aim is to assure the quality of treatment programs. Seventeen of the 22 countries with PEPFAR partnership frameworks identified efforts to improve treatment program quality as a key goal shared by partner countries and PEPFAR. However, OGAC has not established a common set of indicators to assess results of these activities. We identified several examples of quality assurance activities, such as developing and implementing partner-country quality improvement strategies, including roles and responsibilities for health facility supervision; establishing treatment site-level performance improvement plans, including quality improvement council meetings to identify solutions to problems affecting service quality; and training health facility managers and staff to track and use facility-level performance data.

OGAC’s 2010 Next Generation Indicators Reference Guide sought to emphasize program quality indicators to help strengthen partner countries’ HIV/AIDS programs. To this end, the guidance added patient retention rate to the list of essential, reported treatment program indicators. The same guidance also recommended tracking data for several indicators of treatment program quality—for example, the number of patients with a documented CD4 or viral load test, the number of patients who have attended the recommended number of clinical visits, and the percentage of health facilities providing treatment using CD4 monitoring in line with partner-country guidelines or policies.

In the three countries we visited, we found that PEPFAR implementing partners were using a wide range of indicators to report on their quality assurance activities. Examples of such indicators include percentages of (1) HIV-positive patients assessed for treatment eligibility, (2) patients on treatment who adhere to the dosing instructions and other requirements for taking ARV medicines, and (3) patients with good clinical outcomes. However, even where indicators in the three countries were generally the same, definitions varied slightly. For example, in reporting on the indicator of appointments kept, three PEPFAR implementing partners—all providing quality assurance assistance to treatment facilities in the same country—used two different definitions. One implementing partner reported on the percentage of HIV-positive patients who kept their appointments in the previous month or quarter, while the other two implementing partners reported on the percentage of HIV-positive patients who missed their appointments.

OGAC’s 2010 Next Generation Indicators Reference Guide stated that additional guidance on quality assurance indicators for PEPFAR implementing agencies would be forthcoming. However, as of February 2013, OGAC has not issued this additional guidance. The lack of PEPFAR-wide guidance on quality assurance indicators and definitions inhibits development of standardized measurement tools used by PEPFAR country teams to monitor treatment facilities supported by PEPFAR and ultimately track the results of PEPFAR’s quality assurance efforts, including technical assistance and other support.

PEPFAR Supports Countries’ M&E Systems, but Weaknesses Limit the Timeliness and Completeness of Treatment Results Data

To track the results of partner-country treatment programs and to help ensure that they are effective, PEPFAR supports countries’ monitoring and evaluation (M&E) systems. While some progress has been made in expanding and upgrading them, these systems often are unable to produce timely and complete treatment data, limiting their usefulness for managing programs and reporting program results. PEPFAR country teams fulfill many M&E functions at facilities where PEPFAR supports direct treatment services, and they assist partner countries in carrying out their M&E responsibilities by providing staff, training, and technical assistance and other support. Nevertheless, partner countries’ M&E systems continue to face a number of weaknesses. Consequently, PEPFAR country teams primarily use data drawn from PEPFAR-specific systems to report on PEPFAR treatment program results. OGAC has not yet issued guidance needed to support PEPFAR’s continued progress in transitioning to using partner-country M&E systems for program management and results reporting.

PEPFAR Provides Technical Assistance, Training, and Staff to Strengthen Partner Countries’ M&E Systems

Among other activities, OGAC technical guidance to PEPFAR country teams calls for the development of partner countries’ M&E systems, among other activities. Fully functioning M&E systems are essential for effective patient monitoring and patient management and also generate data that PEPFAR country teams and partner countries need to track treatment program results. All countries and regions with PEPFAR partnership frameworks identified strengthening M&E systems as a key goal shared by partner countries and PEPFAR. To support this goal, according to fiscal year 2012 PEPFAR operational plans, PEPFAR country teams provide, among other things, technical assistance, training, and staff to treatment facilities, district health offices, and national ministries of health to collect, aggregate, and report treatment program information through partner countries’ M&E systems (see sidebar). This support included recruiting, mentoring, and training health facility staff and district health officials responsible for collecting, analyzing, aggregating, and reporting data through the country’s M&E system. PEPFAR country teams were also assisting partner countries with conducting surveys and surveillance, such as those needed to estimate the number of people with advanced HIV infection. In addition, according to OGAC officials, as of February 2013, 15 PEPFAR partner countries have expressed interest in a single, open-source data system, and several of these countries have started implementation. This health information software tool supports collecting, analyzing, and reporting national health data, including data on treatment programs. According to these officials, PEPFAR will support partner-country expansion of the software tool to include PEPFAR and Global Fund reporting in two additional PEPFAR partner countries.

Weaknesses Remain in Partner Countries’ M&E Systems

PEPFAR country teams have documented weaknesses in a number of partner countries’ M&E systems, which are at various stages of implementation. Our review of PEPFAR country teams’ fiscal year 2011 annual reports to OGAC and fiscal year 2012 operational plans identified two key challenges for partner countries’ M&E systems.

Partner countries’ M&E systems often are unable to produce complete and timely data, thus limiting their usefulness for patient, clinic, or program management. In their 2011 annual reports to OGAC, 12 PEPFAR country teams cited timeliness of partner-country treatment program data as a challenge, often because partner-country reporting time frames differed from the U.S. government fiscal year. In addition, three PEPFAR country teams noted that data provided by partner countries’ M&E systems were incomplete—not all provinces or treatment facilities reported data into the system. In addition, one PEPFAR country was not able to collect data on the number of patients currently receiving treatment, because the partner country provided data only on the cumulative number of patients who had ever started treatment; the country team noted that this had the likely effect of inflating the partner country’s treatment coverage rate.

Furthermore, partner-country health officials, often lacking technical capacity, do not always use available data for decision making. Our review of the PEPFAR country teams’ operational plans found that 23 teams cited the need to improve data use at treatment facilities or other levels of the health care system. For example, one country team reported that partner-country health officials tended to focus on data collection for reporting rather than for policy, planning, and program decision making. Another country team reported that lack of data reporting by treatment facilities limited analysis of treatment patients across facilities, and a third country team noted that human resource limitations and weak research capacity impeded use of M&E data. In addition, studies of PEPFAR partner countries’ M&E systems that we identified provided additional information; appendix III provides a summary of this information.

PEPFAR Country Teams Aim to Integrate M&E Systems, but OGAC Has Not Established Minimum Standards to Guide This Effort

Because of the limitations associated with the data from national M&E systems, PEPFAR country teams primarily use data drawn from systems created specifically for reporting PEPFAR treatment program results. OGAC’s 2010 Next Generation Indicators Reference Guide states that PEPFAR country teams may need to rely on these systems in the short term but should continue working to integrate these systems into partner countries’ M&E systems. Our review of PEPFAR country teams’ operational plans found that PEPFAR country teams maintained PEPFAR program performance management systems to routinely collect, compile, and analyze patient monitoring and management data from the health facilities where PEPFAR directly supports treatment services. In addition, PEPFAR country teams use these data to generate their semiannual and annual reports to OGAC. PEPFAR country teams supplement information from their own systems with data from partner countries’ M&E systems, including numbers of patients on treatment and rates of treatment coverage.

OGAC’s 2010 Next Generation Indicators Reference Guide recommends several indicators for tracking partner-country outcomes related to strengthening health systems, such as the existence of M&E plans and the percentage of health facilities with record-keeping systems for In addition, OGAC’s technical guidance monitoring HIV/AIDS programs.to country teams for developing partner countries’ M&E systems identifies a number of key efforts, such as developing M&E leadership and organizations, improving the policy environment, and ensuring the advancement and sustainability of technical capacity in PEPFAR partner countries. The technical guidance states that these efforts should support national capacity building. However, OGAC has not issued guidance identifying minimum standards that data generated by partner countries’ M&E systems should meet—such as standards related to completeness and timeliness—in order for PEPFAR country teams to assess, together with partner countries and other donors, whether the systems are ready for use in PEPFAR program management and results reporting. The lack of such standards leaves uncertain the point at which partner-country M&E systems are mature enough for PEPFAR to rely on them. This uncertainty is likely to delay achievement of PEPFAR’s goal of using partner-country M&E systems to generate data for PEPFAR treatment program management and reporting.

Conclusions

Ensuring that PEPFAR treatment programs continue to improve and to operate as effectively as possible requires careful, complex monitoring and evaluation (M&E), not only of program results but also of quality assurance efforts. Available data on PEPFAR program results show some progress. In particular, PEPFAR and UNAIDS data indicate a steady increase in the number of people on treatment, improved treatment coverage rates, and high rates of patient retention at many facilities. However, PEPFAR’s contributions to expansion of partner countries’ treatment programs are not fully reflected in its results data because OGAC’s current method for deriving PEPFAR’s direct treatment indicator does not fully account for PEPFAR’s efforts to improve partner countries’ capacity to manage their treatment programs. This limits the usefulness of PEPFAR’s direct treatment indicator for assessing progress toward expanding partner-country treatment programs. Furthermore, OGAC has not yet established a common set of indicators to measure the results of PEPFAR technical assistance and other support intended to improve the quality of treatment programs. Lacking a standard set of quality assurance indicators, PEPFAR is limited in its ability to track the results of PEPFAR’s quality assurance efforts.

As PEPFAR continues to shift responsibility for managing treatment programs to partner countries, those countries will need robust M&E systems to generate the data that are indispensable for ensuring effective treatment and efficient program management. PEPFAR has dedicated resources specifically for these efforts, but problems with untimely and incomplete data collection, as well as with data use, persist. As a result, PEPFAR has to rely on the M&E systems its implementing partners have developed rather than on country-managed systems for collecting and reporting results data. In its guidance, OGAC has not yet established minimum standards that data generated by partner countries’ M&E systems should meet in order for PEPFAR country teams to assess these systems. Without such standards, uncertainty remains as to when partner-country M&E systems will be ready to be integrated with PEPFAR systems, thus delaying the achievement of PEPFAR’s goal of using partner-country M&E system data for PEPFAR treatment program management and reporting.

Recommendations for Executive Action

To ensure the outcomes and quality of treatment programs supported by PEPFAR, we recommend that the Secretary of State direct the U.S. Global AIDS Coordinator to take the following three actions in collaboration with PEPFAR implementing agencies: develop a method that better accounts for PEPFAR’s contributions to establish a common set of indicators to measure the results of treatment program quality improvement efforts; and establish a set of minimum standards for data generated by partner countries’ M&E systems, to enable PEPFAR country teams to assess those systems’ readiness for use in treatment program management and reporting.

Agency Comments and Our Evaluation

We provided a draft of this report to State, USAID, and CDC. Responding jointly with CDC and USAID, State provided written comments (see app. IV for a copy of these comments). State and CDC also provided technical comments and supplementary information relating to our findings and recommendations. In response to the technical comments, we incorporated changes to the draft report, as appropriate. After reviewing the supplementary information, we clarified our findings and recommendations relating to PEPFAR’s direct treatment indicator and its support for partner countries’ M&E systems.

In its written comments, State generally agreed with our three recommendations. First, State affirmed that it supports our recommendation to develop a method for fully accounting for PEPFAR’s contributions to partner-country treatment programs. Observing that PEPFAR’s direct treatment indicator was intended to capture only essential components of direct treatment services, State noted that PEPFAR has recently begun an effort to revise its monitoring, evaluation, and reporting framework, including an expansion of indicators that would allow for implementing partners to report on their efforts to help partner countries build capacity and develop sustainable treatment programs. Second, State also agreed with the finding leading to our recommendation regarding the development of indicators to measure the results of treatment program quality improvement efforts. State cited the need for a harmonized PEPFAR strategy on treatment quality, including key indicators, and noted steps it is taking to develop such a strategy. In addition, stressing that treatment retention indicators are relatively new and difficult to operationalize, State detailed steps PEPFAR is taking to help improve treatment retention measurement, evaluation, and performance. Third, State noted that PEPFAR supports the strengthening of partner country reporting systems and works with partner countries to help them develop such systems, both to support national programs as well as to provide data for PEPFAR and other donors. As part of these efforts, PEPFAR also works with WHO and the Global Fund on system standardization and standards for data exchange. State specifically identified an indicator developed by WHO for documenting data completeness and timeliness and stated that this indicator can be used to monitor efforts to develop these reporting systems. We agree that such an indicator could be useful for PEPFAR country teams trying to determine when partner-country M&E systems are ready to be integrated with PEPFAR systems. However, we note that PEPFAR guidance does not instruct country teams to use the WHO indicator or any other indicator for this purpose. We believe that specifying one or more indicators for PEPFAR country teams to use is important to ensure a consistent approach to systems integration across the program. Doing this would emphasize for country partners the importance of harmonizing M&E systems for mutually beneficial purposes.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of State and the U.S. Global AIDS Coordinator. The report also will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov or contact Marcia Crosse at (202) 512-7114 or crossem@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

In this report, we examine the President’s Emergency Plan for AIDS Relief’s (PEPFAR) (1) treatment program results and how the Department of State’s (State) Office of the U.S. Global AIDS Coordinator (OGAC) measures them and (2) assistance to improve partner countries’ monitoring and evaluation (M&E) systems. To address both of these objectives, we collected and analyzed information from the following sources: interviews and fieldwork; guidance documents and past GAO work; PEPFAR partnership frameworks, operational plans, and performance reports; and studies of treatment programs.

We interviewed officials from OGAC, the Department of Health and Human Services’ (HHS) Centers for Disease Control and Prevention (CDC), and the U.S. Agency for International Development (USAID) in Washington, D.C., and Atlanta, Georgia. We also conducted fieldwork in three PEPFAR partner countries—Kenya, South Africa, and Uganda—in June 2012 to obtain information on PEPFAR efforts to support partner- country treatment program outcomes and quality as well as challenges faced by PEPFAR implementing agencies and stakeholders. We selected these countries based on program size, availability of cost data, travel logistics, and other factors. We interviewed U.S. agency officials, representatives of key implementing partners, and partner government health officials and conducted visits to selected treatment facilities in these countries.

We reviewed guidance provided to PEPFAR country teams by OGAC in collaboration with CDC, USAID, and other PEPFAR implementing agencies and previous GAO work to identify criteria related to requirements for collecting, validating, and reporting treatment program These guidance documents also provided results and their measures.criteria for examining PEPFAR assistance to improve partner countries’ M&E systems. Guidance documents included OGAC’s 2010 Next Generation Indicators Reference Guide, issued in August 2009; fiscal years 2011 and 2012 operational plan guidance and associated technical considerations; and annual and semiannual performance reporting guidance. We also consulted relevant guidance and reports issued by international organizations such as the United Nations Joint Programme on HIV/AIDS (UNAIDS) and the World Health Organization (WHO). These included Global Report: UNAIDS Report on the Global AIDS Epidemic, 2012 and Organizing Framework for a Functional National HIV Monitoring and Evaluation System as well as WHO’s 2010 Antiretroviral Therapy for HIV Infection in Adults and Adolescents: Recommendation for a Public Health Approach and 2006 Patient Monitoring Guidelines for HIV Care and Antiretroviral Therapy. We also reviewed WHO’s 2011 Retention in HIV Programmes: Defining the Challenges and Identifying Solutions.

We examined PEPFAR operational plans, performance reports, and partnership frameworks. First, to identify key PEPFAR treatment program goals—including those related to treatment program supervision, M&E systems, and quality assurance—we reviewed partnership framework agreements between the United States and 22 PEPFAR partner countries Next, to identify ongoing and planned PEPFAR activities, as and regions.

As of February 2013, the United States had signed partnership frameworks with the following 22 partner countries and regions: Angola, Botswana, the Caribbean region, the Central American region, the Democratic Republic of the Congo, the Dominican Republic, Ethiopia, Ghana, Haiti, Kenya, Lesotho, Malawi, Mozambique, Namibia, Nigeria, Rwanda, South Africa, Swaziland, Tanzania, Ukraine, Vietnam, and Zambia. narrative descriptions to OGAC regarding (1) the number of adults and children with advanced HIV infection who are currently receiving treatment directly supported by PEPFAR (PEPFAR direct number of people currently on treatment); (2) the percentage of adults and children with advanced HIV infection who are receiving treatment in partner countries’ treatment programs (national coverage rates); and (3) the percentage of adults and children known to be alive and on treatment 12 months after starting treatment directly supported by PEPFAR (PEPFAR direct treatment retention rates). We reviewed and summarized the 23 country teams’ narrative descriptions accompanying each of these three PEPFAR indicators for information related to treatment program results and assistance to improve PEPFAR partner countries’ M&E systems.

In the case of the number of people on treatment directly supported by PEPFAR, to show changes over time, we analyzed aggregate data for fiscal years 2004 through 2012 provided by OGAC, which it derived from PEPFAR country teams’ semi-annual and annual reports. We previously have reviewed OGAC guidance and procedures for collecting, analyzing, In addition, to identify factors considered by and assessing these data.PEPFAR country teams when reporting on PEPFAR treatment program results, we reviewed the narrative descriptions provided in these PEPFAR country teams’ fiscal year 2011 annual reports. On the basis of these reviews, as well as interviews with OGAC officials, we determined that the PEPFAR data on number of people on treatment were sufficiently reliable for reporting totals rounded to the nearest hundred. In addition, to illustrate PEPFAR’s contribution to the number of people on treatment in low- and middle-income countries, we obtained data on the numbers of people on treatment in all low- and middle-income countries from the UNAIDS website ( 2011. UNAIDS sets international standards for these data, which it collects from national governments and uses to report on estimated numbers of people on treatment; therefore, we deemed them sufficiently reliable for the purposes of our reporting. Data on the number of people on treatment in low- and middle-income countries were not available for 2012 at the time of this report’s publication. To be able to illustrate PEPFAR’s contribution to the number of people on treatment in low- and middle-income countries in 2012, we derived a rough estimate based on ) for calendar years 2004 through changes in PEPFAR’s and UNAIDS’s reported numbers of people on treatment from 2010 to 2011. We observed that the number of people on treatment directly supported by PEPFAR made up about half of the increase in the number of people on treatment in low- and middle-income countries in these years. For fiscal years 2011 to 2012, the number of people on treatment directly supported by PEPFAR increased by about 1.2 million; if we had assumed that this number continued to make up half of the increase in low- and middle-income countries in these years, the estimated increase would have been 2.4 million. Thus, our estimate of an increase of about 1.5 million people on treatment from 2011 to 2012— leading to an estimate of 9.5 million people on treatment in low- and middle-income countries in those years—represents a conservative projection of possible scenarios.

In the case of partner countries’ treatment coverage rates, we analyzed data provided in UNAIDS’s 2012 Global Report: UNAIDS Report on the Global AIDS Epidemic as well as data available for 2009 through 2011 on UNAIDS’s website (www.unaids.org). UNAIDS sets international standards for these data, which it collects from national governments and uses to report on national treatment coverage rates. On the basis of review of treatment coverage rates reported by PEPFAR country teams to OGAC, as well as discussions with OGAC officials, we determined that UNAIDS’s treatment coverage data were the most complete and current data available and were thus sufficiently reliable for the purposes of our reporting.

In addition to information provided by PEPFAR country teams and reported by OGAC, we also drew on information from a selected set of studies of treatment programs and M&E systems in PEPFAR partner countries. We used the information from these studies to identify illustrative examples of factors affecting treatment program outcomes and M&E system strengths and weaknesses. Appendix III contains examples of information provided by studies we reviewed.

First, although we did not intend to develop an exhaustive list of all available studies, we took a number of steps to identify relevant and up- to-date studies assessing treatment programs and M&E systems in PEPFAR countries. These studies included: (1) studies collected under our previous review of evaluations of PEPFAR programs; (2) studies provided by PEPFAR country teams in the three countries we visited; (3) articles published in special issues of Health Affairs and the Journal of Acquired Immune Deficiency Syndromes dedicated to PEPFAR programs, both published in 2012; and (4) a citations review for PEPFAR public health evaluations, evaluations provided by CDC headquarters, and relevant articles appearing in Health Affairs and the Journal of Acquired Immune Deficiency Syndromes from 2009 through 2012. We identified more than 200 studies addressing our objectives.

To perform our review, we first reviewed the studies’ titles and abstracts to categorize each study according to one or more topics, such as M&E systems or treatment program retention. Focusing on studies that fell into categories related to M&E systems and to treatment program retention and patient-level outcomes, we then reviewed key sections—such as findings and conclusions—of each study to identify common themes. We also targeted studies that addressed topics covered in the body of the report, such as factors affecting treatment program retention and loss to follow-up and strengths and weaknesses of M&E systems. Having identified subsets of relevant studies, we reviewed them in more depth to verify our initial judgments about the studies’ findings and to select illustrative examples of the themes we had identified. This additional review included the development of narrative work papers synthesizing and categorizing more detailed findings and results from the selected studies. We then presented these findings and examples in appendix III. Our analysis is not a summary of the full set of studies we identified in the initial phase but rather a presentation of several key, high-level results derived from a select set of studies in both areas, supported with citations to illustrative studies.

We conducted this performance audit from May 2012 to April 2013 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Criteria for Assessing PEPFAR Direct Support

The Department of State’s (State) Office of the U.S. Global AIDS Coordinator’s (OGAC) Next Generation Indicators Reference Guide, effective beginning in fiscal year 2010 and updated in February 2013, provides a list of indicators for setting targets, monitoring results, and reporting to OGAC. The guidance distinguishes between President’s Emergency Plan for AIDS Relief (PEPFAR) direct indicators and national indicators. PEPFAR direct indicators describe the results of PEPFAR programs through its funded activities. National indicators describe the achievements of all contributors—including public and private sector organizations and other donors—to a partner country’s HIV/AIDS programs.

Furthermore, OGAC’s guidance provides a checklist for determining whether a site-specific service supported by PEPFAR can be counted in reporting on PEPFAR direct indicators. According to the guidance, to be characterized as PEPFAR direct, an activity must fulfill at least one criterion in each of the two panels shown in table 5. If an activity meets at least one of each set of criteria, PEPFAR support is assumed to be direct and to likely provide sufficient support for claiming 100 percent of the site- specific results. If an activity meets a criterion in only one of the panels, PEPFAR support may be insufficient for claiming 100 percent of site- specific results. In that case, country teams must (1) determine whether there is sufficient justification to claim the results as direct and, if there is, (2) justify the method used to estimate the appropriate fraction of the total commensurate with PEPFAR support to the site and (3) document the estimation procedures used.

Appendix III: Summary of Studies

In addition to analyzing information provided by President’s Emergency Plan for AIDS Relief (PEPFAR) country teams and reported by the Department of State’s (State) Office of the U.S. Global AIDS Coordinator (OGAC), we also performed reviews of a selected set of studies. Although these studies may not represent national treatment program conditions and may be limited by incomplete data, they provide additional information related to treatment retention and patient-level health outcomes, as well as strengths and weaknesses of partner-country monitoring and evaluation (M&E) systems. (See app. I for information on how we identified and used these studies for the purposes of our reporting.)

Treatment Retention and Patient-Level Outcomes

Treatment retention rates reported by PEPFAR country teams indicate some progress but have certain limitations related to data completeness and varying methods and definitions. Treatment program studies—which use treatment facility data not routinely reported by PEPFAR country teams to OGAC—also provide information about treatment retention, adherence, and patient loss-to-follow-up.

Several studies, although not representative of national program conditions, identified factors associated with patient loss-to-follow-up, such as advanced illness and personal economics. They also noted measures that could potentially reduce the number of patients lost to follow-up and thus increase retention rates, such as starting treatment earlier, shortening the distance to health care facilities, expanding personal outreach by community health volunteers, and using mobile phone messaging for patient follow-up. In addition, several studies that we reviewed identified interventions found to be successful for reducing the occurrence of drug resistance and treatment failure, including (1) increased outreach to patients to increase adherence to medications and (2) effective supervision and training of lower-level health facility staff. Other studies identified factors that may limit the positive health effects of treatment, such as poor access to nutrition and clean water. Such findings can serve as the basis for adopting new treatment program policies and management approaches.

Incomplete data may limit studies’ ability to fully examine patient-level health outcomes and the factors that affect them. For example, we identified one study that reported some positive adult treatment program outcomes, based on a nationally representative sample of treatment facilities in Rwanda, but noted incomplete data on patient weight and CD4 cell count, among other limitations. Another study on the prevalence of drug-resistant HIV strains noted that very few published data on drug resistance are available, particularly for the HIV strains that tend to be prevalent in low- and middle-income countries. Devising mitigation strategies for drug resistance can be difficult without these data. Likewise, one study noted a lack of data on potential adverse effects of treatment on the growth and development of pediatric patients.

Strengths and Weaknesses of Partner-Country M&E Systems

Studies of partner countries’ M&E systems note that some progress has been made in expanding and upgrading these systems. The studies cite increased use of electronic systems for health information management and reporting. For example, one study found that electronic systems, where in use, enabled health facilities to report on health indicators more easily as well as to support patient and facility management.

Nevertheless, these studies also found that partner countries’ M&E systems are unable to produce timely and complete treatment data, thus limiting their usefulness for patient, clinic, or program management. These studies cited several challenges, including the following:

Human resource limitations can hinder the full use of M&E system data. For example, one study noted that health facility staff lacked trained data managers to regularly analyze basic data.

Data may not always be analyzed when reported from the facility to the district; and data may not always be disseminated properly, or may not be used in decision making. For example, one study noted limited access to data and lack of capacity as factors negatively affecting data use.

Health facility staff may prioritize data reporting to districts, national ministries, and donors over data use, including use for improving the quality of treatment services. For example, one study found that data were used for decision making at 38 percent of health facilities reviewed and in 44 percent of districts reviewed.

Appendix IV: Comments from the Department of State

Appendix V: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the contacts named above, Jim Michels (Assistant Director), Todd M. Anderson, David Dayton, Brian Hackney, and Grace Lui made key contributions to this report. In addition, the following GAO staff provided technical assistance and other support: Sada Aksartova, Chad Davenport, David Dornisch, Lorraine Ettaro, Katherine Forsyth, Kay Halpern, Erika Navarro, and Jane Whipple.

Related GAO Products

President’s Emergency Plan for AIDS Relief: Per-Patient Costs Have Declined Substantially, but Better Cost Data Would Help Efforts to Expand Treatment. GAO-13-345. Washington, D.C.: March 15, 2013.

President’s Emergency Plan for AIDS Relief: Agencies Can Enhance Evaluation Quality, Planning, and Dissemination. GAO-12-673. Washington, D.C.: May 31, 2012.

President’s Emergency Plan for AIDS Relief: Program Planning and Reporting. GAO-11-785. Washington, D.C.: July 29, 2011.

Global Health: Trends in U.S. Spending for Global HIV/AIDS and Other Health Assistance in Fiscal Years 2001-2008. GAO-11-64. Washington, D.C.: October 8, 2010.

President’s Emergency Plan for AIDS Relief: Efforts to Align Programs with Partner Countries’ HIV/AIDS Strategies and Promote Partner Country Ownership. GAO-10-836. Washington, D.C.: September 20, 2010.

President’s Emergency Plan for AIDS Relief: Partner Selection and Oversight Follow Accepted Practices but Would Benefit from Enhanced Planning and Accountability. GAO-09-666. Washington, D.C.: July 15, 2009.

Global HIV/AIDS: A More Country-Based Approach Could Improve Allocation of PEPFAR Funding. GAO-08-480. Washington, D.C.: April 2, 2008.

Global Health: Global Fund to Fight AIDS, TB and Malaria Has Improved Its Documentation of Funding Decisions but Needs Standardized Oversight Expectations and Assessments. GAO-07-627. Washington, D.C.: May 7, 2007.

Global Health: Spending Requirement Presents Challenges for Allocating Prevention Funding under the President’s Emergency Plan for AIDS Relief. GAO-06-395. Washington, D.C.: April 4, 2006.

Global Health: The Global Fund to Fight AIDS, TB and Malaria Is Responding to Challenges but Needs Better Information and Documentation for Performance-Based Funding. GAO-05-639. Washington, D.C.: June 10, 2005.

Global HIV/AIDS Epidemic: Selection of Antiretroviral Medications Provided under U.S. Emergency Plan Is Limited. GAO-05-133. Washington, D.C.: January 11, 2005.

Global Health: U.S. AIDS Coordinator Addressing Some Key Challenges to Expanding Treatment, but Others Remain. GAO-04-784. Washington, D.C.: June 12, 2004.

Global Health: Global Fund to Fight AIDS, TB and Malaria Has Advanced in Key Areas, but Difficult Challenges Remain. GAO-03-601. Washington, D.C.: May 7, 2003.