President's Emergency Plan for AIDS Relief: Agencies Can Enhance Evaluation Quality, Planning, and Dissemination
Released: 2012-05-31T00:00:00

Letter

Background

OGAC establishes overall PEPFAR policy and program strategies and coordinates PEPFAR program activities. In addition, OGAC allocates PEPFAR resources from the Global Health and Child Survival account to PEPFAR implementing agencies, primarily CDC and USAID. The agencies execute PEPFAR program activities through agency headquarters offices and interagency teams consisting of PEPFAR implementing agency officials in the countries and regions with PEPFAR- funded programs (PEPFAR country and regional teams). OGAC coordinates these activities through its approval of operational plans, which serve as annual work plans and document planned investments in, and the anticipated results of, HIV/AIDS-related programs. OGAC provides annual guidance on how to develop and submit operational plans.

In fiscal years 2009 through 2011, OGAC approved operational plans representing $11.7 billion in PEPFAR program activities. These activities fall primarily in three broad program areas—prevention, treatment, and care—and 18 related program areas. Program activities aimed at preventing HIV infection and at treating those infected each represented about 30 percent of approved PEPFAR funding, while activities aimed at caring for AIDS patients represented about 20 percent. The remaining approximately 20 percent funded a variety of other program areas, such as health systems strengthening and building laboratory infrastructure. Figure 1 summarizes approved funding for these program areas in fiscal years 2009 through 2011.

To carry out activities in these program areas, CDC and USAID use implementing mechanisms—grants, cooperative agreements, and contracts—with a variety of implementing partners. These partners include partner country governments, nongovernmental and international organizations, and academic institutions. CDC and USAID used more than 3,000 implementing mechanisms in fiscal years 2008 through 2010.

CDC and USAID offices employ a wide variety of individuals and organizations to conduct PEPFAR evaluations, including implementing agency officials, consultants, and academic institutions as well as partner government organizations and implementing partners. Evaluation teams sometimes comprise representatives from several of these organizations. OGAC coordinates, and PEPFAR implementing agencies also engage in, several related activities that support evaluation, such as oversight of implementing partners, routine performance planning and reporting, biological and behavioral health surveillance, baseline studies and needs assessments, and development of health management information systems.

PEPFAR evaluations are subject to common evaluation standards defined in various agency-specific and governmentwide guidance. This guidance includes CDC’s Framework for Program Evaluation in Public Health and USAID’s evaluation policy and Automated Directives System guidance. In addition, GAO published guidance on designing evaluations and assessing social program impact evaluations.

Also, in September 2010, the AEA published a framework to guide the development and implementation of federal agency evaluation programs and policies. The framework offers a set of general principles intended to facilitate the integration of evaluation activities with program management. These principles include developing evaluation policies and procedures; developing evaluation plans; ensuring independence of evaluators in designing, conducting, and determining findings of their evaluations; ensuring professional competence of evaluators; and disseminating evaluation results publicly and in a timely fashion.

PEPFAR Agencies Have Evaluated a Broad Range of PEPFAR Programs, but Results Are Not Fully Supported in Many Evaluations

OGAC, CDC, and USAID managed and conducted evaluations of a wide variety of PEPFAR programs that were ongoing during fiscal years 2008 through 2010. However, we found that many of these evaluations— particularly evaluations managed by PEPFAR country and regional teams—did not consistently adhere to common evaluation standards, in many cases calling into question the evaluations’ support for their findings, conclusions, and recommendations.

OGAC, CDC, and USAID Evaluated a Broad Range of Programs

OGAC, CDC, and USAID provided 496 evaluations addressing programs ongoing during fiscal years 2008 to 2010 in all PEPFAR program areas relating to HIV/AIDS treatment, prevention, and care. Of these 496 evaluations, 18 were public health evaluations (PHE), managed by OGAC; 42 were program evaluations provided by CDC and USAID headquarters officials; and 436 were program evaluations provided by CDC and USAID country and regional team officials. (For more information about these evaluations, see app. III.)  OGAC-managed evaluations. OGAC provided 18 PHEs that CDC and USAID had completed as of November 2011 under an OGAC- managed approval, implementation review, and reporting process. The completed PHEs addressed the following program areas: prevention of mother-to-child transmission, testing and counseling, adult care and support, adult treatment, sexual prevention, and pediatric care and support. In addition, OGAC indicated that 82 other PHEs had been initiated as of November 2011. According to OGAC, PHEs are intended to assess the effectiveness and impact of PEPFAR programs; compare evidence-based program models in complex health, social, and economic contexts; and address operational questions related to program implementation within existing and developing health systems infrastructures. OGAC guidance states that these evaluations focus on strategies to increase program efficiency and impact to guide program development and inform the public, using rigorous quantitative or qualitative methods that permit broad generalization. For all PHEs, OGAC requires PEPFAR country and regional teams to submit evaluation concepts or protocols for approval by an interagency subcommittee and requires periodic progress and closeout reports.  CDC and USAID headquarters-managed evaluations. CDC headquarters officials provided 20 evaluations in the following program areas: blood safety, injection safety, adult treatment, pediatric treatment, and strategic information. USAID headquarters officials provided 22 evaluations in the following program areas: abstinence/be faithful, sexual prevention, orphans and vulnerable children, strategic information, and health systems strengthening programs. Four CDC and USAID headquarters evaluations addressed more than one program area.  Country and regional team-managed evaluations. CDC and USAID officials representing 31 PEPFAR country and 3 regional teams provided a total of 436 evaluations; CDC officials provided 185 evaluations, and USAID officials provided 251 evaluations. The evaluations addressed 18 program areas related to PEPFAR prevention, treatment, and care, with about one-fifth of the evaluations addressing activities in more than one program area (see fig. 2). CDC and USAID officials also provided copies of evaluation protocols and statements of work, indicating that additional evaluations had been initiated. Further, based on our analysis of a randomly selected sample of 78 evaluations, we estimate that 51 percent of the evaluations used qualitative methods, 35 percent used quantitative methods, and 14 percent used a mix of quantitative and qualitative methods. In addition, evaluations provided by USAID tended to employ qualitative methods (32 of 48 evaluations), while those provided by CDC tended to use quantitative methods (20 of 30 evaluations). (See app. III for additional results of our analysis.)

Our assessments of judgmental and randomly selected samples of PEPFAR evaluations indicate that many—particularly those managed by PEPFAR country and regional teams—contain findings, conclusions, and recommendations that are not fully supported. To determine the extent to which these elements are supported, we synthesized our assessments of the extent to which evaluations generally adhered to several common evaluation standards defined in guidance issued by CDC, USAID, and GAO. Specifically, we considered whether the evaluations describe the program to be evaluated and its objectives, the purpose of the evaluation, and the criteria used to reach conclusions about the achievement of the program’s objectives. We also considered the extent to which evaluations incorporate appropriate designs, sample selection methods, measures, and data collection and analysis methods.

All OGAC-managed PHEs that we reviewed generally adhered to these standards and thus their findings, conclusions, and recommendations were fully supported. We found similar results for most CDC and USAID headquarters’ program evaluations we reviewed. However, PEPFAR country and regional teams’ evaluations did not consistently adhere to common evaluation standards, and thus, in most cases, their findings, conclusions, and recommendations were not fully supported.

OGAC-managed evaluations. Our assessment of seven OGAC- managed PEPFAR PHEs indicates that they all generally adhered to common evaluation standards, and thus their findings, conclusions, and recommendations were fully supported. All of the evaluations that we reviewed identified program and evaluation objectives and used appropriate measures, and most used appropriate evaluation designs and data collection and analysis methods. Three of the evaluations employed fully appropriate sampling methods. Table 1 summarizes our assessments of these evaluations.

CDC and USAID headquarters-managed evaluations. Our assessment of 15 CDC and USAID headquarters-managed evaluations indicates that most generally adhere to common evaluation standards. As a result, we found that findings, conclusions, and recommendations were fully supported in 9 evaluations and partially supported in 6 evaluations. Most of the evaluations employed appropriate evaluation designs, measures, and data collection and analysis methods. However, 7 evaluations did not fully identify the evaluation criteria, and 8 did not employ fully appropriate sampling methods. Table 2 summarizes our assessments of these evaluations.

Country and regional team-managed evaluations. We found that evaluations managed by country and regional teams, which make up the bulk of all PEPFAR program evaluations, did not consistently adhere to common evaluation standards. Based on our analysis of a randomly selected sample of country and regional team evaluations, we estimate that findings, conclusions, and recommendations were fully supported in 41 percent of all evaluations provided to us by country and regional teams, partially supported in 44 percent of these evaluations, and not supported in 15 percent of these evaluations. We estimate that 24 percent of these evaluations did not identify any evaluation criteria, and more than half did not employ evaluation designs, sampling methods, measures, or data collection and analysis methods that were fully appropriate. For example, an evaluation of activities for providing care to orphans and vulnerable children drew conclusions about results and made recommendations, based almost exclusively on favorable anecdotal information collected from selected program participants and beneficiaries. As a result, the objectivity and credibility of these evaluations’ findings, conclusions, and recommendations are in question. Table 3 summarizes our assessments of these evaluations.

Further analysis of the results of our assessments showed that evaluations using qualitative methods were more likely to contain results that were partially supported or not supported than evaluations using quantitative methods. (See app. III for additional results of our analysis.)

PEPFAR Policies and Procedures Do Not Fully Adhere to AEA Evaluation Principles Relating to Planning, Independence, and Dissemination

State, OGAC, CDC, and USAID have developed policies and procedures that apply to evaluations of PEPFAR programs, as called for in the AEA Roadmap. However, they have not fully adhered to other AEA Roadmap principles regarding evaluation planning, independence and competence of evaluators, and dissemination of evaluation results. First, OGAC has not developed PEPFAR evaluation plans at the program level or required the development of such plans in individual countries and regions, limiting its own ability to ensure that evaluation resources are appropriately targeted. Second, State, OGAC, CDC, and USAID guidance does not specify how to document the independence and competency of evaluators, and almost half of the evaluations we reviewed did not provide sufficient information to fully determine whether evaluators were free of conflicts of interest. Finally, not all evaluation reports are available online, thus limiting their accessibility and usefulness to PEPFAR decision makers and other stakeholders.

State, OGAC, and PEPFAR Implementing Agencies Have Issued Evaluation Policies and Procedures

In accordance with AEA principles, State, OGAC, CDC, and USAID have issued policies and procedures that are applicable to PEPFAR program evaluation.  State evaluation policy. In February 2012, State’s Bureau of Resource Management issued an evaluation policy that applies to all State bureaus and OGAC. The policy provides a framework for implementing evaluations of State’s various programs and projects and encourages evaluations for programs and projects at all funding levels.  OGAC operational plan guidance. According to OGAC officials, OGAC generally has deferred to implementing agency policies. OGAC also issues annual guidance to PEPFAR implementing agencies for preparation of their operational plans. OGAC’s fiscal year 2012 operational plan guidance to PEPFAR country and regional teams, issued in August 2011, addresses some elements of evaluation. The guidance differentiates three types of evaluation and research: basic program evaluation, which focuses on descriptive and normative evaluation questions; operations research, which focuses on program delivery and optimal allocation of resources; and impact evaluation, which measures the change in an outcome attributable to a particular program.  CDC evaluation framework. In September 1999, the Program Evaluation Unit at CDC’s Office of the Associate Director for Program issued an evaluation framework for CDC programs. The framework summarizes essential elements of program evaluation, clarifies program evaluation steps, and reviews standards for effective program evaluation, among other things. According to CDC’s Chief Evaluation Officer, as of May 2012, CDC plans to issue evaluation guidelines and recommendations as well as additional guidance for using the evaluation framework.  USAID evaluation policy. In January 2011, USAID’s Bureau for Policy, Planning, and Learning revised evaluation policy to supplement existing evaluation guidance in USAID’s Automated Directive System. According to USAID, this revised policy was intended to address a decline in the quantity and quality of evaluation practice within the agency in the recent past. The policy clarifies for USAID staff, partners, and stakeholders the purposes of evaluation; the types of evaluations that are required and recommended; and USAID’s approach for conducting, disseminating, and using evaluations. Among other things, the policy sets forth the purposes of evaluation, the roles and responsibilities of USAID operating units, and evaluation requirements and practices for all USAID programs and projects. The policy requires all USAID operating units to consult with program office experts to ensure that scopes of work for external evaluations meet evaluation standards. The policy also states that operating units, in collaboration with the program office, must ensure that evaluation draft reports are assessed for quality by management and through an in-house peer technical review.

PEPFAR Lacks Evaluation Plans

OGAC has not yet developed a program-level PEPFAR evaluation plan or required implementing agencies or country and regional teams to develop evaluation plans as called for by the AEA Roadmap.  OGAC. State’s recently issued evaluation policy requires that each State bureau, including OGAC, develop and submit a bureauwide evaluation plan that encompasses major policy initiatives and new programs as well as existing programs and projects. According to a senior OGAC official, at the time of our review, OGAC was discussing with State’s Bureau of Resource Management how it will comply with this new requirement.  CDC and USAID headquarters. OGAC defers to implementing agencies to plan evaluations of their headquarters-managed PEPFAR program activities, but CDC and USAID have not developed evaluation plans for such activities included in recent headquarters operational plans. OGAC’s 2011 guidance for developing the headquarters operational plan requires a plan for technical area program priorities but does not address evaluation planning. Similarly, the fiscal year 2012 guidance does not include a requirement for an evaluation plan.  Country and regional teams. OGAC defers to PEPFAR country and regional teams to plan evaluations of their program activities, but does not require that the teams develop and submit annual evaluation plans. OGAC’s 2011 guidance on developing country and regional operational plans urges country and regional teams to prioritize program evaluation in order to make PEPFAR programs more effective and sustainable. In addition, OGAC’s fiscal year 2012 guidance calls for country and regional teams to address monitoring and evaluation in describing individual implementing partners’ activities. However, neither the 2011 guidance nor the 2012 guidance instructs all country teams to develop evaluation plans. We reviewed PEPFAR country and regional operational plans for fiscal year 2011 and found that they did not include evaluation plans. Instead, these documents generally included (1) descriptions of ongoing or planned evaluations and related activities (e.g., surveillance) in program area narrative summaries and (2) descriptions of monitoring and evaluation activities in implementing partner activity narratives.

In our analysis of information provided by country and regional teams, as well as CDC and USAID headquarters, we did not detect an evaluation rationale or strategy. Based on responses to our survey of CDC and USAID officials in 31 PEPFAR country and 3 regional teams, we calculated that evaluations had been conducted or were ongoing for about one-third of these countries’ program activities in fiscal years 2008 through 2010. In addition, based on these officials’ responses, we found similar percentages of ongoing and completed evaluations across the broad program areas of prevention, treatment, and care. We also analyzed CDC and USAID headquarters officials’ responses to our survey and found that evaluations had been conducted or were ongoing for about half of the PEPFAR program activities managed by agencies’ headquarters and implemented during fiscal years 2008 to 2010. However, we found no relationships between the percentages of program activities with ongoing or completed evaluations and budgets at the country, program area (i.e., prevention, treatment, or care), or program activity levels.

Evaluator Independence and Qualifications Are Not Consistently Documented

State, CDC, and USAID policies and procedures address the independence of evaluators but do not consistently require that evaluation reports identify the evaluation team or address whether there are any potential conflicts of interest. In addition, some agency policies and procedures address the need to ensure that evaluators have appropriate qualifications, but none require that evaluations document those qualifications or certify that they are adequate.  State. State’s recently issued evaluation policy addresses evaluator independence and integrity, stating that evaluators should be free from program managers and not subject to their influence. This policy does not address evaluator qualifications.  OGAC. OGAC’s operational plan guidance to country and regional teams does not address the independence or professional qualifications of evaluators. According to OGAC officials, OGAC defers to implementing agency evaluation policies.  CDC. CDC’s evaluation framework addresses the need to assemble an evaluation team with the needed competencies, highlighting the importance of ensuring that evaluators have no particular stake in the results of the evaluation. The CDC evaluation framework also discusses appropriate ways to assemble an evaluation team.  USAID. USAID’s evaluation policy recommends that most evaluations be external and requires a disclosure of conflicts of interest for all evaluation team members. In addition, USAID’s evaluation policy requires that evaluation-related competencies be included in staffing selection policies.

Our analysis of a randomly selected sample of evaluations submitted by 31 PEPFAR country and 3 regional teams found that the evaluations often did not address whether evaluators have potential conflicts of interest, as called for by the AEA Roadmap. We estimate that 27 percent of the evaluations fully addressed potential conflicts of interest, 59 percent partially addressed the issue, and 14 percent did not address the issue. In addition, while we were unable to determine whether potential conflicts of interest existed with the information provided in some of the evaluation reports, it appeared that there were evaluations in which potential conflicts of interest existed but were not addressed. For example, one evaluation report, relating to strengthening a partner country’s nongovernmental HIV/AIDS organizations, indicated that the evaluation team was employed by the program activity’s implementing partner, but the report did not address potential conflict of interest.

Furthermore, some country and regional program evaluations sometimes did not provide enough identifying information about evaluators to allow an assessment of evaluator independence or qualifications. We estimate that 86 percent of the evaluations fully identified the evaluators, while 14 percent provided either partial or no information. For example, an evaluation report we reviewed relating to HIV prevention program activities in one region named the organization that conducted the evaluation but did not provide any information on the evaluation team members. Moreover, we were unable to find any information about this organization in an online search based on the limited information available in the report.

PEPFAR Evaluations Are Not All Publicly Disseminated

Agency policies and procedures generally support dissemination of evaluation results, but OGAC, CDC, and USAID have not ensured that evaluation methods, data, and evaluation results are made fully and easily accessible to the public.  State. State’s newly released evaluation policy requires bureaus to submit evaluations to a central repository.  OGAC. OGAC officials told us that the office supports dissemination of the results of important global HIV/AIDS research and evaluations to a variety of stakeholders. For example, OGAC officials noted that the PEPFAR website contains information on PEPFAR results as well as monitoring and evaluation guides. OGAC officials also noted that dissemination strategies are a common component of evaluation protocols and the procurement mechanisms that fund them. In addition, OGAC maintains an intranet site, which is accessible to PEPFAR implementing agency officials and contains information about evaluation. However, OGAC does not have a mechanism for publicly and systematically disseminating evaluation results.  CDC. CDC policy advises that effort is needed to ensure that evaluation findings are disseminated appropriately but does not require online dissemination of evaluation reports. CDC officials told us that they recently made changes to CDC’s public website, which, as of April 2012, includes some information on program evaluations. In addition, CDC’s Division of Global HIV/AIDS (DGHA) Science Office maintains a catalog of published journal articles coauthored by DGHA officials. However, CDC does not maintain a complete online inventory of evaluations.  USAID. USAID’s policy states that evaluation findings should be shared as widely as possible with a commitment to full and active disclosure. USAID requires submission of completed evaluations to the Development Experience Clearinghouse (DEC), the agency’s online repository of research documentation, but does not enforce this requirement. In 2010, USAID reported that practices for disseminating evaluation results were generally limited, that dissemination practices varied across the agency, and that the requirement to submit completed evaluations to the DEC had not been fully enforced. Additionally, USAID found that documents in the DEC were sometimes difficult to find. In February 2012, USAID also found that missions had reported submitting only 20 percent of their evaluations to the DEC in fiscal year 2009.

Although documents submitted by 31 PEPFAR country and 3 regional teams showed that CDC and USAID have disseminated evaluation findings within these countries and regions in several ways, we found no publicly accessible and easily searchable Internet source for PEPFAR program evaluations. We received abstracts from annual meetings and conferences, presentations to partner government officials and stakeholders, published journal articles, and periodic agency reports, which may be publicly accessible via the Internet. However, as of the time of our review, our searches of five key websites generated far fewer PEPFAR evaluations than the 496 evaluations we received from country teams, CDC and USAID headquarters, and OGAC. We searched PubMed, the U.S. National Library of Medicine’s online database, but a search using “PEPFAR” and “evaluation” as search terms generated seven results. Likewise, as of April 2012, our search of USAID’s DEC, using “HIV/AIDS” and “evaluation” as search terms, generated 87 results, including some that were not evaluations, but USAID officials, in response to our request, later provided us nearly 300 evaluations. We also found some evaluations at two USAID-maintained websites, OVCsupport.net and AIDStar-One, but neither site was comprehensive or fully searchable. In addition, a website called Global HIV M&E Information provides a repository of voluntarily submitted monitoring and evaluation resources; however, we found few evaluations of PEPFAR programs.

Conclusions

PEPFAR’s authorizing legislation emphasizes the importance of program evaluation as a tool for OGAC to ensure, among other things, that funds are spent on programs that show evidence of success. State, CDC, and USAID have demonstrated a clear commitment to program evaluation by conducting a wide variety of program evaluations that address at least one activity in each PEPFAR program area. However, many evaluations managed by PEPFAR country and regional teams lack fully supported findings, conclusions, and recommendations, evidenced by a lack of general adherence to common evaluation standards. Without fully supported findings, conclusions, and recommendations, these PEPFAR program evaluations have limited usefulness as a basis for decision making and may supply incomplete or misleading information for managers’ and stakeholders’ efforts to direct PEPFAR funding to programs that produce the desired outcomes and impacts.

State, CDC, and USAID have demonstrated their commitment to program evaluation by developing policies and procedures that apply to evaluations, in accordance with established general principles. However, without a requirement that country and regional teams prepare and submit annual evaluation plans—for example, as a component of operational plans—OGAC is unable to ensure that program activities are subject to appropriate levels of evaluation. Moreover, without documentation of the independence and competence of PEPFAR program evaluators, OGAC, agency program managers, and other stakeholders have limited assurance that evaluation results are unbiased and credible. Finally, unless evaluation results are publicly and systematically disseminated and made easily searchable online, program officials and public health researchers may be unable to assess the credibility of their findings or use them for program decision making.

Recommendations for Executive Action

We recommend that the Secretary of State direct the U.S. Global AIDS Coordinator to take the following four actions in collaboration with CDC and USAID to enhance PEPFAR evaluations: 1. develop a strategy to improve PEPFAR implementing agencies’ and country and regional teams’ adherence to common evaluation standards; 2. require implementing agency headquarters and country and regional teams to include evaluation plans in their annual operational plans; 3. provide detailed guidance for implementing agencies and country and regional teams on assessing, ensuring, and documenting the independence and competence of PEPFAR program evaluators; and 4. increase the online accessibility of PEPFAR program evaluation results.

Agency Comments

We provided a draft of this report to State, HHS’s CDC, and USAID. Responding jointly with CDC and USAID, State OGAC provided written comments (see app. IV). CDC and USAID also provided technical comments, which we incorporated as appropriate.

In its written comments, State agreed with our recommendations and, emphasizing the interagency nature of the PEPFAR program, indicated that it will coordinate with PEPFAR agencies to implement our recommendations. First, State explained that it will work with PEPFAR implementing agencies to carry out the agencies’ evaluation policies and practices, which State deemed generally consistent with AEA principles, and will develop strategies to ensure the appropriate application of common evaluation standards. Second, State responded that it will work through PEPFAR interagency processes to develop PEPFAR program evaluation plans, which it noted could be included in annual PEPFAR operational plans. Third, State will work with PEPFAR implementing agencies to put in place guidance to document program evaluators’ independence and qualifications. Fourth, State affirmed that OGAC will collaborate with PEPFAR implementing agencies to develop strategies for improving dissemination of evaluation results and will use PEPFAR’s public website to link to agencies’ online resources.

We are sending copies of this report to the Secretary of State, the Office of the U.S. Global AIDS Coordinator, U.S. Agency for International Development’s Office of HIV/AIDS, the Department of Health and Human Services’ Office of Global Affairs, the Centers for Disease Control and Prevention’s Division of Global HIV/AIDS, and appropriate congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

This report (1) identifies President’s Emergency Plan for AIDS Relief (PEPFAR) evaluation activities and examines the extent to which evaluation results are supported and (2) examines the extent to which PEPFAR policies and procedures adhere to established principles for the evaluation of U.S. government programs. To identify PEPFAR program evaluations and examine the extent to which they generated supported evaluation results, we collected and analyzed program evaluation documents provided by Centers for Disease Control and Prevention (CDC) and U.S. Agency for International Development (USAID) officials in the 31 PEPFAR countries and 3 regions with PEPFAR country or regional operational plans in fiscal year 2010, as well as the Department of State’s (State) Office of the U.S. Global AIDS Coordinator (OGAC) and CDC and USAID headquarters officials. To examine the extent to which PEPFAR program evaluation policies and procedures adhered to principles in the American Evaluation Association’s (AEA) An Evaluation Roadmap for a More Effective Government (AEA Roadmap), we reviewed the general principles for conducting federal government program evaluations, as well as OGAC, State, USAID, and CDC policies and guidance. In addition, we surveyed CDC and USAID officials in the 31 PEPFAR countries and 3 regions with PEPFAR annual country or regional operational plans in fiscal year 2010, as well as CDC and USAID headquarters officials, regarding ongoing and completed evaluations. Finally, we conducted interviews with OGAC, CDC, and USAID officials in Washington, D.C., and Atlanta, Georgia.

Survey of PEPFAR Officials

To survey PEPFAR country and regional team officials, we took the following steps: 1. We consulted with OGAC and CDC and USAID headquarters officials and decided to use implementing mechanism as a proxy for a program activity. We determined that using implementing mechanisms was the only viable unit of analysis to estimate the percentage of PEPFAR programs with evaluations because (1) OGAC officials maintained updated data on implementing mechanisms and (2) PEPFAR officials regularly used and understood data on implementing mechanisms. However, in some of these cases, if the broader program was evaluated, not all implementing mechanisms under the larger program were necessarily evaluated. We also recognized that evaluations may not be appropriate for all implementing mechanisms (such as those that provide funding for staffing costs). To the extent possible, we eliminated these implementing mechanisms from our analysis. 2. We obtained lists of program activities for fiscal years 2008 through 2010 from OGAC for each country and region. We then analyzed program activities by country (or region) and agency; the lists included identification numbers, names, and partner names for each of the program activities. Each survey tool then contained a list of program activities relevant to the country or regional team. 3. Based on GAO and OGAC guidance, we developed the following working definition of evaluation: Evaluations are systematic studies to assess how well a program is working. Evaluations are often conducted by experts external to the program, either inside or outside the agency. Types of evaluations include process, outcome, impact, or cost-benefit analysis. 4. We developed a survey tool for ongoing and completed evaluations of PEPFAR programs. We consulted with OGAC and CDC and USAID headquarters officials about the survey tool and made revisions as appropriate. For example, based on input from CDC and USAID headquarters officials, we determined that some PEPFAR evaluations could address several implementing mechanisms. In addition, in some of these cases, if a broader program (e.g., national treatment program) was evaluated, not all implementing mechanisms under the broader program were necessarily evaluated. In response, we included questions in our survey prompting PEPFAR officials to indicate whether an implementing mechanism has been evaluated as part of a broader evaluation of several implementing mechanisms. 5. We tested the survey tool with officials in two PEPFAR countries— Angola and Ethiopia—and finalized the survey tool based on discussions with these officials. 6. We sent the final survey tool to PEPFAR country contacts (PEPFAR coordinators and CDC and USAID officials) identified by OGAC and CDC and USAID headquarters. The survey tool instructed CDC and USAID country or regional team officials to provide “yes” or “no” responses to the following questions for each implementing mechanism in the country’s (or region’s) agency-specific lists: Is this one of your agency’s fiscal year 2008-2010 country or regional operational plan program activities?  Has at least one evaluation specific to this implementing mechanism been completed?

Is at least one evaluation specific to this implementing mechanism ongoing?  Has at least one evaluation covering, but broader than, this implementing mechanism been completed?

Is at least one evaluation covering, but broader than, this implementing mechanism ongoing?

We also prompted the country or regional officials to provide additional information for each implementing mechanism, such as explanations for program activities that do not belong to the agency and identification of duplicate program activities. Officials were instructed to either e-mail the completed surveys to GAO or upload them to a website regularly used by OGAC and country and regional teams for submitting and sharing planning and reporting documents.

In some cases, we met with country or regional team officials via telephone, or corresponded via e-mail, to clarify the purpose of the survey, the questions themselves, and the evaluation document request as well as to correct anomalies and ask follow-up questions. One GAO analyst also attended the May 2011 PEPFAR implementing agency annual meeting in Johannesburg, South Africa, to provide information about the survey and evaluation document request to PEPFAR country and regional team officials also attending the annual meeting. We received responses from all 31 PEPFAR countries and 3 regions with fiscal year 2010 operational plans.

Using a similar survey tool, we also conducted surveys of CDC and USAID headquarters officials regarding program activities managed by agency headquarters and listed in PEPFAR headquarters operational plans for 2008 through 2010.

Analysis of Survey Responses

To analyze country and regional teams’ survey responses, we made the following assumptions regarding the survey responses: If officials did not provide a response to the question “Is this one of your agency’s fiscal year 2008-2010 country or regional operational plans program activities?” we included that implementing mechanism in the analysis. Program activities with responses of “no” or “duplicate” were eliminated from the analysis.

If officials did not respond to any of the four questions regarding ongoing or completed evaluations, we assumed that there were no ongoing or completed evaluations for that implementing mechanism.

In addition, we reviewed narrative comments provided by country and regional team officials. We recognized that evaluations may not be appropriate for all implementing mechanisms (such as those that provide funding for staffing costs). To the extent possible, we eliminated these implementing mechanisms from our analysis. Based in part on our review of the narrative comments, we flagged and eliminated implementing mechanisms with evidence indicating that the implementing mechanism was either “to be determined” (i.e., the agency had yet to make an award to an implementing partner), related to staffing costs, related to strategic information and monitoring and evaluation, recently begun, a duplicate of another implementing mechanism, or listed in error.

Once the survey responses were ready for analysis, we calculated the summary statistics that are reported in the body of the report. We also included the survey responses provided by officials in CDC and USAID headquarters in the analysis. To check the reliability of the data analysis, a second independent analyst reviewed the statistical programs used to analyze the data for accuracy.

Program Evaluation Document Collection

In addition to our survey of CDC and USAID officials in the 31 countries and 3 regions with fiscal year 2011 operational plans, we requested program evaluation documents. To do this, the survey tool instructions prompted CDC and USAID officials to provide documentation of completed and ongoing evaluations. Specifically, for implementing mechanisms where officials indicated that at least one evaluation had been completed, we requested documentation—such as an evaluation report—of all such completed evaluations. For implementing mechanisms where officials indicated that at least one evaluation was ongoing, we requested documentation—such as terms of work or an evaluation plan.

We generally advised country and regional team officials to err on the side of inclusion when in doubt about whether to submit documentation of ongoing and completed evaluations. We instructed these officials to e- mail, or, in some cases, mail electronic versions of the program evaluation documents to GAO, or to upload them to a website regularly used by OGAC and country and regional teams for submitting and sharing planning and reporting documents.

In response to this document request, we received more than 1,350 documents. For example, we received documentation of ongoing or planned evaluations, such as statements of work or evaluation protocols and protocol approval forms. We also received meeting minutes, trip reports, financial review and audit documents, presentation slides, abstracts, and conference posters. To determine which documents met our definition of evaluation, we reviewed each of these documents and categorized them as meeting the definition of evaluation or not, following a set of decision rules. For example, we included data quality assessments, costing studies that compared costs and explained cost differences, and analyses of surveillance data pre- and postintervention. We excluded surveillance studies that simply reported the results of a surveillance activity (but did not link it to a specific program or intervention); needs assessments, baseline studies, and situation analyses; trip and site visit reports; and pre- and postevent (e.g., workshop) questionnaires or surveys. We identified and eliminated duplicate documents. This categorization was checked by a second analyst and yielded 436 program evaluations. We believe that this final set of evaluations constitutes an essentially full universe of PEPFAR country and regional program evaluation documents.

In addition to the program evaluation documents collected from CDC and USAID officials in PEPFAR countries and regions, we requested documents from OGAC related to PEPFAR public health evaluations. We also requested evaluation documents related to PEPFAR program managed by CDC and USAID headquarters from officials at each agency’s headquarters. OGAC provided copies of 18 completed public health evaluations, CDC headquarters provided copies of 22 completed evaluations, and USAID headquarters provided copies of 24 completed evaluations.

We reviewed the program evaluation documents submitted by PEPFAR country and regional teams as well as CDC and USAID headquarters officials. We identified whether each program evaluation was ongoing or completed as well as which program area or areas (e.g., prevention, treatment, care, or other) were evaluated. To do this, we used program categories defined by OGAC’s fiscal years 2011 operational plan guidance, resulting in the program areas and related areas reported in the report. This categorization was checked by a second analyst. Table 4 provided descriptions of the PEPFAR program areas.

To determine the degree to which these evaluations were conducted in adherence with common evaluation standards, we used an assessment tool to systematically conduct in-depth analyses of a probability sample of the evaluations submitted by the PEPFAR country and regional teams and a nonprobability sample of the evaluations submitted by OGAC and CDC and USAID headquarters officials. Our PEPFAR evaluation assessment tool was based on an assessment tool used for a prior GAO report, which we updated using guidance on evaluation from USAID, CDC, the Organization for Economic Cooperation and Development (OECD), and GAO. We piloted the assessment tool with three PEPFAR program evaluation documents provided by CDC and USAID headquarters officials and revised the evaluation assessment as appropriate. After piloting and revising the tool, we finalized the tool and used it to conduct the in-depth analyses of program evaluation documents. Table 5 lists the questions and supporting questions included in the assessment tool.

To allow us to generalize to the entire set of evaluations provided by PEPFAR country and regional teams, we randomly selected a sample of 84 of 436 evaluations submitted by CDC and USAID officials in 31 PEPFAR countries and 3 regions. The list of all evaluations was sorted by total approved operational plan budgets for each country or region for fiscal years 2008 through 2010, so that a systematic sample would ensure representation of countries with relatively large, medium, and small budgets for fiscal years 2008 through 2011.

After sampling, 6 evaluations—including, for example, baseline and feasibility studies—were found to be out of scope, resulting in a final sample of 78. Results based on random probability samples are subject to sampling error. The sample we drew for our survey is only one of a large number of samples we might have drawn. Because different samples could have provided different estimates, we express our confidence in the precision of our particular sample results as a 95 percent confidence interval. This is the interval that would contain the actual population values for 95 percent of the samples we could have drawn. The margin of error associated with proportion estimates is no more than plus or minus 11 percentage points at the 95 percent level of confidence and estimates of totals have a margin of error no larger than 44 evaluations.

For the 18 public health evaluations submitted by OGAC, as well as the 20 and 22 evaluations submitted by CDC and USAID headquarters, respectively, we selected a nonprobability sample based on the type of program (e.g., prevention, treatment, care, or other) evaluated as well as country or countries addressed by each evaluation. Because this is a nonprobability sample, the results of our assessments of these evaluations cannot be used to make inferences about all evaluations managed by OGAC and CDC and USAID headquarters. However, they do represent a mix of the types of evaluations managed by OGAC and CDC and USAID headquarters.

Assessing Program Evaluation Documents

Using our evaluation assessment tool, we conducted in-depth analyses of the evaluation documents submitted by the PEPFAR country and regional teams and also those submitted by OGAC, USAID, and CDC headquarters. To do so, one analyst conducted an initial review of the evaluation document and then completed the evaluation assessment tool. The analyst also recorded basic information about each evaluation, including title, author, date of publication, and the country or countries included in the evaluation. For each of the questions in the assessment tool (see table 1), analysts were instructed to (1) respond using “yes,” “no,” “partial,” “not sure,” or “not applicable” and (2) summarize or cite relevant information from the evaluation documents. Analysts then were instructed to weigh the evidence and answers to these questions and provide “yes,” “no,” “partial,”, “not sure,” or “not applicable” responses for each category. Based on the analysis of the elements addressed in the assessment tool, analysts determined the extent to which each evaluation’s findings, conclusions, and recommendations were supported using “yes,” “no,” “partial,” or “not sure” as their responses. This overall determination was not based on a tally of responses to individual elements in the evaluation assessment tool, but rather a synthesis of these responses and an assessment of the contribution of each element to the overall support for the evaluation’s findings, conclusions, and recommendations. To help ensure consistency in the application of the standards and questions, the assessors met weekly during the assessment period to clarify the instructions and discuss their observations. After each assessment was complete, a second analyst independently verified the results of the analysis by reviewing the program evaluation document and the completed evaluation assessment tool. In cases where the two analysts did not concur on the results, or where there was a “not sure” response, they met to discuss the evidence and documented a final determination. All the results for the evaluation assessment tools were then entered into a spreadsheet and analyzed.

Analyzing Data Generated by Evaluation Assessments

To assess potential associations between key attributes of the sample of 78 evaluations we randomly selected, we calculated chi-square tests and the associated odds ratios for all pairs of the following variables: agency, methods used, evaluation type, and program type. Key results from these analyses are presented in the report. Additional results can be found in appendix III. We also employed logistic regressions to assess which of these variables (i.e., agency, methods used, evaluation type, and program type) had the strongest effects on the extent to which sampled evaluations contained support for findings, conclusions, and recommendations.

Assessing State, OGAC, CDC, and USAID Evaluation Policies

To assess State, OGAC, CDC, and USAID evaluation policies, we developed an assessment tool based on nine AEA Roadmap principles. For each principle, we developed a question or series of questions asking how the policies addressed the AEA Roadmap principles. One analyst reviewed each agency’s policy and filled out the tool by citing evidence that would support the policy’s consistency with the AEA Roadmap principle, or a conclusion that no evidence could be found to support adherence to the principle. The analyst then concluded whether the policy was consistent with each principle assessed. A second analyst conducted a review of the completed assessment tools and either concurred with or disputed the conclusion for each principle. In cases where the two analysts did not concur, they met to discuss the evidence and made a final determination.

To determine the extent to which operational plans contained evaluation plans, we reviewed OGAC’s fiscal year 2011 and 2012 annual guidance to implementing agency headquarters regarding development of the annual PEPFAR headquarters operational plan. We documented instances where the guidance addressed program evaluation and determined whether it constituted instructions to develop an evaluation plan. We conducted similar analysis of OGAC’s fiscal year 2011 and 2012 annual guidance to PEPFAR country and regional teams to identify instances where the guidance addressed evaluation and, finally, to determine whether the guidance constituted instructions for developing evaluation plans. In addition, we assessed 11 of the 33 country operational plans and 2 of the 3 regional operational plans submitted to OGAC for fiscal year 2011, the most recent year in which plans were available. We documented instances where these operational plans discussed evaluation and whether they contained evaluation plans.

To determine the extent to which the program evaluations documented potential conflicts of interest and the identity of evaluators, we included questions on these two elements in our evaluation assessment tool. Analysts were instructed to respond using “yes,” “no,” or “partial” to these questions and to cite relevant evidence. After each assessment was complete, a second analyst verified the results of the analysis by reviewing the program evaluation document and the completed evaluation assessment tool. In cases where the two analysts did not concur on the results, they met to discuss the evidence and documented a final determination. All the results for the evaluation assessment tools were then entered into a spreadsheet and analyzed.

We searched five Internet databases referenced by OGAC, CDC, and USAID officials to determine the public accessibility of PEPFAR program evaluations. These five sites included the Development Experience Clearinghouse (http://dec.usaid.gov/index.cfm), PubMed (http://www.ncbi.nlm.nih.gov/pubmed/), OVCsupport.net (http://www.ovcsupport.net/s/), AIDSTAR-One (http://www.aidstar- one.com/), and Global HIV M&E Info (https://www.globalhivmeinfo.org/Pages/HomePage.aspx). For each of these websites, we conducted searches using keywords that would capture any PEPFAR-related program evaluations or documentation, such as “PEPFAR,” “evaluation,” and “HIV/AIDS.” Where applicable, we then captured the results and counted the number of documents that could reasonably be considered documentation of a PEPFAR program evaluation.

We conducted this performance audit from August 2011 to May 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO Evaluation Definitions and Standards

Past GAO work has emphasized evaluation as a key source of information to help agency officials and Congress make decisions about the programs they oversee. GAO distinguishes performance measurement—the ongoing monitoring and reporting of program accomplishments—from evaluation, which is defined as individual, systematic studies conducted periodically or on an ad hoc basis to assess how well a program is working. Further, according to GAO guidance, experts external to the program, program managers, or both conduct evaluations to examine the performance of a program within a given context to understand not only whether a program works but also how to improve results. GAO guidance identifies four types of evaluation:  Process evaluation. This type of evaluation assesses the degree to which a program is operating as it was intended. It typically assesses program activities’ conformance to statutory or regulatory requirements, program design, and professional standards or customer expectations.  Outcome evaluation. This type of evaluation assesses the degree to which a program achieves its outcome-oriented objectives. It focuses on outputs and outcomes (including unintended effects) to judge program effectiveness, but may also assess program process to understand how outcomes are produced.

Impact evaluation. This is a form of outcome evaluation that assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of the program. Impact evaluation is used when external factors are known to influence the program’s outcomes, in order to isolate the program’s contribution to achievement of its objectives.  Cost-benefit or cost-effectiveness analysis. This type of evaluation compares a program’s outputs or outcomes with the costs to produce them. Cost-effectiveness analysis assesses the cost of meeting a single objective and can be used to identify the least costly alternative for meeting that goal.

In addition, GAO guidance provides basic information about the more commonly used evaluation methods; introduces key issues in planning evaluation studies of federal programs to best meet decision makers’ needs; and describes different types of evaluations for answering varied questions about program performance, the process of designing evaluation studies, and key issues to consider in ensuring overall study quality. Further, the guidance recommends standards for evaluation design, including establishing evaluation objectives, identifying constraints, and assessing the appropriateness of the evaluation design.

Appendix III: Statistical Comparison of PEPFAR Evaluations

We conducted a statistical analysis of the adequacy of support for findings in evaluations provided to us by CDC and USAID, to determine whether the adequacy of support differed by agency, by methods used, or by type of evaluation. Our analysis indicated that fully supported findings were more likely in CDC’s evaluations than in USAID’s evaluations; in evaluations that used quantitative methods than in evaluations that used qualitative or mixed methods; and in cost-benefit or impact evaluations, as well as outcome evaluations, than in process evaluations. However, while CDC’s evaluations’ findings were more likely to be fully supported than USAID’s evaluations’ findings, the difference was not statistically significant after we accounted for the method used in the evaluations. This lack of statistical significance suggests that the difference was driven partly by the agencies’ choice of evaluation method.

Table 6 shows technical details of our statistical analysis of the level of support for findings in CDC and USAID evaluations.

In table 6, the chi-square statistics at the base of each of the three panels show that the adequacy of support for findings varied significantly between the two agencies and differed significantly based on the methods used and type of evaluations. The odds ratios in the far-right column show that the odds of evaluations’ being fully supported were 3.6 times greater for CDC than for USAID; 18 times greater for quantitative evaluations than for qualitative or mixed-methods evaluations; 23 times greater for cost-benefit or impact evaluations than for process evaluations; and 3.7 times greater for outcome evaluations than for process evaluations.

In addition, we estimated binary logistic regression models to determine whether the difference in adequacy of support for findings in CDC’s and USAID’s evaluations resulted from differences in the methods used or differences in the types of evaluations conducted. Table 7 shows the odds ratios that result from fitting logistic regression models to estimate the effects of the three different factors (agency, methods used, and type of evaluation) on the adequacy of support for findings. Models 1, 2, and 3 are bivariate models, which regress “support” on dummy variables for agency, methods used, and type of evaluation, with each variable considered one at a time. These produce the same odds ratios that we obtained from the observed data in table 6. In contrast, model 4 estimates the effects of agency and methods simultaneously, and model 5 estimates the effects of agency and type of evaluation. In comparing these models, we found that controlling for the methods used (model 4) rendered insignificant the differences between agencies in adequacy of support for findings, whereas controlling for type of evaluation (model 5) did not.

Appendix IV: Comments from the Department of State

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Jim Michels, Assistant Director; Todd M. Anderson; Chad Davenport; David Dornisch; Lorraine Ettaro; Justin Fisher; Brian Hackney; Kay Halpern; Fang He; Reid Lowe; Grace Lui; and Erika Navarro made key contributions to this report. In addition to these staff, the following GAO staff assisted by conducting in-depth assessments of selected evaluations: Sada Aksartova, Gergana Danailova-Trainor, Leah DeWolf, Rachel Girshick, Jordan Holt, Kara Marshall, Jeff Miller, Steven Putansu, Mona Sehgal, and Doug Sloane. Sushmita Srikanth and Katy Crosby assisted with quality assurance reviews.

Related GAO Products

President’s Emergency Plan for AIDS Relief: Program Planning and Reporting. GAO-11-785. Washington, D.C.: July 29, 2011.

Global Health: Trends in U.S. Spending for Global HIV/AIDS and Other Health Assistance in Fiscal Years 2001-2008. GAO-11-64. Washington, D.C.: October 8, 2010.

President’s Emergency Plan for AIDS Relief: Efforts to Align Programs with Partner Countries’ HIV/AIDS Strategies and Promote Partner Country Ownership. GAO-10-836. Washington, D.C.: September 20, 2010.

President’s Emergency Plan for AIDS Relief: Partner Selection and Oversight Follow Accepted Practices but Would Benefit from Enhanced Planning and Accountability. GAO-09-666. Washington, D.C.: July 15, 2009.

Global HIV/AIDS: A More Country-Based Approach Could Improve Allocation of PEPFAR Funding. GAO-08-480. Washington, D.C.: April 2, 2008.

Global Health: Global Fund to Fight AIDS, TB and Malaria Has Improved Its Documentation of Funding Decisions but Needs Standardized Oversight Expectations and Assessments. GAO-07-627. Washington, D.C.: May 7, 2007.

Global Health: Spending Requirement Presents Challenges for Allocating Prevention Funding under the President’s Emergency Plan for AIDS Relief. GAO-06-395. Washington, D.C.: April 4, 2006.

Global Health: The Global Fund to Fight AIDS, TB and Malaria Is Responding to Challenges but Needs Better Information and Documentation for Performance-Based Funding. GAO-05-639. Washington, D.C.: June 10, 2005.

Global HIV/AIDS Epidemic: Selection of Antiretroviral Medications Provided under U.S. Emergency Plan Is Limited. GAO-05-133. Washington, D.C.: January 11, 2005.

Global Health: U.S. AIDS Coordinator Addressing Some Key Challenges to Expanding Treatment, but Others Remain. GAO-04-784. Washington, D.C.: June 12, 2004.

Global Health: Global Fund to Fight AIDS, TB and Malaria Has Advanced in Key Areas, but Difficult Challenges Remain. GAO-03-601. Washington, D.C.: May 7, 2003.