International Education Assistance: USAID Has Implemented Primary Grade Reading Programs but Has Not Yet Measured Progress toward Its Strategic Goal
Released: 2015-05-14T00:00:00

Letter

Background

USAID’s 2011-2015 Education Strategy

USAID’s 2011-2015 education strategy, issued in February 2011, includes three goals: improved reading skills for 100 million children in primary grades by 2015 (Goal 1), improved ability of tertiary and workforce development programs to produce a workforce with relevant skills to support country development goals by 2015 (Goal 2), and increased equitable access to education in crisis and conflict environments for 15 million learners by 2015 (Goal 3). Goal 1 represents a shift from an earlier broader focus on achieving universal enrollment of children in primary school as well as improving the quality of education.

In August 2011, USAID issued guidance for implementing the education strategy, directing missions with basic education programs in non-crisis and conflict environments to narrow the focus of their interventions to achieving the goal of improving early grade reading by the end of fiscal year 2012. According to the guidance, missions that were not in conflict or crisis-affected environments were to focus resources on primary grade reading and discontinue programming in such areas as early childhood education, secondary education, or adult literacy. In addition, missions’ programming was to be based on evidence of what works to improve reading outcomes. The guidance stated that the timing of missions’ alignment of education programs with the education strategy was critical because programs required at least 3 years of implementation to contribute to the strategy’s numerical target.

USAID’s Performance and Evaluation Requirements

USAID requires that missions’ primary grade reading programs adhere to the agency’s performance monitoring and evaluation framework requirements in order to track progress annually against the full set of expected results. Performance monitoring periodically tracks changes to determine whether desired results are occurring and whether projects are implemented as designed. USAID missions and implementing partners monitor programs by using performance management plans or equivalent documents that define what is to be measured, the unit of measurement, and the level of data disaggregation, among other information. USAID guidance for implementing the education strategy requires that these plans include, at a minimum, either of two standard outcome indicators related to primary grade reading, if relevant:

The proportion of students who demonstrate by the end of two grades of primary schooling that they can read and understand the meaning of grade-level text.

The proportion of students who by the end of the primary school years are able to read and demonstrate understanding as defined by a country curriculum or standards or as agreed to by national experts.

While performance monitoring is required for all programs, USAID’s evaluation policy requires performance or impact evaluations of all large projects. The policy states that performance evaluations should focus on questions such as what a project or program has achieved, how it is being implemented, and how it is perceived and valued. Impact evaluations measure the change in a development outcome that is attributable to the project or program, based on models of cause and effect. In addition, USAID’s guidance for implementing the education strategy instructs missions to integrate plans for performance or impact evaluation into program design during the planning stages and to derive data for evaluations from reading assessments. The guidance also states that performance and impact evaluations should be provided by an external, separately contracted team.

A comparison of baseline assessment data (i.e., data collected before a reading program starts) and endline assessment data (i.e., data collected at or near the end of the program or before the end of fiscal year 2015) can be used by missions to report on the standard outcome indicators for performance monitoring and by the agency’s contractors to estimate progress toward the goal of 100 million students with improved reading skills by December 2015. USAID officials noted that the need to assess learners toward the end of a school year, which often ends in May or June, may result in some missions assessing learning outcomes later than December 2015. See U.S. Agency for International Development, 2011 USAID Education Strategy Technical Notes (Washington, D.C.: revised April 2012). least two phases of assessment are necessary to measure reading improvement.the Philippines.

USAID Basic Education Funding and Management

In fiscal year 2013—the first year in which missions’ education programs were expected to be fully aligned with the goals of the education strategy—USAID allocated about $790 million for basic education assistance divided between programs supporting the education strategy’s Goal 1—improving primary grade reading—and programs supporting Goal 3—increasing equitable access to education in crisis and conflict environments. USAID provided basic education funding to 41 countries for primary grade reading programs in fiscal year 2013. Table 1 shows USAID’s allocations for basic education for fiscal years 2010 through 2014 by appropriation account. More than half of the 2013 funding was allocated for basic education activities in countries in Sub-Saharan Africa, where baseline reading scores were among the lowest globally, according to USAID. Appendix II shows the fiscal year 2013 allocations for basic education by country.

USAID’s basic education programs are planned and implemented primarily by USAID’s overseas missions. In Washington, D.C., the Office of Education in USAID’s Bureau for Economic Growth, Education and Environment provides technical leadership, research, and field support. The office also assists in the development, implementation, and evaluation of the USAID education strategy and resource allocation priorities. In addition, USAID’s geographic bureaus advise missions on their education programs.

USAID Missions in Five Selected Countries Are Implementing, Monitoring, and Evaluating Reading Programs Consistent with USAID Guidance

In the five countries where we conducted fieldwork, the USAID missions had implemented primary grade reading programs that included intervention elements recommended in USAID’s guidance related to its education strategy. In several of the countries, the need to develop new reading materials in mother-tongue languages and train classroom teachers in the use of these materials (see fig. 3) resulted in long lead times before fully reaching the targeted population of students. (Table 2 shows information about the programs in the five countries.) As a result, portions of targeted student populations in four countries are not expected to benefit from the programs in time to contribute to USAID’s strategic goal of 100 million students with improved reading skills by December 2015. The programs’ monitoring efforts produced data showing very low baseline reading skills among targeted students. In addition, the programs in the five countries all included plans for external performance or impact evaluations.

Missions in All Five Countries Have Reading Programs with Recommended Elements, although Some Programs Will Not Reach All Targeted Students by 2015

Our fieldwork at the USAID missions in Ethiopia, Malawi, the Philippines, Peru, and Uganda showed that each mission had implemented primary grade reading programs with elements recommended in USAID’s technical notes to the education strategy (see text box for key recommendations from the technical notes).countries, the programs included teacher training and coaching in reading instruction; outreach to encourage community and parental support for reading during and after school (see fig. 4); and emphasis on five key components of reading—phonemic awareness, phonics, fluency, vocabulary, and comprehension. Moreover, in four of the countries— Ethiopia, the Philippines, Malawi, and Uganda—the development of new reading instruction materials in mother-tongue languages as well as training and coaching teachers in the use of these materials were central to the reading programs, according to mission officials. The programs in these countries also included improving reading instruction in English.

U.S. Agency for International Development’s Key Recommended Program Elements for Primary Grade Reading Programs

Instruction in a language that students speak and understand, particularly in the early years of school.

Instruction focused on five key component skills of reading alphabetic languages—phonemic awareness, phonics, fluency, vocabulary, and comprehension.

Teacher training and coaching in reading instruction.

Development of instructional material, including textbooks, decodable readers, teaching manuals, and lesson plans.

Regular classroom-based, teacher-led assessment of students’ progress.

Periodic formal assessment to measure impact over time.

Increasing community and parental support for reading, such as support for parent-teacher associations, teacher’s aides, and after-school tutoring.

Improving host-country education policy, such as increasing instruction time and adopting mother-tongue curriculum.

Capacity building, such as helping host countries conduct early-grade reading assessments.

Performance monitoring efforts in the five countries where we conducted fieldwork reveal very low reading skills among targeted student populations. Consistent with USAID’s guidance for performance monitoring, all five programs included in their performance monitoring plans the following standard outcome indicator: “Proportion of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text.” To develop this outcome indicator, program implementers in each country, after reaching consensus with host governments, set benchmarks for reading grade-level text with fluency and comprehension. Program implementers in each country also conducted standardized reading assessments to establish baselines for the proportion of students meeting the benchmarks. Because they are tied to unique circumstances and linguistic differences, the benchmarks may vary by country or language. For example, Ethiopia’s second-grade benchmark for oral reading fluency ranges from 55 words read correctly per minute for Tigrignia to 40 for Hadiyyisa. In Malawi and Uganda, the benchmark for reading fluently is 40 words read correctly per minute. With the exception of Peru, where the host government has well-established benchmarks for grade-level reading, the benchmarks used are not final and may be adjusted as reading assessment data become available for further analysis by program implementers and local stakeholders, such as ministries of education. A USAID official in one of the missions we visited noted that some countries may decide to set more aspirational targets than others to try to motivate regions and partners.

Uganda’s benchmark for reading with comprehension is 80 percent of comprehension questions (i.e., 4 out of 5) answered correctly. shows, the program in Ethiopia set similar targets for improving reading despite students’ very low baseline skills.

All Five Missions Have Planned Performance or Impact Evaluations of Primary Grade Reading Programs

Reflecting the emphasis on program evaluation in the education strategy and other guidance, all five missions where we conducted fieldwork planned for performance or impact evaluations of their reading programs, to be performed under separate contracts with external teams. The missions planned to conduct impact evaluations, using experimental designs that include both treatment and control groups, in accordance with USAID’s evaluation policy, and integrated the evaluation plans in their program designs. According to USAID officials, host countries are sometimes reluctant to accept program designs that intentionally withhold program interventions to some groups of recipients. For example, in Malawi, according to a senior government official, the government strongly preferred that all schools in the 11 participating school districts receive the program interventions but, to allow for USAID’s planned impact evaluation, ultimately agreed to some schools being part of the control group and not receiving the program’s intervention. USAID officials stated that in Ethiopia, where USAID’s program targets all schools, the program will receive a performance evaluation of certain intervention components, such as mother-tongue curriculum and teacher training, and an impact evaluation of other components, such as community outreach. Final evaluations of the reading programs in these countries are planned for 2017 or later.

As of April 2015, USAID Was Unable to Estimate Progress in Improving Reading Skills for 100 Million Children

As of April 2015, USAID lacked the ability to estimate overall progress toward its primary grade reading goal because of four factors. First, because it took some missions longer to implement primary grade reading programs than USAID estimated in its implementation guidance to the education strategy, only about two-thirds of missions are expected to have data that could be used to estimate progress toward USAID’s reading goal by the end of 2015. Second, only a small number of the missions that have collected reading assessment data have provided complete data and supporting information needed to aggregate and analyze missions’ reading assessment results. Agency officials attributed the incomplete data and information to a lack of timely guidance, among other factors. In 2014, USAID updated its guidance to clarify the agency’s reporting requirements. Third, USAID has not selected and documented a methodology for estimating total numbers of children with improved reading skills, owing in part to the lack of complete reading assessment data and supporting information. Documenting methodologies used to derive an estimate and reporting the information used to evaluate its reliability, such as information on sampling designs, sample sizes, and modes of data collection, help ensure that stakeholders understand the appropriate uses of the estimate and interpret the results according to generally accepted survey guidelines. Fourth, the education strategy did not set interim targets for measuring progress toward USAID’s reading goal in comparison to planned performance, as suggested by leading performance management practices. Without a methodology and interim targets, USAID officials cannot determine USAID’s progress toward its goal or identify a realistic goal for a future education strategy.

About Two-Thirds of Missions with Primary Grade Reading Programs Are Expected to Have Data for Assessing Overall Progress toward USAID’s Reading Goal by 2015

Because it took some missions longer to implement primary grade reading programs than USAID estimated in its implementation guidance to the education strategy, USAID expects only about two-thirds of missions with funding for such programs to provide baseline and midline or endline assessment data in time for the data to be used in estimating As figure 8 the agency’s progress toward its primary grade reading goal.shows, 26 missions (62 percent) of the 42 with funding for such programs had implemented them by the end of fiscal year 2012, the target date established in the education strategy’s implementation guidance to allow at least 3 years of implementation. Thirty-six of the 42 missions (86 percent) had implemented primary grade reading programs by the end of fiscal year 2014.

USAID expects that most of the missions that implemented primary grade reading programs by 2012, as well as several that implemented their programs in fiscal years 2013 or 2014, will complete baseline and midline or endline reading assessments in time to contribute to meeting the agency’s goal by the end of 2015.

Of the 26 missions that implemented their programs by the end of fiscal year 2012, all but 3 missions are expected to have completed a baseline assessment and at least a midline or an endline assessment by calendar year 2015.

Of the 36 missions that implemented primary grade reading programs by the end of fiscal year 2014, 25 missions completed a baseline reading assessment by the end of calendar year 2013, generally consistent with the agency’s fiscal year 2013 target date. USAID expects that 28 of the 36 missions will have completed a baseline assessment as well as a midline or an endline assessment by calendar year 2015.

Figure 9 shows the calendar years when USAID missions with funding for primary grade reading programs completed, or are expected to complete, baseline, midline, and endline reading assessments. (See app. III for information about the quality of the assessment data collected at the five missions we selected for fieldwork.)

USAID officials said that several factors contributed to the length of time required for missions to implement primary grade reading programs and collect assessment data in accordance with the education strategy. Designing projects in collaboration with host governments and then awarding contracts or cooperative agreements with implementing partners can take a year or more. USAID officials cited the following factors that contributed to the time needed to begin reading programs aligned with the education strategy:

Host governments may have had different education priorities than those articulated in USAID’s education strategy and may not fully support a focus on primary grade reading. In such cases, there is a need for discussion with host government officials to balance the education priorities of USAID with those of the host government. For example, the Jordanian government’s education reform efforts were focused on early childhood education and secondary education rather than primary grade reading. In December 2012, the mission in Jordan received USAID’s approval for exemption from full compliance with the education strategy; however, the mission stated that it would add an emphasis on primary grade reading to its education program.

Some missions had other, competing development objectives. For example, the mission in Morocco delayed implementing a primary grade reading program because the mission’s education assistance was tied to the competing objective of addressing at-risk youths. However, in 2013 the mission began planning for implementation of a primary grade reading program to support school retention and facilitate reenrollment for primary grade dropouts.

Designing new primary grade reading programs in countries where USAID did not have an existing program required additional time. For example, the mission in Nepal had not implemented a primary grade reading program before the education strategy was issued in 2011 and, as a result, needed time to assess the host government’s capacity to operate such a program.

USAID Lacks Complete Data and Information Needed to Aggregate and Analyze Reading Assessment Results

Although the USAID contractor responsible for aggregating and analyzing missions’ reading assessment results has begun to collect available data, as of April 2015, only a small number of the missions with completed reading assessments had provided both complete data and the supporting information that the contractor needs. In April 2015, the contractor reported that it had received complete baseline assessment data from 27 missions and midline or endline assessment data from 14 missions. However, the contractor reported that it had received complete supporting information about the programs and assessments from only 4 missions. Examples of the supporting information that the contractor needs to finish aggregating and analyzing the data include a description of the reading interventions, the implementation dates for the interventions, the numbers of children exposed to the interventions, the dates of the assessments and the assessment instrument, and a description of the sampling design for the assessments and population of children assessed. Using this information, the contractor checks the quality of the data; analyzes the effects of the reading interventions on children’s reading skills; compares sampled children’s reading skills at baseline (before the reading interventions begin) and at endline (at or near the end of the interventions); and extrapolates the assessment results for the samples of children to estimate improvements in reading skills for all children exposed to the reading interventions, including children who were not assessed.

Senior USAID officials noted that prior to the 2011-2015 education strategy, USAID’s Office of Education had no precedent for such a data collection effort, which required USAID to develop a contractual and legal framework and guidance to facilitate the transfer of reading assessment data and supporting information from missions’ implementing partners. and implementing partners to request and verify the data and to correct errors or account for missing information.

Building on its 2012 implementation guidance, in August 2014, USAID issued updated reporting guidance to help ensure that missions have a clear understanding of the requirements and processes for reporting assessment results. The updated guidance clarifies USAID’s expectations for the control and use of the assessment data and supporting information. The guidance also specifies the time frames and procedures for the transfer of assessment data and supporting information to USAID. In addition, the guidance specifies the types of data and supporting information required as well as the formats for the transfer of the data and information. Contractor staff said that the updated guidance has made the transfer of the required assessment data and supporting information more straightforward and consistent among missions and implementing partners.

USAID Has Not Selected a Methodology for Estimating Total Numbers of Children with Improved Reading, Limiting Its Ability to Measure Progress toward Its Reading Goal

As of April 2015, USAID had not yet selected and documented a methodology for using missions’ assessment data and supporting information to estimate progress toward its goal of improved reading skills for 100 million children by 2015. According to generally accepted survey guidelines, documenting methodologies used to derive an estimate and reporting the information used to evaluate its reliability—for example, sampling designs, sample sizes, and modes of data collection—is important to help ensure that stakeholders are able to understand the appropriate uses of the estimate and interpret the results accurately.

U.S. Agency for International Development, 2011 USAID Education Strategy Technical Notes. capacity to test the proposed methodology. For example, according to the contractor, the fact that few missions had provided data from both baseline and midline or endline assessments made it difficult to test for the minimum levels of reading improvement that children must meet to be counted as improved readers in different languages. In addition, the lack of information from multiple countries about the populations of children exposed directly or indirectly to the programs’ reading interventions limited the contractor’s ability to develop possible methods for extrapolating assessment results for samples of children to all children exposed to the interventions. According to contractor staff, as missions and implementing partners transfer data from additional assessments, with complete supporting information, the contractor will be better able to test and implement a methodology for measuring reading improvements in samples of assessed children and to extrapolate the results.

Although USAID proposed a methodology in 2012, it recognized that the methodology had limitations that needed to be resolved prior to its application. After unsuccessfully attempting to resolve these limitations, USAID requested that a second contractor examine the methodology and explore alternative methodologies in July 2014. USAID also asked the second contractor to conduct further technical work to implement a methodology, using selected reading programs for which baseline and midline or endline assessment data were available.

The second contractor reported key challenges to developing a methodology. These challenges included the difficulty of selecting a methodology that is applicable to different programs and of using a cross- sectional approach, in which different samples of children are assessed at baseline and at endline, as recommended by USAID’s implementation guidance.

According to the second contractor, before USAID can produce an estimate of improved readers and monitor progress toward its goal, the agency must define the minimum increase in reading levels that children must demonstrate to be counted as improved readers, select a methodology for calculating the percentages of assessed children demonstrating improved reading skills, and select procedures for extrapolating the results from the assessed sample to the targeted population.

In November 2014, after completing its technical work, the second contractor reported to USAID regarding two possible methodologies for calculating the percentages of assessed children demonstrating improved reading skills: the threshold methodology, which USAID proposed in its technical notes to the education strategy, and the gain score methodology, which the contractor proposed as an alternative. According to the second contractor, once USAID selects a methodology, the contractor can calculate the percentages of assessed children who demonstrate improved reading skills. The contractor will then extrapolate these percentages to the populations of children exposed to the programs to estimate total numbers of children with improved reading skills.

Threshold methodology. The threshold methodology calculates the percentages of improved readers on the basis of changes in the proportion of children who improve from below a threshold level of reading in a baseline assessment to a level at or above that threshold in an endline assessment. For example, different threshold levels of reading are created using the average numbers of words read correctly per minute and numbers of reading comprehension questions answered correctly for the sample of assessed children at baseline. The percentage of improved readers is then calculated on the basis of the differences between the proportions of children at each of these threshold levels at baseline and at endline.

Gain score methodology. The gain score methodology ranks assessment results for a sample of children assessed at baseline and a sample assessed at endline and matches results for individuals in each sample to calculate percentages with improved reading skills. For example, assessment results for the child reading the 10th fewest words correctly per minute at baseline are matched with results for the child reading the 10th fewest words correctly at endline. The percentages of children with improved reading skills are calculated on the basis of the numbers of matched pairs whose reading skills increased by a given minimum—for instance, who read an additional 1, 5, or 10 words correctly per minute.

The USAID contractor staff used assessment data from eight USAID programs to compare percentages of improved readers that they calculated using the two methodologies and various minimum reading levels. According to the contractor’s analysis, using the two methodologies to analyze the same assessment results produced different estimates of the percentages of children demonstrating improved reading skills. Specifically, the contractor found that the gain score methodology consistently produced higher estimates of improved readers than did the threshold methodology. According to the contractor, the discrepancy in the estimates from the two methodologies occurs because the threshold methodology only counts children whose reading improves across a threshold level. The threshold methodology does not count children who demonstrate some level of improved reading, but whose reading skills remain below or above a threshold level.

The USAID contractor has also reviewed possible procedures for extrapolating the percentages of samples of assessed children demonstrating reading improvements to all children exposed to the reading interventions. However, the contractor noted that it needs to conduct further technical work before making recommendations regarding these procedures to USAID.

USAID Lacks Information Needed to Assess the Current Education Strategy and Plan a Post- 2015 Strategy

Lack of information about progress relative to USAID’s primary grade reading goal limits the agency’s ability to assess its current approach to achieving its reading goal and establish a realistic goal for a post-2015 education strategy. Because USAID has not selected a methodology for estimating total numbers of children with improved reading skills resulting from exposure to primary grade reading interventions, it has not produced interim estimates of its progress toward the goal of 100 million children with improved reading skills by December 2015. In addition, USAID’s education strategy and guidance did not establish any interim targets for measuring progress toward USAID’s primary grade reading goal. As a result, USAID officials and other stakeholders currently lack aggregate information needed to monitor and report on the numbers of children whose reading skills have improved as a result of the USAID-supported programs. Establishing interim performance targets and using reliable estimates to assess progress toward an agency goal allow managers and stakeholders to address any performance issues that may affect the outcome of the agency’s goal, according to leading practices for performance management. Frequently reporting progress in achieving interim performance targets allows managers to review the information in time to make improvements. Without interim targets and reliable estimates, USAID cannot assess its actual progress against its planned performance and use the information to make any adjustments needed to better achieve its goal.

Furthermore, as the current education strategy’s December 2015 end date approaches, the lack of aggregate information about USAID’s progress in achieving its primary grade reading goal may constrain its efforts to develop a new education strategy. According to agency officials, USAID is presently engaged in developing its post-2015 strategy. We have previously reported that agencies can use performance information to make decisions that affect future strategies. helps program managers decide among competing priorities and reassess their performance goals and strategies. The lack of information about the outcome of USAID’s current primary grade reading goal could limit USAID officials’ capacity to assess USAID’s current strategy and goal and make effective decisions about a realistic goal for USAID’s education strategy after 2015.

Conclusions

See GAO-05-927. improved reading skills by December 2015 and to provide data for program monitoring and evaluation. In the five countries where we conducted fieldwork, the USAID missions have implemented primary grade reading programs with elements recommended in USAID guidance and have conducted standardized reading assessments. However, the very low reading skills revealed by data from these assessments suggest that improving skills to the point where students can read with fluency and comprehension will be a long-term effort in these and other developing countries.

USAID’s use of standardized reading assessments represents an important step in its efforts to measure the outcomes of its basic education assistance. However, in part, because of the time required for missions to design and implement programs and assess children’s reading skills, USAID’s contractor has limited access to reading assessment data. USAID officials acknowledged that the education strategy’s time frames were unrealistic. Moreover, USAID has not yet selected and documented a methodology—including reporting information on the sampling designs, sample sizes, and modes of data collection—for estimating total numbers of children whose reading skills have improved. Until it selects and documents a methodology for calculating the percentages of assessed children demonstrating improved reading skills, and extrapolating the results to estimate the total numbers of children with improved reading skills, USAID cannot monitor and reliably report estimates of the results of its efforts to improve reading skills of 100 million children.

Furthermore, unless it selects and documents a methodology and sets targets for measuring progress toward a reading goal in any future education strategy, USAID will be limited in its ability to assess progress toward that goal in comparison with planned performance. Accurate reporting of USAID’s progress toward its strategic goal would inform stakeholders about the impact of the agency’s efforts to improve children’s reading skills in the final year of the current education strategy and could help inform USAID officials’ efforts to develop a realistic goal for a future education strategy.

Recommendations for Executive Action

We recommend that the Acting USAID Administrator take the following three actions.

To help ensure that USAID reports a reliable estimate of the results of its efforts to achieve the goal of improving reading skills of 100 million children, we recommend that the Acting Administrator select a methodology for estimating the total numbers of children with improved reading skills as a result of exposure to primary grade reading programs and document a description of the selected methodology as well as the information necessary to evaluate the estimate (i.e., sampling designs, sample sizes, and modes of data collection) when reporting progress toward the reading goal.

To improve USAID’s ability to measure progress in achieving a quantitative reading goal in any future education strategy, we recommend that the Acting Administrator ensure that the future strategy includes targets that will allow USAID to monitor interim progress toward its goal in comparison with planned performance.

Agency Comments and Our Evaluation

We provided a draft of this report to USAID for comment. In its written comments, reproduced in appendix IV, USAID agreed with our findings and recommendations. USAID also provided technical comments, which we incorporated throughout the report, as appropriate.

We are sending copies of this report to interested congressional committees and the Acting USAID Administrator. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

We were asked to review several aspects of the U.S. Agency for International Development’s (USAID) basic education programming. We focused our review on basic education programming related to Goal 1 of USAID’s education strategy. This report (1) examines five USAID missions’ implementation, performance monitoring, and evaluation of primary grade reading programs and (2) assesses USAID’s efforts to estimate progress toward the 2011-2015 education strategy’s primary grade reading goal.

For both of these objectives, we analyzed funding for basic education for fiscal years 2010 through 2014 and reviewed USAID’s 2011-2015 education strategy, as well as guidance for implementing the education strategy and USAID’s technical notes to the education strategy. We also interviewed USAID officials in the Bureau for Economic Growth, Education and Environment; the Bureau for Policy, Planning, and Learning; and four of the agency’s geographic bureaus. In addition, we conducted fieldwork in five countries—Ethiopia, Malawi, Peru, the Philippines, and Uganda—which we selected because they are geographically diverse and received relatively large levels of U.S. funding for basic education in fiscal year 2013. We selected the three African countries because this region received more than 50 percent of USAID’s allocation for basic education assistance in fiscal year 2013 and was identified in the education strategy as a priority region for investment of resources. In addition, we selected the five countries because the USAID missions in these countries had primary grade reading programs under way and had conducted at least baseline reading assessments. During our fieldwork in these countries, we met with USAID mission officials; USAID’s implementing partners; host government ministry of education officials; local school district officials; school principals; teachers; and members of parent groups, such as parent-teacher associations.

To examine the five missions’ implementation, monitoring, and evaluation of primary grade reading programs—our first objective—we reviewed key planning and reporting documents associated with USAID’s primary grade reading program in each of the five countries. These documents included, among others, contract awards or cooperative agreements, monitoring and evaluation plans, performance management plans, and implementing partners’ annual work plans and quarterly and annual reports.

To determine whether the reading programs in the five countries included key program elements that USAID’s education strategy recommends as effective in improving reading outcomes, we compared elements in these programs, as identified in their planning documents, with those recommended in USAID’s technical notes to the education strategy. We also visited schools, where we observed classroom reading lessons that used instructional materials developed through USAID’s programs and met with local teachers, education officials, and community organizations to discuss program interventions. In Ethiopia, we observed local teams of teachers, story writers, editors, and typists in the process of developing reading curriculums in seven mother-tongue languages for grades 5 through 8. To identify the programs’ timelines for delivering improved reading instruction to targeted students, we reviewed the programs’ performance management plans and annual work plans and met with USAID officials at the country missions as well as implementing partner officials.

To examine the five USAID missions’ performance monitoring of the current primary grade reading programs, we reviewed outcome and output measures identified in the missions’ and reading programs’ monitoring and evaluation plans and relevant standardized reading assessment reports. We identified each country’s benchmarks for reading at grade level—number of correct words read per minute and percentage of comprehension questions answered correctly—and summarized each program’s baseline data showing the percentage of students meeting these benchmarks. To examine each mission’s plans for conducting performance or impact evaluations of current primary grade reading and interviewed programs, we reviewed USAID’s evaluation policyUSAID officials at missions and implementing partners responsible for conducting external evaluations of these programs. We also reviewed the missions’ related requests for proposals and award agreements, as well as work plans and interim or quarterly reports submitted to the missions by these implementing partners.

To assess USAID’s efforts to estimate progress toward the education strategy’s primary grade reading goal—our second objective—we reviewed the USAID education strategy, guidance for implementing the strategy, technical notes to the strategy, and the strategy’s reporting guidance to identify the agency’s policies and guidelines for completing reading assessments at USAID missions and for analyzing the data and supporting information. We analyzed USAID reports on missions’ collection of reading assessment data. We also analyzed reports prepared by the USAID contractor responsible for aggregating and analyzing the reading assessment data, to determine the missions’ progress in transferring reading assessment data and supporting information to the contractor. In addition, we interviewed USAID officials and USAID contractor staff in Washington, D.C., regarding the guidance and procedures for collecting and analyzing the reading assessment data and supporting information as well as missions’ progress in transferring the data to USAID for aggregation and analysis.

To examine USAID’s potential methodologies for estimating total numbers of children with improved reading skills, we reviewed USAID contractors’ documentation of proposed methodologies. We also reviewed the USAID contractors’ reports on the development of potential methodologies.contractor staff in Washington, D.C., about the development and implementation of potential methodologies to measure improved reading skills, and the challenges and limitations to collecting and analyzing missions’ reading assessment data. We used GAO standards and In addition, we interviewed USAID officials and USAID generally accepted survey guidelines to assess USAID’s efforts to estimate progress toward its primary grade reading goal.

To assess the reliability of the reading assessment data, we compared the design, data collection, and analysis of results for the assessments at the five missions against generally accepted survey guidelines as outlined in Office of Management and Budget (OMB) guidance and USAID guidance.accepted professional survey practices, including the collection of interview-based survey data. We also found that the majority of OMB guidelines were also reflected in USAID’s own implementation guidelines. GAO methodologists compared the information collected from agency documents and during interviews with agency officials and implementing partners and generally found that the reading assessments met these guidelines. We also determined that the data we obtained for both objectives were sufficiently reliable for the purposes of our review.

Many OMB guidelines represent long-established, generally We conducted this performance audit from May 2014 to May 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Basic Education Funding in Fiscal Year 2013 by Country and Region

In fiscal year 2013—the first year in which missions’ education programs were expected to be fully aligned with the goals of the education strategy—the U.S. Agency for International Development (USAID) allocated funding to 41 countries for programs to improve the reading skills of 100 million children in primary grades by December 2015—Goal 1 of its education strategy. Table 4 shows USAID funding in support of Goals 1 and 3 of the strategy.

As figure 10 shows, more than half of USAID’s funding for Goal 1 of the strategy was directed to countries in Sub-Saharan Africa, where baseline reading scores were among the lowest globally, according to USAID.

Afghanistan and Pakistan received the next largest share, a combined total of 18 percent of USAID’s 2013 basic education allocations.

Appendix III: Reading Assessment Data Collected At Five Selected Missions

We found that the design and implementation of the reading assessments in the five countries where we conducted fieldwork—Ethiopia, Malawi, Peru, the Philippines, and Uganda—met generally accepted survey guidelines and U.S. Agency for International Development (USAID) guidance on implementing Early Grade Reading Assessments (EGRA). In four of the five countries we visited, a U.S.-based contractor assisted in the design and data collection using the EGRA. Contractor staff in headquarters provided input into both instrument and sample design and conducted in-country training for assessors and supervisors. For example, in each country, assessor and supervisor reliability scores were calculated during training and those assessors and supervisors with low scores were either retrained or not allowed to collect data. Also, sampling designs appropriately considered analytical goals, such as expected precision of subpopulation estimates, and the geographical distribution of schools and the population of students. In some countries, the sampling design also included information about languages. We also generally found that the contractor appropriately calculated country-level estimates based on the complex sample design after assessment data were collected.

In three of four countries that used the EGRA, data were collected electronically on tablets and regularly transmitted to contractor headquarters for review (see fig. 11). Data were reviewed on a timely basis so that any potential problems could be identified and corrected in the field. For example, if an assessor was consistently taking too little time administering a reading assessment, that assessor could be identified and retrained if necessary. One country using an EGRA, Ethiopia, collected data using paper questionnaires—a method with the potential for more errors than electronic data collection. However, certain data quality procedures were in place, such as requiring that a consultant verify all of the data entered.

One country where we conducted fieldwork, Peru, did not use an EGRA to collect reading assessment data but instead relied on reading comprehension scores from a national assessment administered by the Ministry of Education to all second graders. Ministry officials told us that certain safeguards are in place to protect the integrity of the data collected. For example, assessors attend training and must pass a test, and ministry officials supervise testing sites where the validity of previous results has been questioned. Ministry officials also told us that the current methodology had been in place since 2007 and asserted that year-to-year comparisons of test scores since 2007 were valid. Assessment scores are provided to each region and school, including those receiving basic education assistance, to track children’s reading comprehension progress.

Appendix IV: Comments from the U.S. Agency for International Development

Appendix V: GAO Contacts and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the individual named above, Michael Rohrback (Assistant Director), Howard Cott, Justin Fisher, Bradley Hunt, Jill Lacey, Reid Lowe, Grace Lui, and Michelle Serfass made significant contributions to this report.