International Education Assistance: USAID Has Implemented Primary Grade Reading Programs but Has Not Yet Measured Progress toward Its Strategic Goal
Published: 2015-05-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: While many developing countries have achieved important gains in primary school enrollment, students' reading skills remain very low. In response, USAID's 2011-2015 Education Strategy set a goal of improved reading skills for 100 million primary grade children by the end of 2015 (Goal 1). The 42 missions with funding for primary grade reading programs were to start programs aligned with the strategy by the end of fiscal year 2012 to allow time to assess results by 2015.
GAO was asked to review USAID's efforts to implement primary grade reading programs—Goal 1 of its education strategy. This report (1) examines five USAID missions' implementation, monitoring, and evaluation of reading programs and (2) assesses USAID's efforts to estimate progress toward its reading goal. GAO analyzed USAID's education strategy and other guidance, reading assessment data and methodologies, performance monitoring plans and reports, and program evaluations, and conducted fieldwork in five countries selected based on geographic diversity, funding for basic education, and availability of reading assessment data.
section_title: What GAO Found
paragraphs: U.S. Agency for International Development (USAID) missions in all five countries where GAO conducted fieldwork were implementing primary grade reading interventions recommended in USAID's education strategy guidance. A key intervention in four of these countries is developing new reading instruction materials in mother-tongue languages—in some instances, multiple languages. In Ethiopia, for example, USAID's reading program developed new student textbooks and teacher's guides for each of eight grades and seven languages and involved teams of local teachers, language experts, and story writers, among others. The complexity of these interventions in Ethiopia, Malawi, the Philippines, and Uganda increased the time before they could be introduced in classrooms. As a result, some targeted student populations are not expected to benefit from improved reading instruction until late 2015 or 2016, too late to be measured to contribute to USAID's strategic goal of 100 million students with improved reading skills by the end of 2015. In addition, the missions were monitoring their reading programs and planned to conduct program evaluations consistent with USAID guidance.
USAID is currently unable to estimate progress toward the education strategy's reading goal because of four factors. First, because it took some missions longer to implement reading programs than USAID estimated, only about two-thirds of missions are expected to have data to estimate progress toward the goal by the end of 2015. Second, only a small number of missions provided USAID the complete data and supporting information that it needs to aggregate and analyze reading assessment results. Agency officials attributed the lack of complete data and information to an absence of timely guidance. In 2014, USAID issued updated reporting guidance. Third, USAID has not selected a methodology for calculating percentages of assessed children demonstrating improved reading skills, and extrapolating the results to estimate the total numbers of children with improved reading skills. Along with incomplete data and information, other factors contributed to a delay in identifying a methodology—for example, USAID did not ask its contractor to examine the methodology proposed in 2012 and explore alternatives until 2014. Fourth, the education strategy did not set interim targets for assessing progress toward the reading goal as suggested by leading performance management practices. Without a methodology and interim targets, USAID officials and others lack aggregate information about progress toward the goal to assess the current strategy and plan a realistic goal for a future strategy.
section_title: What GAO Recommends
paragraphs: USAID should select a methodology for estimating total numbers of children with improved reading skills, document a description of the methodology when reporting results, and set interim targets to assess progress toward a reading goal in any future education strategy. USAID concurred with GAO's recommendations.

section_title: Letter
paragraphs: Learning to read is the foundation for future learning, and greater  educational attainment contributes to economic growth, improved health  outcomes, and democratic governance. However, the U.S. Agency for  International Development (USAID) reports that while many developing  countries have achieved important gains in enrolling children in primary  school, reading skills remain very low. In some countries, 70 to 90  percent of students tested at the end of 2 to 3 years of schooling were  unable read a single word correctly in the first line of a simple passage.
U.S. Agency for International Development, Education: Opportunity through Learning.  USAID Education Strategy 2011-2015 (Washington, D.C.: February 2011). students’ reading skills. In addition, the strategy included guidelines for  monitoring and evaluating USAID’s progress in achieving its basic  education goals. As the current strategy’s end date approaches, USAID is  deliberating its post-2015 strategy for basic education assistance.
You asked us to review USAID’s basic education programming. This  report (1) examines five USAID missions’ implementation, performance  monitoring, and evaluation of primary grade reading programs and (2)  assesses USAID’s efforts to estimate progress toward the 2011-2015  education strategy’s primary grade reading goal.
In preparing our report, we analyzed USAID’s 2011-2015 education  strategy, related guidance, and funding for basic education for fiscal years  2010 through 2014. We also conducted fieldwork in five countries— Ethiopia, Malawi, Peru, the Philippines, and Uganda. We selected these  countries because they are geographically diverse and had received  relatively large levels of U.S. funding for basic education. We also  selected these countries because the USAID missions in these countries  had implemented primary grade reading programs and had conducted at  least baseline reading assessments. To examine the five missions’  implementation, performance monitoring, and evaluation of primary grade  reading programs—our first objective—we reviewed USAID guidance as  well as documentation from the missions and interviewed USAID officials.  To assess USAID’s efforts to estimate progress toward its primary grade  reading goal—our second objective—we reviewed the USAID education  strategy and USAID policies and guidelines for completing reading  assessments and analyzed USAID reports on missions’ collection of  reading assessment data. We also reviewed USAID contractors’  documentation of proposed methodologies for estimating total numbers of  children with improved reading skills, and we used GAO standards and  generally accepted survey guidelines to assess these efforts to estimate   To assess the reliability  progress toward its primary grade reading goal.of reading assessment data, we compared the design, data collection,  and analysis of results for the assessments at the five missions against  generally accepted survey guidelines and USAID guidance. In addition,  we interviewed USAID officials and contractor staff in Washington, D.C.,  and USAID mission officials, USAID implementing partner officials, host  government officials, and others in each of the five selected countries. We  determined that the data we obtained were sufficiently reliable for the  purposes of our review.
We conducted this performance audit from May 2014 to May 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
subsections: section_title: USAID’s 2011-2015 Education Strategy
paragraphs: USAID’s 2011-2015 education strategy, issued in February 2011,  includes three goals: improved reading skills for 100 million children in  primary grades by 2015 (Goal 1), improved ability of tertiary and  workforce development programs to produce a workforce with relevant  skills to support country development goals by 2015 (Goal 2), and  increased equitable access to education in crisis and conflict  environments for 15 million learners by 2015 (Goal 3). Goal 1 represents  a shift from an earlier broader focus on achieving universal enrollment of children in primary school as well as improving the quality of education.
In August 2011, USAID issued guidance for implementing the education  strategy, directing missions with basic education programs in non-crisis  and conflict environments to narrow the focus of their interventions to  achieving the goal of improving early grade reading by the end of fiscal  year 2012. According to the guidance, missions that were not in conflict  or crisis-affected environments were to focus resources on primary grade  reading and discontinue programming in such areas as early childhood  education, secondary education, or adult literacy. In addition, missions’  programming was to be based on evidence of what works to improve  reading outcomes. The guidance stated that the timing of missions’  alignment of education programs with the education strategy was critical  because programs required at least 3 years of implementation to  contribute to the strategy’s numerical target.
section_title: USAID’s Performance and Evaluation Requirements
paragraphs: USAID requires that missions’ primary grade reading programs adhere to  the agency’s performance monitoring and evaluation framework  requirements in order to track progress annually against the full set of  expected results. Performance monitoring periodically tracks changes to  determine whether desired results are occurring and whether projects are  implemented as designed. USAID missions and implementing partners  monitor programs by using performance management plans or equivalent  documents that define what is to be measured, the unit of measurement,  and the level of data disaggregation, among other information. USAID  guidance for implementing the education strategy requires that these  plans include, at a minimum, either of two standard outcome indicators  related to primary grade reading, if relevant:
The proportion of students who demonstrate by the end of two grades  of primary schooling that they can read and understand the meaning  of grade-level text.
The proportion of students who by the end of the primary school years  are able to read and demonstrate understanding as defined by a  country curriculum or standards or as agreed to by national experts.
While performance monitoring is required for all programs, USAID’s  evaluation policy requires performance or impact evaluations of all large  projects. The policy states that performance evaluations should focus  on questions such as what a project or program has achieved, how it is  being implemented, and how it is perceived and valued. Impact  evaluations measure the change in a development outcome that is  attributable to the project or program, based on models of cause and  effect. In addition, USAID’s guidance for implementing the education  strategy instructs missions to integrate plans for performance or impact  evaluation into program design during the planning stages and to derive  data for evaluations from reading assessments. The guidance also states  that performance and impact evaluations should be provided by an  external, separately contracted team.
A comparison of baseline assessment data (i.e., data collected before a reading  program starts) and endline assessment data (i.e., data collected at or near the end of the  program or before the end of fiscal year 2015) can be used by missions to report on the  standard outcome indicators for performance monitoring and by the agency’s contractors  to estimate progress toward the goal of 100 million students with improved reading skills  by December 2015. USAID officials noted that the need to assess learners toward the end  of a school year, which often ends in May or June, may result in some missions assessing  learning outcomes later than December 2015. See U.S. Agency for International  Development, 2011 USAID Education Strategy Technical Notes (Washington, D.C.:  revised April 2012). least two phases of assessment are necessary to measure reading  improvement.the Philippines.
section_title: USAID Basic Education Funding and Management
paragraphs: In fiscal year 2013—the first year in which missions’ education programs  were expected to be fully aligned with the goals of the education  strategy—USAID allocated about $790 million for basic education  assistance divided between programs supporting the education strategy’s  Goal 1—improving primary grade reading—and programs supporting  Goal 3—increasing equitable access to education in crisis and conflict  environments. USAID provided basic education funding to 41 countries  for primary grade reading programs in fiscal year 2013. Table 1 shows  USAID’s allocations for basic education for fiscal years 2010 through  2014 by appropriation account. More than half of the 2013 funding was  allocated for basic education activities in countries in Sub-Saharan Africa,  where baseline reading scores were among the lowest globally, according  to USAID. Appendix II shows the fiscal year 2013 allocations for basic  education by country.
USAID’s basic education programs are planned and implemented  primarily by USAID’s overseas missions. In Washington, D.C., the Office  of Education in USAID’s Bureau for Economic Growth, Education and  Environment provides technical leadership, research, and field support.  The office also assists in the development, implementation, and  evaluation of the USAID education strategy and resource allocation  priorities. In addition, USAID’s geographic bureaus advise missions on  their education programs.
section_title: USAID Missions in Five Selected Countries Are Implementing, Monitoring, and Evaluating Reading Programs Consistent with USAID Guidance
paragraphs: In the five countries where we conducted fieldwork, the USAID missions  had implemented primary grade reading programs that included  intervention elements recommended in USAID’s guidance related to its  education strategy. In several of the countries, the need to develop new  reading materials in mother-tongue languages and train classroom  teachers in the use of these materials (see fig. 3) resulted in long lead  times before fully reaching the targeted population of students. (Table 2  shows information about the programs in the five countries.) As a result,  portions of targeted student populations in four countries are not expected  to benefit from the programs in time to contribute to USAID’s strategic  goal of 100 million students with improved reading skills by December  2015. The programs’ monitoring efforts produced data showing very low  baseline reading skills among targeted students. In addition, the  programs in the five countries all included plans for external performance  or impact evaluations.
subsections: section_title: Missions in All Five Countries Have Reading Programs with Recommended Elements, although Some Programs Will Not Reach All Targeted Students by 2015
paragraphs: Our fieldwork at the USAID missions in Ethiopia, Malawi, the Philippines,  Peru, and Uganda showed that each mission had implemented primary  grade reading programs with elements recommended in USAID’s  technical notes to the education strategy (see text box for key  recommendations from the technical notes).countries, the programs included teacher training and coaching in reading  instruction; outreach to encourage community and parental support for  reading during and after school (see fig. 4); and emphasis on five key  components of reading—phonemic awareness, phonics, fluency,  vocabulary, and comprehension. Moreover, in four of the countries— Ethiopia, the Philippines, Malawi, and Uganda—the development of new  reading instruction materials in mother-tongue languages as well as  training and coaching teachers in the use of these materials were central  to the reading programs, according to mission officials. The programs in  these countries also included improving reading instruction in English.
subsections: section_title: U.S. Agency for International Development’s Key Recommended Program Elements for Primary Grade Reading Programs
paragraphs: Instruction in a language that students speak and understand, particularly in the early years of school.
Instruction focused on five key component skills of reading alphabetic languages—phonemic awareness, phonics, fluency,  vocabulary, and comprehension.
Teacher training and coaching in reading instruction.
Development of instructional material, including textbooks, decodable readers, teaching manuals, and lesson plans.
Regular classroom-based, teacher-led assessment of students’ progress.
Periodic formal assessment to measure impact over time.
Increasing community and parental support for reading, such as support for parent-teacher associations, teacher’s aides, and  after-school tutoring.
Improving host-country education policy, such as increasing instruction time and adopting mother-tongue curriculum.
Capacity building, such as helping host countries conduct early-grade reading assessments.
Performance monitoring efforts in the five countries where we conducted  fieldwork reveal very low reading skills among targeted student  populations. Consistent with USAID’s guidance for performance  monitoring, all five programs included in their performance monitoring  plans the following standard outcome indicator: “Proportion of students  who, by the end of two grades of primary schooling, demonstrate that  they can read and understand the meaning of grade level text.” To  develop this outcome indicator, program implementers in each country,  after reaching consensus with host governments, set benchmarks for  reading grade-level text with fluency and comprehension. Program  implementers in each country also conducted standardized reading  assessments to establish baselines for the proportion of students meeting  the benchmarks. Because they are tied to unique circumstances and  linguistic differences, the benchmarks may vary by country or language.  For example, Ethiopia’s second-grade benchmark for oral reading fluency  ranges from 55 words read correctly per minute for Tigrignia to 40 for  Hadiyyisa. In Malawi and Uganda, the benchmark for reading fluently is  40 words read correctly per minute. With the exception of Peru, where  the host government has well-established benchmarks for grade-level  reading, the benchmarks used are not final and may be adjusted as  reading assessment data become available for further analysis by  program implementers and local stakeholders, such as ministries of  education. A USAID official in one of the missions we visited noted that  some countries may decide to set more aspirational targets than others to  try to motivate regions and partners.
Uganda’s benchmark for reading with comprehension is 80 percent of comprehension  questions (i.e., 4 out of 5) answered correctly. shows, the program in Ethiopia set similar targets for improving reading  despite students’ very low baseline skills.
section_title: All Five Missions Have Planned Performance or Impact Evaluations of Primary Grade Reading Programs
paragraphs: Reflecting the emphasis on program evaluation in the education strategy  and other guidance, all five missions where we conducted fieldwork  planned for performance or impact evaluations of their reading programs,  to be performed under separate contracts with external teams. The  missions planned to conduct impact evaluations, using experimental  designs that include both treatment and control groups, in accordance  with USAID’s evaluation policy, and integrated the evaluation plans in  their program designs. According to USAID officials, host countries are  sometimes reluctant to accept program designs that intentionally withhold  program interventions to some groups of recipients. For example, in  Malawi, according to a senior government official, the government  strongly preferred that all schools in the 11 participating school districts  receive the program interventions but, to allow for USAID’s planned  impact evaluation, ultimately agreed to some schools being part of the  control group and not receiving the program’s intervention. USAID  officials stated that in Ethiopia, where USAID’s program targets all  schools, the program will receive a performance evaluation of certain  intervention components, such as mother-tongue curriculum and teacher  training, and an impact evaluation of other components, such as  community outreach. Final evaluations of the reading programs in these  countries are planned for 2017 or later.
section_title: As of April 2015, USAID Was Unable to Estimate Progress in Improving Reading Skills for 100 Million Children
paragraphs: As of April 2015, USAID lacked the ability to estimate overall progress  toward its primary grade reading goal because of four factors. First,  because it took some missions longer to implement primary grade  reading programs than USAID estimated in its implementation guidance  to the education strategy, only about two-thirds of missions are expected  to have data that could be used to estimate progress toward USAID’s  reading goal by the end of 2015. Second, only a small number of the  missions that have collected reading assessment data have provided  complete data and supporting information needed to aggregate and  analyze missions’ reading assessment results. Agency officials attributed  the incomplete data and information to a lack of timely guidance, among  other factors. In 2014, USAID updated its guidance to clarify the agency’s  reporting requirements. Third, USAID has not selected and documented a  methodology for estimating total numbers of children with improved  reading skills, owing in part to the lack of complete reading assessment  data and supporting information. Documenting methodologies used to  derive an estimate and reporting the information used to evaluate its  reliability, such as information on sampling designs, sample sizes, and  modes of data collection, help ensure that stakeholders understand the  appropriate uses of the estimate and interpret the results according to  generally accepted survey guidelines. Fourth, the education strategy did  not set interim targets for measuring progress toward USAID’s reading  goal in comparison to planned performance, as suggested by leading  performance management practices. Without a methodology and  interim targets, USAID officials cannot determine USAID’s progress  toward its goal or identify a realistic goal for a future education strategy.
subsections: section_title: About Two-Thirds of Missions with Primary Grade Reading Programs Are Expected to Have Data for Assessing Overall Progress toward USAID’s Reading Goal by 2015
paragraphs: Because it took some missions longer to implement primary grade  reading programs than USAID estimated in its implementation guidance  to the education strategy, USAID expects only about two-thirds of  missions with funding for such programs to provide baseline and midline  or endline assessment data in time for the data to be used in estimating   As figure 8  the agency’s progress toward its primary grade reading goal.shows, 26 missions (62 percent) of the 42 with funding for such programs  had implemented them by the end of fiscal year 2012, the target date  established in the education strategy’s implementation guidance to allow  at least 3 years of implementation. Thirty-six of the 42 missions (86  percent) had implemented primary grade reading programs by the end of  fiscal year 2014.
USAID expects that most of the missions that implemented primary grade  reading programs by 2012, as well as several that implemented their  programs in fiscal years 2013 or 2014, will complete baseline and midline  or endline reading assessments in time to contribute to meeting the  agency’s goal by the end of 2015.
Of the 26 missions that implemented their programs by the end of  fiscal year 2012, all but 3 missions are expected to have completed a  baseline assessment and at least a midline or an endline assessment  by calendar year 2015.
Of the 36 missions that implemented primary grade reading programs  by the end of fiscal year 2014, 25 missions completed a baseline  reading assessment by the end of calendar year 2013, generally  consistent with the agency’s fiscal year 2013 target date. USAID  expects that 28 of the 36 missions will have completed a baseline  assessment as well as a midline or an endline assessment by  calendar year 2015.
Figure 9 shows the calendar years when USAID missions with funding for  primary grade reading programs completed, or are expected to complete,  baseline, midline, and endline reading assessments. (See app. III for  information about the quality of the assessment data collected at the five  missions we selected for fieldwork.)
USAID officials said that several factors contributed to the length of time  required for missions to implement primary grade reading programs and  collect assessment data in accordance with the education strategy.  Designing projects in collaboration with host governments and then  awarding contracts or cooperative agreements with implementing  partners can take a year or more. USAID officials cited the following  factors that contributed to the time needed to begin reading programs  aligned with the education strategy:
Host governments may have had different education priorities than  those articulated in USAID’s education strategy and may not fully  support a focus on primary grade reading. In such cases, there is a  need for discussion with host government officials to balance the  education priorities of USAID with those of the host government. For  example, the Jordanian government’s education reform efforts were  focused on early childhood education and secondary education rather  than primary grade reading. In December 2012, the mission in Jordan  received USAID’s approval for exemption from full compliance with  the education strategy; however, the mission stated that it would add  an emphasis on primary grade reading to its education program.
Some missions had other, competing development objectives. For  example, the mission in Morocco delayed implementing a primary  grade reading program because the mission’s education assistance  was tied to the competing objective of addressing at-risk youths.  However, in 2013 the mission began planning for implementation of a  primary grade reading program to support school retention and  facilitate reenrollment for primary grade dropouts.
Designing new primary grade reading programs in countries where  USAID did not have an existing program required additional time. For  example, the mission in Nepal had not implemented a primary grade  reading program before the education strategy was issued in 2011  and, as a result, needed time to assess the host government’s  capacity to operate such a program.
section_title: USAID Lacks Complete Data and Information Needed to Aggregate and Analyze Reading Assessment Results
paragraphs: Although the USAID contractor responsible for aggregating and analyzing  missions’ reading assessment results has begun to collect available data,  as of April 2015, only a small number of the missions with completed  reading assessments had provided both complete data and the  supporting information that the contractor needs. In April 2015, the  contractor reported that it had received complete baseline assessment  data from 27 missions and midline or endline assessment data from 14  missions. However, the contractor reported that it had received complete  supporting information about the programs and assessments from only 4  missions. Examples of the supporting information that the contractor  needs to finish aggregating and analyzing the data include a description  of the reading interventions, the implementation dates for the  interventions, the numbers of children exposed to the interventions, the  dates of the assessments and the assessment instrument, and a  description of the sampling design for the assessments and population of  children assessed. Using this information, the contractor checks the  quality of the data; analyzes the effects of the reading interventions on  children’s reading skills; compares sampled children’s reading skills at  baseline (before the reading interventions begin) and at endline (at or  near the end of the interventions); and extrapolates the assessment  results for the samples of children to estimate improvements in reading  skills for all children exposed to the reading interventions, including  children who were not assessed.
Senior USAID officials noted that prior to the 2011-2015 education strategy, USAID’s  Office of Education had no precedent for such a data collection effort, which required  USAID to develop a contractual and legal framework and guidance to facilitate the transfer  of reading assessment data and supporting information from missions’ implementing  partners. and implementing partners to request and verify the data and to correct  errors or account for missing information.
Building on its 2012 implementation guidance, in August 2014, USAID  issued updated reporting guidance to help ensure that missions have a  clear understanding of the requirements and processes for reporting  assessment results. The updated guidance clarifies USAID’s expectations  for the control and use of the assessment data and supporting  information. The guidance also specifies the time frames and procedures  for the transfer of assessment data and supporting information to USAID.  In addition, the guidance specifies the types of data and supporting  information required as well as the formats for the transfer of the data and  information. Contractor staff said that the updated guidance has made the  transfer of the required assessment data and supporting information more  straightforward and consistent among missions and implementing  partners.
section_title: USAID Has Not Selected a Methodology for Estimating Total Numbers of Children with Improved Reading, Limiting Its Ability to Measure Progress toward Its Reading Goal
paragraphs: As of April 2015, USAID had not yet selected and documented a  methodology for using missions’ assessment data and supporting  information to estimate progress toward its goal of improved reading skills  for 100 million children by 2015. According to generally accepted survey  guidelines, documenting methodologies used to derive an estimate and  reporting the information used to evaluate its reliability—for example,  sampling designs, sample sizes, and modes of data collection—is  important to help ensure that stakeholders are able to understand the  appropriate uses of the estimate and interpret the results accurately.
U.S. Agency for International Development, 2011 USAID Education Strategy Technical  Notes. capacity to test the proposed methodology. For example, according to the  contractor, the fact that few missions had provided data from both  baseline and midline or endline assessments made it difficult to test for  the minimum levels of reading improvement that children must meet to be  counted as improved readers in different languages. In addition, the lack  of information from multiple countries about the populations of children  exposed directly or indirectly to the programs’ reading interventions  limited the contractor’s ability to develop possible methods for  extrapolating assessment results for samples of children to all children  exposed to the interventions. According to contractor staff, as missions  and implementing partners transfer data from additional assessments,  with complete supporting information, the contractor will be better able to  test and implement a methodology for measuring reading improvements  in samples of assessed children and to extrapolate the results.
Although USAID proposed a methodology in 2012, it recognized that the  methodology had limitations that needed to be resolved prior to its  application. After unsuccessfully attempting to resolve these limitations,  USAID requested that a second contractor examine the methodology and  explore alternative methodologies in July 2014. USAID also asked the  second contractor to conduct further technical work to implement a  methodology, using selected reading programs for which baseline and  midline or endline assessment data were available.
The second contractor reported key challenges to developing a  methodology. These challenges included the difficulty of selecting a  methodology that is applicable to different programs and of using a cross- sectional approach, in which different samples of children are assessed at  baseline and at endline, as recommended by USAID’s implementation  guidance.
According to the second contractor, before USAID can produce an  estimate of improved readers and monitor progress toward its goal, the  agency must define the minimum increase in reading levels that children must  demonstrate to be counted as improved readers,  select a methodology for calculating the percentages of assessed  children demonstrating improved reading skills, and  select procedures for extrapolating the results from the assessed  sample to the targeted population.
In November 2014, after completing its technical work, the second  contractor reported to USAID regarding two possible methodologies for  calculating the percentages of assessed children demonstrating improved  reading skills: the threshold methodology, which USAID proposed in its  technical notes to the education strategy, and the gain score  methodology, which the contractor proposed as an alternative. According  to the second contractor, once USAID selects a methodology, the  contractor can calculate the percentages of assessed children who  demonstrate improved reading skills. The contractor will then extrapolate  these percentages to the populations of children exposed to the programs  to estimate total numbers of children with improved reading skills.
Threshold methodology. The threshold methodology calculates the  percentages of improved readers on the basis of changes in the  proportion of children who improve from below a threshold level of  reading in a baseline assessment to a level at or above that threshold  in an endline assessment. For example, different threshold levels of  reading are created using the average numbers of words read  correctly per minute and numbers of reading comprehension  questions answered correctly for the sample of assessed children at  baseline. The percentage of improved readers is then calculated on  the basis of the differences between the proportions of children at  each of these threshold levels at baseline and at endline.
Gain score methodology. The gain score methodology ranks  assessment results for a sample of children assessed at baseline and  a sample assessed at endline and matches results for individuals in  each sample to calculate percentages with improved reading skills.  For example, assessment results for the child reading the 10th fewest  words correctly per minute at baseline are matched with results for the  child reading the 10th fewest words correctly at endline. The  percentages of children with improved reading skills are calculated on  the basis of the numbers of matched pairs whose reading skills  increased by a given minimum—for instance, who read an additional  1, 5, or 10 words correctly per minute.
The USAID contractor staff used assessment data from eight USAID  programs to compare percentages of improved readers that they  calculated using the two methodologies and various minimum reading  levels. According to the contractor’s analysis, using the two  methodologies to analyze the same assessment results produced  different estimates of the percentages of children demonstrating improved  reading skills. Specifically, the contractor found that the gain score  methodology consistently produced higher estimates of improved readers  than did the threshold methodology. According to the contractor, the  discrepancy in the estimates from the two methodologies occurs because  the threshold methodology only counts children whose reading improves  across a threshold level. The threshold methodology does not count  children who demonstrate some level of improved reading, but whose  reading skills remain below or above a threshold level.
The USAID contractor has also reviewed possible procedures for  extrapolating the percentages of samples of assessed children  demonstrating reading improvements to all children exposed to the  reading interventions. However, the contractor noted that it needs to  conduct further technical work before making recommendations regarding  these procedures to USAID.
section_title: USAID Lacks Information Needed to Assess the Current Education Strategy and Plan a Post- 2015 Strategy
paragraphs: Lack of information about progress relative to USAID’s primary grade  reading goal limits the agency’s ability to assess its current approach to  achieving its reading goal and establish a realistic goal for a post-2015  education strategy. Because USAID has not selected a methodology for  estimating total numbers of children with improved reading skills resulting  from exposure to primary grade reading interventions, it has not produced  interim estimates of its progress toward the goal of 100 million children  with improved reading skills by December 2015. In addition, USAID’s  education strategy and guidance did not establish any interim targets for  measuring progress toward USAID’s primary grade reading goal. As a  result, USAID officials and other stakeholders currently lack aggregate  information needed to monitor and report on the numbers of children  whose reading skills have improved as a result of the USAID-supported  programs. Establishing interim performance targets and using reliable  estimates to assess progress toward an agency goal allow managers and  stakeholders to address any performance issues that may affect the  outcome of the agency’s goal, according to leading practices for  performance management. Frequently reporting progress in achieving  interim performance targets allows managers to review the information in  time to make improvements. Without interim targets and reliable  estimates, USAID cannot assess its actual progress against its planned  performance and use the information to make any adjustments needed to  better achieve its goal.
Furthermore, as the current education strategy’s December 2015 end  date approaches, the lack of aggregate information about USAID’s  progress in achieving its primary grade reading goal may constrain its  efforts to develop a new education strategy. According to agency officials,  USAID is presently engaged in developing its post-2015 strategy. We  have previously reported that agencies can use performance information  to make decisions that affect future strategies. helps program managers decide among competing priorities and  reassess their performance goals and strategies. The lack of information  about the outcome of USAID’s current primary grade reading goal could  limit USAID officials’ capacity to assess USAID’s current strategy and  goal and make effective decisions about a realistic goal for USAID’s  education strategy after 2015.
section_title: Conclusions
paragraphs: See GAO-05-927. improved reading skills by December 2015 and to provide data for  program monitoring and evaluation. In the five countries where we  conducted fieldwork, the USAID missions have implemented primary  grade reading programs with elements recommended in USAID guidance  and have conducted standardized reading assessments. However, the  very low reading skills revealed by data from these assessments suggest  that improving skills to the point where students can read with fluency and  comprehension will be a long-term effort in these and other developing  countries.
USAID’s use of standardized reading assessments represents an  important step in its efforts to measure the outcomes of its basic  education assistance. However, in part, because of the time required for  missions to design and implement programs and assess children’s  reading skills, USAID’s contractor has limited access to reading  assessment data. USAID officials acknowledged that the education  strategy’s time frames were unrealistic. Moreover, USAID has not yet  selected and documented a methodology—including reporting information  on the sampling designs, sample sizes, and modes of data collection—for  estimating total numbers of children whose reading skills have improved.  Until it selects and documents a methodology for calculating the  percentages of assessed children demonstrating improved reading skills,  and extrapolating the results to estimate the total numbers of children with  improved reading skills, USAID cannot monitor and reliably report  estimates of the results of its efforts to improve reading skills of 100  million children.
Furthermore, unless it selects and documents a methodology and sets  targets for measuring progress toward a reading goal in any future  education strategy, USAID will be limited in its ability to assess progress  toward that goal in comparison with planned performance. Accurate  reporting of USAID’s progress toward its strategic goal would inform  stakeholders about the impact of the agency’s efforts to improve  children’s reading skills in the final year of the current education strategy  and could help inform USAID officials’ efforts to develop a realistic goal  for a future education strategy.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Acting USAID Administrator take the following  three actions.
To help ensure that USAID reports a reliable estimate of the results of its  efforts to achieve the goal of improving reading skills of 100 million  children, we recommend that the Acting Administrator  select a methodology for estimating the total numbers of children with  improved reading skills as a result of exposure to primary grade  reading programs and document a description of the selected methodology as well as the  information necessary to evaluate the estimate (i.e., sampling  designs, sample sizes, and modes of data collection) when reporting  progress toward the reading goal.
To improve USAID’s ability to measure progress in achieving a  quantitative reading goal in any future education strategy, we recommend  that the Acting Administrator ensure that the future strategy includes  targets that will allow USAID to monitor interim progress toward its goal in  comparison with planned performance.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to USAID for comment. In its written  comments, reproduced in appendix IV, USAID agreed with our findings  and recommendations. USAID also provided technical comments, which  we incorporated throughout the report, as appropriate.
We are sending copies of this report to interested congressional  committees and the Acting USAID Administrator. In addition, the report is  available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-3149 or gootnickd@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix V.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: We were asked to review several aspects of the U.S. Agency for  International Development’s (USAID) basic education programming. We  focused our review on basic education programming related to Goal 1 of  USAID’s education strategy. This report (1) examines five USAID  missions’ implementation, performance monitoring, and evaluation of  primary grade reading programs and (2) assesses USAID’s efforts to  estimate progress toward the 2011-2015 education strategy’s primary  grade reading goal.
For both of these objectives, we analyzed funding for basic education for  fiscal years 2010 through 2014 and reviewed USAID’s 2011-2015  education strategy, as well as guidance for implementing the education  strategy and USAID’s technical notes to the education strategy. We also  interviewed USAID officials in the Bureau for Economic Growth,  Education and Environment; the Bureau for Policy, Planning, and  Learning; and four of the agency’s geographic bureaus. In addition, we  conducted fieldwork in five countries—Ethiopia, Malawi, Peru, the  Philippines, and Uganda—which we selected because they are  geographically diverse and received relatively large levels of U.S. funding  for basic education in fiscal year 2013. We selected the three African  countries because this region received more than 50 percent of USAID’s  allocation for basic education assistance in fiscal year 2013 and was  identified in the education strategy as a priority region for investment of  resources. In addition, we selected the five countries because the USAID  missions in these countries had primary grade reading programs under  way and had conducted at least baseline reading assessments. During  our fieldwork in these countries, we met with USAID mission officials;  USAID’s implementing partners; host government ministry of education  officials; local school district officials; school principals; teachers; and  members of parent groups, such as parent-teacher associations.
To examine the five missions’ implementation, monitoring, and evaluation  of primary grade reading programs—our first objective—we reviewed key  planning and reporting documents associated with USAID’s primary  grade reading program in each of the five countries. These documents  included, among others, contract awards or cooperative agreements,  monitoring and evaluation plans, performance management plans, and  implementing partners’ annual work plans and quarterly and annual  reports.
To determine whether the reading programs in the five countries included  key program elements that USAID’s education strategy recommends as  effective in improving reading outcomes, we compared elements in these  programs, as identified in their planning documents, with those  recommended in USAID’s technical notes to the education strategy. We  also visited schools, where we observed classroom reading lessons that  used instructional materials developed through USAID’s programs and  met with local teachers, education officials, and community organizations  to discuss program interventions. In Ethiopia, we observed local teams of  teachers, story writers, editors, and typists in the process of developing  reading curriculums in seven mother-tongue languages for grades 5  through 8. To identify the programs’ timelines for delivering improved  reading instruction to targeted students, we reviewed the programs’  performance management plans and annual work plans and met with  USAID officials at the country missions as well as implementing partner  officials.
To examine the five USAID missions’ performance monitoring of the  current primary grade reading programs, we reviewed outcome and  output measures identified in the missions’ and reading programs’  monitoring and evaluation plans and relevant standardized reading  assessment reports. We identified each country’s benchmarks for reading  at grade level—number of correct words read per minute and percentage  of comprehension questions answered correctly—and summarized each  program’s baseline data showing the percentage of students meeting  these benchmarks. To examine each mission’s plans for conducting  performance or impact evaluations of current primary grade reading   and interviewed  programs, we reviewed USAID’s evaluation policyUSAID officials at missions and implementing partners responsible for  conducting external evaluations of these programs. We also reviewed the  missions’ related requests for proposals and award agreements, as well  as work plans and interim or quarterly reports submitted to the missions  by these implementing partners.
To assess USAID’s efforts to estimate progress toward the education  strategy’s primary grade reading goal—our second objective—we  reviewed the USAID education strategy, guidance for implementing the  strategy, technical notes to the strategy, and the strategy’s reporting  guidance to identify the agency’s policies and guidelines for completing  reading assessments at USAID missions and for analyzing the data and  supporting information. We analyzed USAID reports on missions’  collection of reading assessment data. We also analyzed reports  prepared by the USAID contractor responsible for aggregating and  analyzing the reading assessment data, to determine the missions’  progress in transferring reading assessment data and supporting  information to the contractor. In addition, we interviewed USAID officials  and USAID contractor staff in Washington, D.C., regarding the guidance  and procedures for collecting and analyzing the reading assessment data  and supporting information as well as missions’ progress in transferring  the data to USAID for aggregation and analysis.
To examine USAID’s potential methodologies for estimating total  numbers of children with improved reading skills, we reviewed USAID  contractors’ documentation of proposed methodologies. We also  reviewed the USAID contractors’ reports on the development of potential  methodologies.contractor staff in Washington, D.C., about the development and  implementation of potential methodologies to measure improved reading  skills, and the challenges and limitations to collecting and analyzing  missions’ reading assessment data. We used GAO standards and   In addition, we interviewed USAID officials and USAID  generally accepted survey guidelines to assess USAID’s efforts to  estimate progress toward its primary grade reading goal.
To assess the reliability of the reading assessment data, we compared  the design, data collection, and analysis of results for the assessments at  the five missions against generally accepted survey guidelines as outlined  in Office of Management and Budget (OMB) guidance and USAID  guidance.accepted professional survey practices, including the collection of  interview-based survey data. We also found that the majority of OMB  guidelines were also reflected in USAID’s own implementation guidelines.  GAO methodologists compared the information collected from agency  documents and during interviews with agency officials and implementing  partners and generally found that the reading assessments met these  guidelines. We also determined that the data we obtained for both  objectives were sufficiently reliable for the purposes of our review.
Many OMB guidelines represent long-established, generally  We conducted this performance audit from May 2014 to May 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Basic Education Funding in Fiscal Year 2013 by Country and Region
paragraphs: In fiscal year 2013—the first year in which missions’ education programs  were expected to be fully aligned with the goals of the education  strategy—the U.S. Agency for International Development (USAID)  allocated funding to 41 countries for programs to improve the reading  skills of 100 million children in primary grades by December 2015—Goal  1 of its education strategy. Table 4 shows USAID funding in support of  Goals 1 and 3 of the strategy.
As figure 10 shows, more than half of USAID’s funding for Goal 1 of the  strategy was directed to countries in Sub-Saharan Africa, where baseline  reading scores were among the lowest globally, according to USAID.
Afghanistan and Pakistan received the next largest share, a combined  total of 18 percent of USAID’s 2013 basic education allocations.
section_title: Appendix III: Reading Assessment Data Collected At Five Selected Missions
paragraphs: We found that the design and implementation of the reading assessments  in the five countries where we conducted fieldwork—Ethiopia, Malawi,  Peru, the Philippines, and Uganda—met generally accepted survey  guidelines and U.S. Agency for International Development (USAID)  guidance on implementing Early Grade Reading Assessments (EGRA).  In four of the five countries we visited, a U.S.-based contractor assisted in  the design and data collection using the EGRA. Contractor staff in  headquarters provided input into both instrument and sample design and  conducted in-country training for assessors and supervisors. For  example, in each country, assessor and supervisor reliability scores were  calculated during training and those assessors and supervisors with low  scores were either retrained or not allowed to collect data. Also, sampling  designs appropriately considered analytical goals, such as expected  precision of subpopulation estimates, and the geographical distribution of  schools and the population of students. In some countries, the sampling  design also included information about languages. We also generally  found that the contractor appropriately calculated country-level estimates  based on the complex sample design after assessment data were  collected.
In three of four countries that used the EGRA, data were collected  electronically on tablets and regularly transmitted to contractor  headquarters for review (see fig. 11). Data were reviewed on a timely  basis so that any potential problems could be identified and corrected in  the field. For example, if an assessor was consistently taking too little  time administering a reading assessment, that assessor could be  identified and retrained if necessary. One country using an EGRA,  Ethiopia, collected data using paper questionnaires—a method with the  potential for more errors than electronic data collection. However, certain  data quality procedures were in place, such as requiring that a consultant  verify all of the data entered.
One country where we conducted fieldwork, Peru, did not use an EGRA  to collect reading assessment data but instead relied on reading  comprehension scores from a national assessment administered by the  Ministry of Education to all second graders. Ministry officials told us that  certain safeguards are in place to protect the integrity of the data  collected. For example, assessors attend training and must pass a test,  and ministry officials supervise testing sites where the validity of previous  results has been questioned. Ministry officials also told us that the current  methodology had been in place since 2007 and asserted that year-to-year  comparisons of test scores since 2007 were valid. Assessment scores  are provided to each region and school, including those receiving basic  education assistance, to track children’s reading comprehension  progress.
section_title: Appendix IV: Comments from the U.S. Agency for International Development
section_title: Appendix V: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Michael Rohrback (Assistant  Director), Howard Cott, Justin Fisher, Bradley Hunt, Jill Lacey, Reid  Lowe, Grace Lui, and Michelle Serfass made significant contributions to  this report.