THE SURPRISING LACK OF
EVIDENCE ABOUT WHAT WORKS IN EDUCATION‐ IMPLICATIONS FOR POLICYMAKERSNICHOLAS BURNETT
12 NOVEMBER 2014
JICA TRAINING, HIROSHIMA UNIVERSITYOUTLINE
1. The Global Learning Crisis
2. Quiz: What Improves Learning?
3. Why Learning Matters ‐ Evidence on Learning and Economic Development
4. Types of Evidence
5. Video: What Works in Primary Education?
6. Sources of Evidence
7. Why Education Innovation Matters
8. How to Assess Innovation?
9. Implications for Policy Makers
21. THE GLOBAL LEARNING CRISIS

Source: GMR 201331. THE GLOBAL LEARNING CRISIS• Learning (e.g. UWEZO in East Africa)
Test pass rates for children enrolled in Standard 3

Test pass rates for children enrolled in Standard 7

Notes: “combined” refers to passes on both numeracy test and at least one of the literacy test in the survey

Source: UWEZO, 20144

1. THE GLOBAL LEARNING CRISIS• Implications of the Learning Crisis

1. We need to focus on WHAT WORKS FOR LEARNING
2. We need to encourage INNOVATION FOR LEARNING5

2. QUIZ: WHAT IMPROVES LEARNING?
Which of the following improves learning?

1. Increased spending on education  NO
2. Mathematics materials at primary level  YES
3. Increased teachers’ salaries  NOT CLEAR
4. School based management  NOT CLEAR
5. Separate toilets for girls  NO
6. Extracurricular clubs for girls  YES
7. Low cost private schools for the poor  SOMETIMES
8. Conditional cash transfers linked to grades  YES
6

3. EVIDENCE ON LEARNING AND ECONOMIC DEVELOPMENT• Economic Development

7Source Hanushek 2013

3. EVIDENCE ON LEARNING AND ECONOMIC DEVELOPMENT•

It’s about children learning...

• Individual returns to education are linked tolearning, not just to years of education• The overall economy benefits from quality
education

8

3. EVIDENCE ON LEARNING AND ECONOMIC DEVELOPMENT• Returns to cognitive skills (literacy) are generally strong across countries93. EVIDENCE ON LEARNING AND ECONOMIC DEVELOPMENT• Estimated Returns to Cognitive Skills: Examples

 Three studies conducted in the U.S. show a 12% increasein earnings for every 1 SD increase in math scores.
(Mulligan,1999; Murnane et al, 2000; Lazear, 2003)
 Chilean data shows that 1 SD increase in test scores on
the IALS is associated with higher earnings of 15‐20%.
(Sakellariou, 2006)

10

3. EVIDENCE ON LEARNING AND ECONOMIC DEVELOPMENT• Education Quality and Economic Growth

Source Hanushek and Woessman, 200711

3. EVIDENCE ON LEARNING AND ECONOMIC DEVELOPMENT

123. EVIDENCE ON LEARNING AND ECONOMIC DEVELOPMENT• A ‘general’ theory of what influences learning outcomes
Economic, social, political context

DemandEndowments& BehaviorsofStudents
ParentsCommunity

Education
PolicyPolicy actions,specific programs
Systemicreforms

SupplyEndowments& BehaviorsofTeachers
Schools
Education authorities
CommunityStudent learning

Source: Vegas and Petrow 200813

4. TYPES OF EVIDENCE•

Randomized Control Trials – the gold standard for impact evaluation• Use of counterfactual• Advantages ‐ reliable• Drawbacks – costs, findings may not be policy relevant• Difficulty – defining results•

Regression Analysis• Standard statistical and econometric technique• Advantages – simple, cheap• Drawbacks – selection bias, correlation may not be causation• Difficulty – defining results•

Aggregating and Inferring• Crowd sourcing ‐ wisdom of the successful?
• Meta‐analysis• Identification of Essential Ingredients•

None of these say anything about Costs or Poverty Impact• Need benefit‐cost analysis, benefit‐incidence analysis etc. as well14

4. TYPES OF EVIDENCE
What sort of data measure learning?
• Grade completion – all students?
• Exams – all students?
• Assessments – valid statistical samples• National assessments – does your country have?
• International assessments – EGRA, PISA, TIMSS, PIRLS,SACMEQ, PASEC etc.
• School‐based or population‐based – students or school‐age
population?

15

5. VIDEO

Shari Krishnaratne“what works in education in developing countries”
https://www.youtube.com/watch?v=kX1Hp7ZkjO416

6. SOURCES OF EVIDENCE
References (see annex)• Primary Education
Krishnaratne et al. (2013) Quality education for all children? What works in education in developing countries. Working Paper
20. 3ie. http://www.3ieimpact.org/evaluation/working‐papers/working‐paper‐20/• Secondary Education
Banerjee et al (2013) Expanding Access and Increasing Student Learning in Post‐Primary Education in Developing Countries:
A Review of the Evidence. Abdul Latif Jameel Poverty Action Lab (J‐PAL) Post‐Primary Education Initiative Review Paper2
http://www.povertyactionlab.org/publication/expanding‐access‐and‐increasing‐student‐learning‐post‐primary‐education‐
developing‐count• Girls' Education
Unterhalter et al (2014). Girls’ education and gender equality. Education Rigorous Literature Review 2014. EPPI‐Center
https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/326205/Girls__Education_Literature_Revi
ew_2014_Unterhalter.pdf• Low Cost Private SchoolAshley et al (2014) The role and impact of private schools in developing countries. Education Rigorous Literature Review 2014.
EPPI‐Center: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/307032/Private‐schools‐
2014.pdf• Innovations
Education Innovation Foundation. http://www.educationinnovation.org/• Early Childhood Development
Engle etal (2011). Child Development 2: Chapter 2 Strategies for reducing inequalities and improving developmental
outcomes for young children in low‐income and middle‐income countries. Lancet. 378:1339‐53
Behrman et al (2014) Chapter 3: Preschool Program in Developing countries. In P. Glewwe (Eds.). Education Policy in
Developing countries, Chicago: University of Chicago Press17

6. SOURCES OF EVIDENCE• Primary Education
OutcomeEnrolment/Attendance

What works?
CCTs
Health interventions(attendance)

What doesn’t work?
Materials(attendance)

Progression, Repetition,
Dropout

Test Score

Source: 3ie 2013

Materials(maths)Materials (language)

What promising?

Not promising but
could look further
research

What we don’t know

Teacher Resources(attendances)
Buildings
ECD (enrolment)
School Feeding(enrolment)

SBM
School Feeding(attendance)

Teacher Resources(enrolment)
Health (enrolment)
ECD (attendance)
Materials (enrolment)
Vouchers
Schools Fees
Providing Information

Teacher Resources(dropout)
CCTs
SBM (progression)

SBM (dropout)

Teacher Resources
HealthECD
Materials
School Feeding
Vouchers
School Fees
Providing Information

Teacher Resources
SBM (maths)
School Feeding(maths)Teacher Resources(global)Health(maths, language)
CCTs (global)
School Feeding(language)

Health (global)
CCTs
ECD
Materials (global)
SBM
School Feeding
Vouchers
School Fees
Providing Information18

6. SOURCES OF EVIDENCE• Girls’ Education (Impact on Learning)
Interventions

StrongPromising

More needed

Resource andInfrastructure

Resource Intervention in cash for family orchildren impact on success in the grades
achieved

The provision of additional schools in
underserved area impacts on girls’ learning
outcomes

Research into the impact of resources andinfrastructure tends to focus on attendance atschools. More evidence in needed to investigate
the impact of these interventions on learning

Institutional andPolicyGroup learning is associated with improved
learning outcomes

There is promising evidence to show that
informal teacher training to develop attitudes ofinclusion tolerance has a positive impact

More research is needed to reinforce existingpromising evidence that the empowerment ofwomen teachers may have a positive impact on
girls’ learning

Learning outside the classrooms throughformal and informal extracurricular activities(eg tutoring after‐school clubs) has a positiveeffect on girls’ learning outcomes

The involvement of women in school
governance and community mobilization and incommunity leadership may have a positive
impact on girls’ confidence

More research is needed into the impact ofschool choice and inclusive strategies on girls’
learning

Girl‐friendly school, and quality –mix, or
combined school‐level reforms may have apositive impact on girls’ learning
Norm and Inclusion

There is promising evidence thatcomplementary learning spaces to school (e.g.
NGO programmes, girls’ clubs) are effective in
giving opportunities to discuss gender equality
Teaching about personal, social and health issue
linked with sex education both at school and incomplementary programmes is effective inproviding knowledge
Women’s literacy programmes, particularly
when they offer women and girls the chance todevelop literacy knowledge, skills, and spaces
for reflective discussion

Source: EPPI‐Center 2014

More research needed into religion, educationand shifting gender norms note the importanceof key individuals, the significance of supportive
networks within faith communities, and thepotential for religious institutions to sited for
critical reflection on gender norm
Also more needed on the effect of tackling
gender‐based violence in schools, working withboys on gender equality, work to include
marginalized girls, developing combined
progrmmes linking work in schools on genderequality to community work19

6. SOURCES OF EVIDENCE• Low Cost Private Schools
Strong Evidence

Moderate Evidence

Teaching is better in private school than in state schools, interm of higher levels of teacher presence and teachingactivity as well as teaching approaches that are more likely
to lead to improved learning outcomesPrivate school pupils achieve better learning outcomes when compared with state schools.
However, there is ambiguity about the size of the true private school effect. In addition many
children may not be achieving basic competencies even in private schools
The cost of education delivery is lower in private schools than state schools often due tolower salaries for private school teachers compare with their government school
counterparts. More of the evidence does no rigorously analyze the cost‐effectiveness ofprivate schools; however, there is some limited evidence to indicate higher cost‐effectivenessof private schools than state schools in specific contexts
Girls are less likely than boys to be enrolled in private schools. However, this finding iscontext specific with a minority of studies finding that private schools reduce the gender gap
in certain contexts
Perceived better quality of private schools (in terms of teaching, teacher attendance, school
performance, small class size, discipline) compared with state schools is a key factor inparents’ choice of private schools. Other important factors cited include English‐language
instruction, future occupation possibilities and promotion rates to secondary school.
The perception of ‘private schools as better quality’ is informed informally, often throughparents’ informal social networks; such sources play a significant but often under‐recognisedrole in informing users in their choice of school.
Attempts by states to intervene in the private education sector are constrained by a lack ofcapacity, legitimacy and knowledge of the sector to implement effective policy frameworks.
Where state regulation of private schools exists, it is not necessarily effective or may be
selectively enforced offering opportunities for rent seeking and bribery. Although the findings
are mostly negative, there are some examples of positive state regulation supporting theexpansion of private school provision; however there are also concerns that private sector
provision may be promoted by states without adequate regulation and quality controls.

Source: EPPI‐Center 201420

6. SOURCES OF EVIDENCE• Low Cost Private Schools
Inconclusive
The evidence is ambiguous about whether private schools
geographically reach the poor. Although private schools are continuing
to focus on urban areas, they are also becoming increasingly prevalent
in rural areas; but research cautions against assuming this means they
are reaching the poor.
There is a small but consistent evidence base that private schools are
more expensive than state schools in terms of both school fees andhidden costs such as uniforms and books.
The evidence on whether the poor are able to pay private school fees isambiguous. Most is neutral, some is negative, but there is no positive
evidence. A number of studies find that a very small minority of childrenof lower economic quintiles access private schools. Financial constraints
are a key factor limiting or preventing poorer households from enrolling
their children in private schools. Where children of poorer households
do attend private schools, research indicates that welfare sacrifices are
made and continued attendance is difficult to sustain.
There are very few studies addressing rigorously the account ability ofprivate schools to users. Of these, there is some consistent evidence thatusers participate in and influence decision making in private schools.
While a small body of mainly anecdotal evidence indicates that teachersand schools may respond to parents’ demands and complaints andultimately the potential threat of parents exercising choice, no evidence
was found in the studies reviewed that users do in fact change schools in
response to quality concerns, or are more likely to do so in the case ofprivate schools than government ones.

Limited
There is limited evidence to support any firm conclusion about thefinancial stability of private schools. However, the evidence available
indicates that private schools, especially LFPs, may be vulnerable toclosing down after short periods of time. However there was alsocounter evidence of established recognized private schools operatingover many years. Importantly an assessment of the sustainabilityof different financial models is lacking in the literature.
The evidence on subsidies is limited in scope, size and context, but one
donor funded programme in Pakistan indicates that conditional andtargeted subsidies can raise the quality of inputs.
There is little evidence to support or refute the question of the system‐
wide effects of private education. The evidence base on whether private
schools complement or compete with government school provision isvery small, however some evidence indicates a supply‐side synergy
between government and private schools provision and there isevidence that private schools are filling gaps where supply ofgovernment schools is low, but also where government schools areperforming poorly. The evidence on whether the effect of competition isto drive up the quality of public schools or to deplete it by encouraging
more able students to exit the state sector is sparse and contested.

Source: EPPI‐Center 2014
21

7. WHY EDUCATION INNOVATION MATTERS• Out of School Children
Number of out‐of‐school children ofprimary school age by region and sex, 2000‐2012

Number of out‐of‐school adolescents oflower secondary school age by region and sex, 2000‐2012

Source: UNESCO GMR Policy Paper No. 14, 2014

22

7. WHY EDUCATION INNOVATION MATTERS• Learning Crisis

Source: GMR 2013

23

7. WHY EDUCATION INNOVATION MATTERS• EffectivenessSource: GMR 2013

24

7. WHY EDUCATION INNOVATION MATTERS• What are Education Innovations?
Programs, Policies and Practices
that increase access to or improve
quality or reduce costs ofeducation for the poor

25

7. WHY EDUCATION INNOVATION MATTERS• Education Challenge: The Reality & The Opportunity

123

Progress in education does not grow and spread fast enoughSolutions do exist – promising innovations in education areconstantly emerging…BUT their uptake and diffusion is slow!

267. WHY EDUCATION INNOVATION MATTERS• Understanding the Evolution of an Innovation: Four Stages
Takes 5‐10 years to go through first three stages

Blueprint

Develop idea andcompelling value
proposition

Validate

PrepareTest and refine theinnovation. May
involve somegrowth.

Enhance theconditions
required forscaling (i.e.,
customerawareness,
effective supplychains)

Scale

Roll out model toreach a large
number ofcustomers

Source: Blueprint to Scale, produced by Monitor Group in collaboration with Acumen Fund277. WHY EDUCATION INNOVATION MATTERS• The R4D Center for Education Innovations

Global Public Good Platform for Data and Connections
educationinnovations.org
1

Largest global public
database of innovations

2

3Connections leading toimprovement, scale‐up
and adaptation

New analysis about‘what works’, how and why28

8. HOW TO ASSESS INNOVATION?

• Scaling Up ‐ Loose term ……..need to be precise when using

Interpretations
 Expand Same context
 But 5 to 10 very different
from 50 to 10,000
 Replicate
 Different context
 Adapt Same context, or Different context

Requirements
Analytical requirements
Clear identification and analytic
separation of “active ingredients” andof contextual factors.
Assessment of the effectiveness of theactive ingredients
How to do this? Evolving techniques….

29

8. HOW TO ASSESS INNOVATION?
• Proposed Scaling Framework
The activities below represent key checkpoints in the process of identifying promising global models potentially
adaptable to other contexts. The order of the stages may vary by user, desired output, and information/ resources
available. As an iterative process, users may revisit certain steps to further refine analysis.

Search forprogram models

Identify theactive ingredient(s)Based on the problem you are trying to What are the core components orsolve for and specified parameters,
attributes of the program that makewhat models currently exist?
it successful?
What are the contextual factors forthe active ingredient?

Determine
effectiveness

Assessscalability

What evidence exists on theeffectiveness of identified activeingredient(s)?

What are the barriers and levers
to scaling the active ingredient(s)in a specific context?

Narrowed list of active ingredients

List of barriers and levers toadaptation

OUTPUTS
Short list of program models

List of isolated active ingredients30

8. HOW TO ASSESS INNOVATION• Sources of Information about Innovation• R4D Center for Education Innovations (CEI): http://www.educationinnovations.org/• Focus on the poor in low and middle income countries• Over 500 documented programs, mainly non‐state• World Innovation Summit on Education Hub: http://www.wise‐qatar.org/edhub• Over 100 described innovations in rich and poor countries• OECD Center for Education Research and Innovation: http://www.oecd.org/edu/ceri/• Focus on OECD countries• STIR: http://www.stireducation.org/• Focus on teacher innovations, India and now expanding to Uganda31

9. IMPLICATIONS FOR POLICY MAKERS
1. Making policy with insufficient information•

Do not assume you know, e.g. “”we have always done it this way”•

Do not assume what special interests say, e.g. teacher union or banks lending forhigher education•

Constantly scan for relevant innovations•Apply logframe or testable theory of change•

See which elements may have evidence•For those which do not have evidence, make reasonable assumptions but install
M&E to verify or not these assumptions•Another approach: reward results but don’t tell implementers how to achieve them•••Results‐based financing
Output‐based aid
Pay for performance etc.
32

9. IMPLICATIONS FOR POLICY MAKERS
2. Building evidence‐gathering into policy• Allocate funds for research and manage research budgets actively• Insist external finance for research flow through the ministry• Set up policy advisory committees that include researchers• Keep a focus on outcomes and, to a lesser extent, outputs• Learn from other countries’ experience• Always put M&E in place when scaling innovations and new policies

33

Thank you
CONTACT INFO: NBURNETT@R4D.ORG

34