Changing Mindsets: Effectiveness trial
Evaluation Report

July 2019

Francesca Foliano, Heather Rolfe, Jonathan Buzzeo, Johnny Runge,
David Wilkinson

Changing Mindsets
Evaluation Report

The Education Endowment Foundation (EEF) is an independent grant-making charity dedicated to breaking the link
between family income and educational achievement, ensuring that children from all backgrounds can fulfil theirpotential and make the most of their talents.

The EEF aims to raise the attainment of children facing disadvantage by:• identifying promising educational innovations that address the needs of disadvantaged children in
primary and secondary schools in England;• evaluating these innovations to extend and secure the evidence on what works and can be made towork at scale; and• encouraging schools, government, charities, and others to apply evidence and adopt innovations
found to be effective.

The EEF was established in 2011 by the Sutton Trust as lead charity in partnership with Impetus (formerly Impetus
Trust) and received a founding £125m grant from the Department for Education.
Together, the EEF and Sutton Trust are the government-designated What Works Centre for improving education
outcomes for school-aged children.
For more information about the EEF or this report please contact:

Jonathan Kay
Education Endowment Foundation
5th Floor, Millbank Tower
21–24 Millbank
SW1P 4QP
0207 802 1653
jonathan.kay@eefoundation.org.uk
www.educationendowmentfoundation.org.uk

1

Changing Mindsets
Evaluation Report

Contents
About the evaluator..............................................................................................................3
Executive summary..............................................................................................................4
Introduction ..........................................................................................................................6
Methods ..............................................................................................................................10
Impact evaluation ...............................................................................................................19
Implementation and process evaluation ..........................................................................29
Conclusion..........................................................................................................................42
References ..........................................................................................................................45
Appendix A: EEF cost rating .............................................................................................46
Appendix B: Security classification of trial findings .......................................................47
Appendix C: Extract from the training material................................................................48
Appendix D: Consent forms ..............................................................................................49
Appendix E: Mindset measure questions, MSLQ questions and subscales ..................55
Appendix F: Code for randomisation................................................................................58
Appendix G: Effect size from simple models ...................................................................60
Appendix H: Code for the analysis of primary outcomes ...............................................62
Appendix I: Histograms of post-test outcomes ...............................................................65
Appendix J: Additional subgroup analysis ......................................................................68
Appendix K: Results of the missing data analysis and additional estimates ................712

Changing Mindsets
Evaluation Report

About the evaluator
The impact and process evaluations were completed by NIESR researchers. Francesca Foliano led the impact
evaluation, working with David Wilkinson. The process evaluation was carried out by Jonathan Buzzeo and Johnny
Runge. Heather Rolfe oversaw the design and delivery of the process evaluation and contributed to the report; Richard
Dorsett, Cinzia Rienzo, and Matthew Burnsall were involved in the early stages of the research.

3

Changing Mindsets
Evaluation Report

Executive summary
The project
The Changing Mindsets project aimed to improve attainment outcomes at the end of primary school by teaching Year
6 pupils that their brain potential was not a fixed entity but could grow and change through effort exerted. The
programme, delivered by Portsmouth University, taught pupils about the malleability of intelligence through workshops.
Teachers attended short professional development courses on approaches to developing a ‘growth mindset’ before
delivering sessions to pupils weekly, over eight consecutive weeks. Teachers were encouraged to embed aspects ofthe growth mindset approach throughout their teaching—for example, when giving feedback outside of the sessions.
They were also given access to digital classroom resources, such as a video case study of Darwin overcoming adversity
in his own life, as a practical example of the importance of having a growth mindset.
The project was a randomised controlled trial (RCT) and included 101 schools and 5018 pupils across England,
assigned to either intervention or control groups. The trial ran from September 2016 to February 2017. The processevaluation involved interviews with teachers, focus groups with pupils receiving the intervention, lesson observations,and surveys of both treatment schools and control groups throughout the course of the intervention.
Table 1: Key conclusions
Key conclusions
1.

Pupils in schools that received the intervention did not make any additional progress in literacy nor numeracy—as measured
by the national Key Stage 2 tests in reading, grammar, punctuation, and spelling (GPS), and maths—compared to pupils
in the control group. This finding has high security.

2.

This evaluation also examined four measures of non-cognitive skills: intrinsic value, self-efficacy, test anxiety, and selfregulation. The evaluation did not find evidence of an impact on these measures for pupils in schools that received Changing
Mindsets. A positive impact was found for the intrinsic value measure, but the impact was small and was not statisticallysignificant.

3.

Among pupils eligible for free school meals (‘FSM pupils’), those in schools that received the intervention did not make any
additional progress in literacy nor numeracy—as measured by the national Key Stage 2 tests in reading, GPS, and maths—
compared to FSM pupils in schools that did not receive the intervention.

4.

One explanation for the absence of a measurable impact on pupil attainment is the widespread use of the growth mindset
theory. Most teachers in the comparison schools (that did not receive the intervention) were familiar with this, and over athird reported that they had attended training days based on the growth mindset approach.

EEF security rating
These findings have a high security rating. This was an effectiveness trial, delivered as it would be at scale in a rangeof schools. The trial was a well-designed two-armed randomised controlled trial. Attrition rates for the primary outcomes
were approximately 11%, and above 40% for secondary outcomes. The trial was well-powered and the pupils in
Changing Mindsets schools had similar prior attainment to those in the comparison schools.

Additional findings
Separate exploratory analysis was undertaken for three subgroups: females, males, and pupils who were shown in
the pre-trial tests to believe that the ability to learn is a fixed trait. There was no evidence of an impact of the
Changing Mindsets intervention on the outcomes of Year 6 students in these subgroups.
The programme was delivered as intended and well received by schools, teachers, and pupils; the latter showed goodunderstanding of its messages and how they relate to their own learning. Despite the absence of a measurable
impact, teachers perceived changes in pupils’ attitudes, enthusiasm, and perseverance. Based on their experiencesof the programme, some schools were planning to continue and extend its use, including introducing it to younger age
groups, implementing whole-school approaches, and working with parents.
4Changing Mindsets
Evaluation Report

The findings add to an extensive empirical literature that has investigated the effects of the growth mindset approachon pupils’ attainment and learning motivation by using experimental research designs and school-based interventions
based on rigorous randomised trials. Other studies have often focused on older pupils, including interventions run inhigh schools in the U.S. and Norway, but have found evidence that it is possible to shape pupil’s beliefs about theirintelligence with long-term impact on their attainment.
The pervasiveness of growth mindset theory is one potential explanation for the absence of a measurable impact of theintervention: for example, over a third of teachers in comparison schools had previously attended training days basedon growth mindset interventions. However, while many staff members were aware of similar approaches prior to theirinvolvement in the project, few said they had used these within their own teaching practices. Due to the ubiquitous
nature of growth mindset, it may be hard to find schools for future research where awareness of the concept is low.

Cost
The Changing Mindsets intervention costs approximately £4 per pupil. This cost was estimated using the assumptionthat the University of Portsmouth delivers the training to the school during an inset day and provides the intervention
materials for all staff. The Changing Mindsets intervention costs £1,800 in the first year. There is no cost in thesubsequent years.

ImpactTable 2: Summary of impact on primary outcome
Outcome/Group

KS2 maths
KS2 reading
KS2 Grammar
Punctuation andSpelling

Effect size
(95%
confidence
interval)-0.01(-0.04; 0.01)-0.00(-0.02; 0.02)-0.00(-0.03; 0.03)

Estimatedmonths’progress

P value

No. of pupils

EEF security
ratingEEF cost rating

0

0.37

4454

£££££

0

0.72

4437

£££££

0

0.90

4448

£££££

5

Changing Mindsets
Evaluation Report

Introduction
Background evidence
Research in education and labour economics highlights the importance of non-cognitive traits such as perseverance
and resilience in determining long-term outcomes in education and the labour market (Heckman, 2006). A policyrelevant research question is whether schools can foster these non-cognitive skills and, as a consequence, boost pupil
attainment. An extensive literature in psychology developed by Carol Dweck and colleagues introduced the idea that it
is possible to increase pupils’ perseverance and, as a consequence, to improve their attainment by teaching them that
their brain has the potential to grow through effort exerted on challenging tasks (Dweck, 2006; Yeager and Dweck,2012). In this literature, pupils’ beliefs about their ability to learn are referred to as ‘implicit theories of intelligence’; a‘fixed mindset’ is the belief that the ability to learn is a fixed trait and a ‘growth mindset’ is instead the belief that
intelligence can be developed through effort and dedication.
Studies in experimental psychology and economics conducted in the U.S. and Norway have investigated the effect ofschool-based interventions that shape pupils’ beliefs in their ability to learn: they conclude that pupils who are taughtthat their intelligence can be developed through effort and dedication show more perseverance (Bettinger et al., 2016)
and achieve more in mathematics and English (Paunesku et al., 2015 and Yaeger et al., 2016). These findings have
led to a widespread use of interventions inspired by growth mindset theory in the U.S. education system. However, all
the studies that find promising results are based on changing mindsets trials for secondary school pupils. In addition,some of these studies are conducted with only limited samples of students. It is indeed relevant to investigate if similar
or greater effects can be achieved when the intervention is run with younger pupils and in different education systems.
To this end, a pilot trial was first conducted to test the efficacy of the growth mindset approach in improving pupil
attainment in primary schools (Rienzo et al., 2015); within this trial, two small interventions were delivered by Portsmouth
University. The first intervention taught pupils about the malleability of intelligence through six workshops and four
further sessions delivered by two local organisations: its estimated effect was an average of two additional months’
progress in English and maths, although the results were not statistically significant. The second intervention involved
a professional development course of two half-days that trained teachers on approaches to developing and reinforcing
growth mindsets through their teaching: this intervention had no impact on progress in maths for the pupils taught bythose teachers and two months less progress in English (again not statistically significant). The promising nature of thefirst pilot trial led to the development of a more substantial trial that incorporated elements of both the previous
interventions and scaled up the sample size. In this larger scale randomised controlled trial, Year 6 teachers in treatment
schools received training that introduced them to mindset theory and evidence and provided tips for how to embed theapproach in their classrooms (for example, how to communicate incremental beliefs to pupils through feedback andpraise). In addition, teachers were given the materials and training to run an eight-week programme (up 2.5 hours aweek) of weekly lessons and activities with their pupils. Schools in the waitlist control group received the training two
academic terms later. This evaluation seeks to assess the impact on the attainment of pupils who received theintervention, compared to pupils who did not receive the intervention.

Intervention
The Changing Mindsets project was developed to evaluate the effect of a growth mindset on achievement in English
schools. The trial sought to increase the achievement of primary school pupils in numeracy and literacy by helping themto develop a growth mindset. The project’s theory of change is that teacher behaviour and language changes as a resultof the training and that, along with the delivery of mindset classes, pupils change their own mindset and expectationsof self. This leads to changes in pupils’ learning behaviour as they become more resilient and engage in challenge andopportunity. As a consequence, pupils’ academic attainment improves, as well as their self-esteem and mental
wellbeing.
The project combines two interventions from a previous efficacy trial that showed promising results: (1) training pupils
in the treatment schools by using a structured series of learning resources and activities and (2) training teachers inmindset theory and offering tips for every day practice (Rienzo et al, 2015). The Changing Mindsets project trainedteachers in mindset theory but also gave them the resources to deliver a programme in the classroom.
6

Changing Mindsets
Evaluation Report

Pupils were exposed to the mindset theory over a structured series of classes led by their own teachers. The aim ofthese sessions was to change the way that pupils think about their intelligence, in particular to build high expectations
and resilience and to encourage specific plans and goals that emphasise growth, the development of skill, or the
expansion of knowledge. The programme was delivered to pupils through eight sessions. These covered themes
including the meaning of intelligence, dealing with mistakes and emotions, understanding the brain and encouraging
challenge, effort and persistence, stereotypes, and inspirational people. In addition, teachers worked to make theintervention more effective by embedding the growth mindset approach in their everyday class activities, for example,
by repeating the message that making mistakes is an opportunity to learn rather than a negative experience.
Teachers received one day of training that was prepared and delivered by a team of education psychologists from the
University of Portsmouth. This training introduced teachers to mindset theory and evidence and provided tips on how
to embed the approach in their classrooms/schools (for example, how to communicate incremental beliefs to pupils
through feedback and praise). To supplement these suggested changes to everyday practice, teachers were given thematerials and training to run an eight-week programme (up to 2.5 hours a week) of weekly lessons and activities with
their Year 6 pupils. Specifically, teachers were given a training manual that included comprehensive lesson plans forthe eight sessions and a USB stick with additional material to support their interaction with children and their parents.
In addition, teachers were granted free access to online videos supplied by the Positive Edge Foundation;1 these videos
were referenced in the lesson plans and teachers were encouraged to use the videos to explain or emphasize particularconcepts such as resilience and learning from mistakes.2 The videos focus on three famous people (Darwin, Einstein,and Wilma Rudolph), describing their lives and how they overcame adversity, in line with the growth mindset message.
They included, for example, a video about Charles Darwin’s less-than-promising school performance and a medalwinning athlete who overcame a difficult start in life. Other materials provided by the project team included posters on
growth mindset and pupil quizzes.

DesignThe design is for a two-arm trial, with schools participating in the trial randomly assigned to either the intervention groupor the control group. All schools paid £500 to receive the intervention. All schools received the full Changing Mindsets
training. Schools in the intervention group received their training in September or October 2016, while waitlist control
schools received the training two academic terms later. This is an effectiveness trial that was delivered as it would be
at scale and it included primary schools in the South East, South West, Midlands, North East, and North West of
England. Within each school, the trial focused on all Year 6 pupils with the exception of those who did not consent tobe part of the study.
Headteachers and Year 6 teachers of schools in the intervention group were invited to attend one of approximately ten
training days hosted across the U.K. in September and October 2016. Headteachers and Year 5 teachers of schools in
the control (waitlist) group instead were invited to attend one of approximately ten training days hosted across the U.K.
in June and July 2017. The waitlist design was chosen to keep the control schools interested in the trial and to minimize
their attrition. It was also chosen to reduce the possibility of these schools adopting a similar programme during theacademic year of the intervention.
Year 6 teachers in schools that were randomly assigned to the treatment group delivered the intervention to the wholeclass. Schools were eligible if they had not used a systematic mindset programme with their Year 6 cohort before andif they were able to attend one of the training dates provided by the team from the University of Portsmouth. Schoolsthat wanted to enter the randomisation needed to provide the following: (1) a headteacher consent form, (2) confirmationthat parent opt-out consent forms were sent out, (3) pupil data form including UPNs for all Year 6 pupils, (4) the response
to a baseline school questionnaire.

1 Positive Edge Education is a private company that develops holistic educational programs designed to inspire and engage, foundedby Thomas Westenholz.
2 Appendix C includes a sample of slides from the training day provided by the Growing Learners group at the University ofPortsmouth.

7Changing Mindsets
Evaluation Report

Impact and Process Evaluation
The impact evaluation is a rigorous empirical analysis based on experimental data collected during the trial. The primary
aim of the Changing Mindsets impact study is to evaluate whether a growth mindset approach in school can increase
Year 6 literacy and maths scores, measured by Key Stage 2 Standard Attainment Tests. The secondary aim of the
impact study is to evaluate whether the growth mindset approach in schools has an impact on Year 6 pupils’ motivation,
effort, seeking behaviour, and attitude towards tests. These non-cognitive traits are captured by four subscales of theadapted MSLQ, namely intrinsic value, self-efficacy, test anxiety, and self-regulation (Pintrich and Van De Groot, 1990).
It is important to notice here that the protocol only mentions self-efficacy and self-regulation as the secondary outcomesof interest; however, the Statistical Analysis Plan (SAP) includes all four subscales. The aims of this impact evaluation
are therefore in line with the SAP approved by the EEF.3
The impact assessment is supplemented by a comprehensive process evaluation aimed at understanding how the
Changing Mindsets intervention programme was implemented and delivered in treatment schools and whether thisdiffered from the intended delivery model. It also aimed to shed light on implementers’ as well as pupils’ responsivenessto the intervention, and the perceived impact on participants. It also gathered information on the activity of schools within
the control group over the course of the trial period. Methods included fieldwork visits to six treatment schools using
individual interviews with teachers and senior leaders, focus groups with pupils, and observations of three lessons.
Schools were selected to include a range of key characteristics: Ofsted rating, proportion of pupils receiving free school
meals, geographical location, and whether located in urban or rural settings. Interviews used semi-structured topic
guides and were recorded digitally and transcribed. A framework method was used to analyse transcripts thematicallyusing Excel.4 Other methods used to collect data from the wider sample of schools were: an end-of-project survey oftreatment schools, a fidelity survey administered to treatment schools over the course of the intervention, and a control
group survey. Because of the small scale of these surveys, they were analysed manually.

Ethics and trial registration
We take seriously the ethical issues raised in both the quantitative and qualitative elements of the research. NIESR
adheres to the Ethics Guidelines of the Social Research Association (SRA). Members of the process evaluation teamhave Disclosure and Barring Service (DBS) clearance. The ethical implications of the research have been discussedwith reference to the SRA guidelines (in particular their Standard Protocols for Checking Ethical Considerations) bymembers of the project team and NIESR's Senior Management Team to determine the appropriate course of action
and whether further approval is needed. While NIESR has an ethics committee consisting of trustees, we do notconsider that this trial requires such additional clearance; this decision has been ratified by our Director. As part of this
3 The protocol can be found here:

https://educationendowmentfoundation.org.uk/public/files/Projects/Evaluation_Protocols/EEF_Project_Protocol_CHanging_Mindset_Regrant.pdf; the SAP can be found here:
https://educationendowmentfoundation.org.uk/public/files/Projects/Changing_Mindsets_SAP_2018.02.08.pdf

4 A framework approach enables the analysis of qualitative data in a written form, and is therefore appropriate for the analysis oftranscripts of interviews with teachers and other project participants, as well as research notes taken during observation of classroom
interventions. It entails coding data into themes and issues. In this case, codes were a mixture of predetermined ones, developed
during the design of the process evaluation and taking account of the aims of the intervention, as well as those that emerged fromthe text of transcripts and observations. Codes identified different types of information, for example, more tangible one such as
knowledge of Mindset, experiences of the training, and of putting the approach into practice as well others such values andfeelings. Throughout the analysis process, we looked for similarities and differences in the data. The framework approach allows for
tracts of text to be classified under more than one code, and codes were, in some cases, amalgamated to form wider groups,
particularly where substantial issues were concerned.
The codes and groups developed in the analysis of data formed the analytical framework and were used to structure the findings
into a preliminary report. We then re-structured this document to follow the format required by the EEF. This involved structuring thefindings using the main EEF process evaluation criteria: implementation, outcomes, and fidelity.

8Changing Mindsets
Evaluation Report

trial we collected personal information and applied for extracts from the National Pupil Database (NPD) using thisinformation. However, we did not link this data with any other sources; whether participants are in the treatment orcontrol groups they are identified from the school that they attend. Participants’ confidentiality and anonymity wassafeguarded by the methods that we have in place.
Recruitment took place prior to the introduction of the General Data Protection Regulation (GDPR), the new European
Union privacy law that defines guidelines and regulations on how data is processed, used, stored, or exchanged.
Schools that had registered their interest and were eligible for the trial were sent a welcome pack that included aheadteacher informed consent form and a letter for parents. The headteacher informed consent form set out the detailsof the trial and clarified that schools were free to withdraw from the intervention at any point.
Participants’ parents were provided with a letter that included information on the aims of the research and the use ofdata in order for them to be able to make an informed decision about whether to withdraw their child’s data from thestudy at any time. Data from the NPD and pupil information obtained from the schools were transmitted and stored
using the security principles underlined in the NIESR Data Security policy. This includes secure transfer of data anduse of password-protection and encryption as appropriate during data storage.
This trial was registered on the International Standard Randomised Control Trial Number registry at
http://www.isrctn.com/ISRCTN14702744.

Data protection
We processed individual and personal data in line with the requirements of the GDPR and our legal basis for processing
the data is legitimate interest. Additionally, NIESR was certified as GDPR compliant on the 29 of May 2018 by the British
Assessment Bureau.

Project team
The Changing Mindsets project was developed by Growing Learners, a group of education research psychologists, led
by Professor Sherria Hoskins and based in the Psychology Department at the University of Portsmouth. 5
The Growing Learners team included:
Dr Sherria Hoskins,
Dr Frances Warren,
Dr Joanna Nye.
The project also included additional materials and videos from Positive Edge Education, a company that develops
holistic educational programmes designed to inspire and engage, founded by Thomas Westenholz.6
The impact and process evaluations were completed by NIESR researchers. Francesca Foliano led the impact
evaluation, working with David Wilkinson. The process evaluation was carried out by Jonathan Buzzeo and Johnny
Runge. Heather Rolfe oversaw the design and delivery of the process evaluation and contributed to the report; Richard
Dorsett, Cinzia Rienzo, and Matthew Burnsall were involved in the early stages of the research.

5 http://www.port.ac.uk/department-of-psychology/community-collaboration/growing-learners/
6 https://positiveedgeeducation.com/

9Changing Mindsets
Evaluation Report

Methods
Trial design
Table 3: Summary of trial design and randomisation
Trial type and number of arms

Effectiveness trial, two arm randomised controlled trial.

Unit of randomisation

Schools.

Stratification variables

variableKS2 scores in maths, reading, grammar, punctuation and spelling (GPS),and numeracy.

measure (instrument, scale)

Test scores in reading and GPS are measures of literacy (range ofscores: 80–120);
numeracy: KS2 test scores in maths are a measure of numeracy (range ofscores: 80–120).

variable(s)Four subscales of the MSLQ: self-efficacy, intrinsic value, test anxiety, andself-regulation.

measure(s)(instrument, scale)

Four subscales of the MSLQ that were adapted by the Growing Learners
team and administered to all Year
6 pupils in the study. All items are scored from 1–7, (with 1 indicating‘strongly disagree’ and 7 indicating ‘strongly agree’. Subscales are thencalculated as a mean score.

Primary outcome

Secondaryoutcome(s)Tertiles of school-level KS1 performance in 2014/2015 and fivegeographical areas: Midlands, North East, North West, South East, andSouth West of England

Table 3 presents a summary of the trial design. The design of this effectiveness study is a two-arm cluster randomised
trial, with schools participating in the trial randomly assigned to either the intervention group or the control group. Schoollevel randomisation was chosen over class or pupil level to minimise the chance of contamination of control bytreatment. The trial had a waitlist design. All schools paid £500 to receive the intervention. All schools received the full
Changing Mindsets training. Schools in the intervention group received their training in September and October 2016,
while waitlist control schools received the training two academic terms later. Randomisation was carried out using blockrandomisation techniques, controlling for prior attainment at the school level and geographical area. Blocks were definedaccording to terciles of school-level Key Stage 1 (KS1) performance within each of five locations: Midlands, North East,
North West, South East, and South West of England. The purpose of blocking was to improve the balance between thetreatment and control groups in terms of key outcome-related characteristics and also to increase the precision of
estimates. The trial was delivered as it would be at scale and 101 schools with about 5,000 Year 6 pupils were recruited
to the trial.

Participant selection
The study was advertised to schools with an advert on the EEF website, adverts on social media (for example, Facebook
groups for Year 6 teachers), and by contacting schools via email from the National College of Teachers. In addition, asnowball method was utilised whereby interested schools and other school contacts were asked to pass on the advert
to other schools in their own networks, and some schools volunteered via this route.
Schools registered their interest in taking part using a Google Form, and 179 schools did so.
All volunteer schools were telephoned to check that they met the inclusion criteria for the project and to outline the key
commitments for the school.
Schools were eligible to participate in the study if:•they had not used a systematic mindsets programme with their Year 6 cohort before; and10

Changing Mindsets
Evaluation Report•they were able to attend one of the training dates provided by Portsmouth University.

The delivery team agreed to prioritise schools with a high number of free school meal (FSM) pupils, or schools where
there was a significant gap between FSM and non-FSM pupils, as these were considered the schools that needed themost support. However, it was not necessary to apply these criteria as the intervention was never oversubscribed.
One private school expressed an interest but was not included in the trial due to the potential impact of additional
heterogeneity on the robustness of impact estimates.
Schools that were interested in taking part and that met the eligibility criteria were emailed a ‘welcome pack’ which
included a description of the study in detail, a permission form for the headteacher to complete, an information letterto parents giving them the option to withdraw their child from the study, a form to request the children’s UPNs and the‘school questionnaire’ (that included a Mindset scale and adapted MSLQ questions) plus instructions.
Prior to being randomised, schools needed to provide the following:••••

headteacher consent for their school to part in the trial (loco parentis);confirmation that the information letters to parents had been sent out;
a pupil data form including UPNs for all Year 6s; andcompleted pre-test non-cognitive measures.

Schools entered the randomisation phase when they returned the headteacher consent form and were revealed their
allocation group only after returning all the remaining material.
All pupils attending Year 6 in selected schools were eligible for the treatment. Parent/guardian consent forparticipation in the research analysis and linking between pupil non-cognitive tests and administrative data in the
National Pupil Database (NPD) was sought after randomisation to intervention or control group. All consent forms areavailable in Appendix D.
The protocol was not changed during the implementation. Six schools dropped out from the study after the
randomisation and did not provide any pre-treatment information and therefore, in accordance with the SAP, were notincluded in the analysis.

Data
Data used in this evaluation includes administrative records from the National Pupil Database and information on pupils’
attitudes towards school and learning collected before and after the treatment by the project team through an adapted
version of the MSLQ.

Outcome measures
The primary outcomes are reading and numeracy as measured by the KS2 national assessment tests in reading, GPS,and maths. KS2 tests are taken by all pupils at the end of Year 6 (age 10–11) and are externally graded. KS1 teacher
assessments in reading, writing, and maths are jointly used to define a comprehensive set of dummies for priorattainment. In KS1, teachers assess pupils against level of achievement (W, 1, 2C, 2B, 2A, 3, 4). To control for priorattainment, we convert these levels into points by using the conversion tables provided by the DfE (DfE, 2015) and wethen calculate the average KS1 attainment based on non-missing KS1 assessments. As in Crawford et al. (2016), we
define the following categories: 11–14.99 points, ‘did not achieve grade level’; 15–16.99 points, ‘achieved grade level’;
more than 17 points, ‘achieved above grade level’; or ‘all missing’. These categories are then used to create a set ofdummy variables to control for prior attainment in the empirical analysis. In addition, by including in the analysis adummy for the category of ‘all KS1 assessments missing’ we do not exclude all the pupils for whom we have no
information about prior attainment. KS1 assessments for the pupils in the study were held in the academic year
2012/2013 when pupils in the trial were in Year 2 (age 6–7). Specifically, in the regression analysis we use KS2 scaled
point scores, that range from 80 to 120, as primary outcomes and dummies for categories of average scores in KS1 ascontrol for prior attainment.
11Changing Mindsets
Evaluation Report

KS2 test scores are robust measures of achievement and were chosen as primary outcomes for two reasons: they areblind assessments produced by external markers, therefore they cannot be affected by the treatment status of the pupils
and they are finely scored and therefore they introduce high granularity in the data. Dummies for categories of average
KS1 attainment are a more comprehensive measure of prior attainment as they contain information from three different
assessments and they have the advantage of reducing the impact of missing KS1 assessments.
The secondary outcome variables are four measures of non-cognitive skills, captured through four subscales of the
MSLQ. All pupils in the study were administered a ‘school questionnaire’ before and after the intervention. This
questionnaire included three items designed to measure pupils’ mindsets and 31 MSLQ items. The aim of the first three
items was to measure whether pupils had a fixed mindset and believed their intelligence to be a fixed trait, or whether
they had a growth mindset, that is, believed that intelligence could grow with effort. The pupils’ mindsets measurecreated with these questions has been used and validated in several studies in experimental psychology (see for
instance Yaeger et al., 2016). The standard 31 MSLQ questions were adapted by Portsmouth University to make them
relevant to the age and British location of participants. Schools administered the pre-trial questionnaires to pupils when
they were in Year 5 (March–July 2016). The post-trial questionnaires were administered by schools after KS2 national
assessment (May–July 2017). Specifically, the secondary outcome measures focus on four subscales of the MSLQ:
self-efficacy, intrinsic value, test anxiety, and self-regulation. The intrinsic value scale measures to what extent pupils’
participation in an academic task is an aim in itself, rather than participation being a means to an aim; the self-efficacy
scale measures how pupils judge their ability to accomplish a task as well as their confidence in their skills to performthat task; the test anxiety scale measures students’ concern about taking an exam; finally the self-regulation scalemeasures pupils’ ability to plan, monitor, and regulate their cognitive strategies to succeed in an academic task. At thetime of writing the protocol, only two subscales of the MSLQ were included, namely self-efficacy and self-regulation. At
the SAP stage, intrinsic value and test anxiety were also included among the secondary outcomes.
The three pupil mindset measures are scored from 1 to 6, with 1 indicating ‘strongly disagree’ and 6 indicating ‘strongly
agree’. All items on the adapted version of the MSLQ are scored from 1 to 7, with 1 indicating ‘strongly disagree’ and 7
indicating ‘strongly agree’.7 Subscales (intrinsic value, self-efficacy, test anxiety, and self-regulation) are then calculated
as a mean score. The full set of pupil mindset questions, MSLQ items, and a description of the subscales are includedin Appendix E.
For secondary outcomes, the MSLQ was chosen as it is a widely-used tool developed for assessing motivation andlearning strategies (Duncan and McKeachie, 2005). The ‘junior’ version of the questionnaire had been already used in
the U.K. schooling system to assess the effectiveness of an intervention to increase engagement in science in Scottish
schools (Moote, Williams and Sproule, 2011; 2013).

Sample size
Our initial calculations of the sample size targeted the participation of 100 schools for a minimum detectable effect sizeof 0.20 based on the following assumptions:
•
•
•
•
•
•

proportion of schools assigned to treatment, 0.50;
40 children per cluster (average cohort size assumption pre-randomisation);
0.05 significance level;
0.8 power;
0.25 intra-cluster correlation; and0.5 cluster variance predicted by KS1 attainment.

The ICC parameter was a deliberately conservative assumption of around twice the size of those published by Allen etal. (2018). A conservative assumption was chosen to reflect the fact that we would be conducting an FSM subgroup
analysis.

7 Three items on the self-regulation scale need reversing before totalling the sub-scale score and the total score.

12Changing Mindsets
Evaluation Report

The minimal detectable effect size (MDES) decreases to 0.19 when considering the achieved sample at randomisationreflecting the involvement of 101 schools rather than 100 assumed at the protocol and a larger cluster size of 48 withall other assumptions unchanged (see Table 4).
The MDES at analysis decreases to 0.11 when considering the achieved sample largely because the actual intra-cluster
correlation was 0.09 rather than 0.25:
•
•
•
•
•
•

proportion of schools assigned to treatment, 0.50;
47 children per cluster;
0.05 significance level;
0.8 power;
0.09 intra-cluster correlation; and0.56 proportion of cluster variance predicted by covariates.

Assuming the FSM subgroup is 15.7% of the total size of the sample (calculated from DFE data for 2014/2015) andignoring that it may be higher if recruited schools are in more disadvantaged areas) and maintaining all otherassumptions (which is likely to be a conservative approach, given lower levels of within-group variation in this subgroup),
this gives eight pupils per cluster in the subgroup analysis and an estimated MDES for this group of 0.27 standard
deviations at protocol stage. At randomisation this falls to 0.23, largely due to larger cluster sizes (17 rather than 8)
since the proportion of FSM pupils in recruited schools is 35.7%, and falls further to 0.15 at analysis stage, again
reflecting a lower observed intra-cluster correlation (0.13 as opposed to 0.25).

Randomisation
An independent NIESR consultant (Dr Richard Dorsett) used Stata 13 to randomise schools 50:50 to the intervention
group or to the control group after school recruitment and consent had been received. Randomisation was within blocks
defined according to tertiles of school-level KS1 performance within each of the five locations. This resulted in 15 blocks.
Each school was assigned a random number between 0 and 1 from a uniform distribution. Schools were then sortedby block and random number. The first school in the list was randomly assigned to be in the treatment or control group.
Each subsequent school in the list was assigned to be in the opposite group of the previous schools, thus assuring anequal distribution allocation. The Stata code for this randomisation is given in Appendix F.
The number of schools in each block is presented in Table 4.
Table 4: Number of schools in randomisation blocks
BlocksNo. ofschools

Midlands—low achievement

12

Midlands—medium achievement

8

Midlands—high achievement

8

North East—low achievement

7

North East—medium achievement

10North East—high achievement

9North West—low achievement

4

North West—medium achievement

6

North West—high achievement

4South East—low achievement

10

South East—medium achievement

7

South East—high achievement

9South West—low achievement

2South West—medium achievement

2South West—high achievement

313Changing Mindsets
Evaluation Report

School data was obtained from the Performance table data for the academic year 2014/2015 downloaded from gov.uk.
The final randomisation consisted of 50 schools in the treatment group and 51 schools in the control group.

Statistical analysis
The two types of schools included in the trial are:a) intervention schools that deliver Changing Mindsets; andb) control schools that will receive the intervention one year later.
The estimates of the impact of the intervention on primary and secondary outcomes are obtained as the difference
between (a) and (b) by using an ‘intention to treat’ approach, that is, all pupils in the trial are included in the final analysis
whenever possible.
Statistical significance is assessed using two-sided tests at the 5% level. Estimates of effect with 95% confidence
intervals (CIs) and p-values are provided.
The original intention was to conduct an analysis into the effect of non-compliance. For this analysis the University ofPortsmouth had to collect information on the amount of tuition received by pupils (number of sessions and ratio of tutor
to pupils in their group). Due to a very low response rate of schools it was decided that the initially-planned analysis into
the effect of non-compliance was not possible due to concerns of data quality.
Primary outcomes analysis
The difference in reading, GPS, and maths attainment between identified pupils in the intervention group and those in
the control group is estimated using multilevel models in order to take into account clustering of pupils within schools,
because pupils’ scores are likely to be correlated within schools. We estimated mixed models using restricted maximum
likelihood estimation with school effects as random variables and a set of binary indicators for the stratification groups
as fixed effects in Stata. We control for KS1 achievement by using binary indicators for categories of average scores in
KS1 tests: 11–14.99 points, ‘did not achieve grade level’; 15–16.99 points, ‘achieved grade level’; more than 17 points,‘achieved above grade level’; or ‘missing’. The baseline category is less than 11.
Estimates are obtained separately for each primary outcome. In addition, models including only the treatment
assignment and KS1 scores are estimated as required by the EEF and results presented in Appendix G.
The estimated equation is:
𝑦 = 𝑋𝛽 + 𝑍µ + 𝜖
Where:
y = vector of outcome scores
X = covariate matrix (KS1 scores in ‘simplest’ model and this plus dummies for stratification groups in the ‘precise’model)
Z = design matrix identifying which school or cluster an individual attended
 = vector of school random effects
β, = fixed effect parameters
εij = residual error term for j-th member of cluster (school) iwith the covariance structure given by ∑, where:𝐼∑= (σa2 + σe2)[ ⋮
0

⋯ 0
⋱ ⋮]
⋯ 𝐼

Where σa2 is a measure of school level variation; σe2 is a measure of student level variation and I is given by:
14

Changing Mindsets
Evaluation Report

1
𝜌
I=[
⋮
𝜌

𝜌 ⋯
1 ⋱
⋱ ⋱
⋯ 𝜌

𝜌
⋮
]
𝜌
1

And 𝜌 is the intra-school correlation coefficient:𝜌=

𝜎𝑎2
𝜎𝑎2 + 𝜎𝑒2

The fixed effect parameters and variance components are estimated by restricted maximum likelihood estimation using
the Stata command:
mixed KS2 KS1 TREAT i.block || ks2_urn: reml - for the precise model and:mixed KS2 KS1 TREAT || ks2_urn: reml - for the simplest model
Because the analysis includes multiple primary outcomes there is a higher chance of detecting spurious results due torandom sampling error, however, the SAP did not specify that any adjustments for multiple testing would be applied(we return to this issue in the Impact Evaluation section).
Analysis was conducted in Stata v15 using the principles of intention to treat and only including prior attainment,
stratification block dummies, and the allocation dummy as covariates. This is in line with the most recent EEF statisticalanalysis guidance.8
Non-compliance with the intervention
The original intention was for the University of Portsmouth to collect information on the amount of tuition received bypupils (number of sessions and ratio of tutor to pupils in their group). The University asked schools to provide data on
the number of pupils absent from each lesson and the extent to which they adapted the material to fit their own style.
Response rates were lower than expected and decreased over time, from 49 schools in session one to nine in session
eight. An explanation for this low response rate could simply be that schools were busy. The delivery team asked
teachers to complete the fidelity survey each week (and sent email reminders each time). This weekly task was probably
considered by teachers as an additional chore and this could explain the worsening response rate. The delivery team
was also mindful of striking the right balance: they did not want schools to feel pressured so much that they would be
inclined to drop out from the project altogether. Given the low response rate it was not possible to conduct the initiallyplanned analysis into the effect of non-compliance due to concerns of data quality.
Secondary outcomes analysis
To assess the changes on the secondary variables we used the same econometric approach as above, but because
the questions were asked pre- and post-intervention and the scores are non-cognitive, we use the post-intervention
non-cognitive score for each subscale as outcome variables and control by the pre-intervention continuous noncognitive score rather than KS1 attainment. We believe this approach is justified by the EEF guidance that states that‘pre-tests should only be conducted if they are needed to evidence the causal model for an intervention’—which is theintention of the non-cognitive test analysis. As with the analysis of the primary outcome variable, we use the simplest
and precise specification and compare results.
Subgroup analysis
We carried out a separate analysis for the FSM subsample following the same empirical approach as that used forprimary outcomes. The p-value from an interaction effect model (by including FSM as a main effect and an interaction

8https://educationendowmentfoundation.org.uk/public/files/Grantee_guide_and_EEF_policies/Evaluation/Writing_a_Protocol_or_S

AP/EEF_statistical_analysis_guidance_2018.pdf

15Changing Mindsets
Evaluation Reportwith treatment status) is also reported, but the subgroup analysis represents the main subgroup result as we believethat is easier for the teaching community to interpret.
Results from additional subgroup analyses that were not pre-specified in the evaluation protocol and the SAP and, assuch, were purely exploratory, are reported in Appendix H. In these additional analyses we considered subgroups ofgirls, boys, and pupils with ‘fixed mindset’.9 In particular we focused on girls and pupils with fixed mindset as they are
the subgroups that showed promising results in previous trials based on U.S. and Norwegian data: girls are more likely
to be affected by stereotypes threats in mathematics and some studies have shown that exposing students to a growing
mindset approach would make their performances less vulnerable to stereotype threats (see for instance Aronson etal., 2002); pupils with a baseline fixed mindset are the ones with more scope to change their beliefs about theirintelligence and, therefore, the ones who could benefit more from the intervention, as found by Bettinger et al. (2018).
Effect size calculation
Effect sizes are calculated based on the adjusted mean difference between the intervention and control group(controlling for prior attainment) and the variance components produced by Stata 15 using the syntax described above.
The effect sizes and 95% confidence intervals are calculated using equations (19) and (20) given in Hedges (2007) for
cluster randomised trials estimated via multilevel models and allowing for unequal cluster sizes.
Missing data analysis
In accordance with the SAP, we describe and summarise the extent of missing data in the primary and secondary
outcomes. In addition, we describe the reasons for missing data. First, we assess whether the missing data is missing
at random (MAR). We use the standard approach where we create an indicator variable for each variable in theimpact model specifying whether the data is missing or not and use logistic regression to test whether the missing
status can be predicted from the following variables: all variables in the precise model plus school average KS1(continuous variable as opposed to terciles), gender, ethnicity, and FSM eligibility. Where predictability is confirmed
we proceed with multiple imputation.

Implementation and process evaluation (IPE)The aims of the IPE were to understand how the Changing Mindsets intervention programme was implemented anddelivered in treatment schools, and to what extent this differed (if at all) from the intended delivery model. It also aimed
to shed light on implementers’ as well as pupils’ responsiveness to the intervention, and the perceived impact the
programme had on those involved. Finally, the process evaluation sought to gather information on the activity of schools
within the control group over the course of the trial period, and establish how far this differed from the intervention
programme.
The IPE used the following methods:•

fieldwork visits to six treatment schools conducted from December 2016 to February 2017, which included:▪

semi-structured interviews with nine Year 6 teachers, three headteachers or deputy
headteachers, and four teachers from other Year groups to understand experiences of
implementation and perceived outcomes;▪

three focus groups with eight to ten Year 6 pupils receiving the intervention to understandpupils’ responses to the programme and views on its impact on their learning; and9 Pupils with a fixed mindset are defined as pupils in the highest tercile of the pre-trial mindset measure obtained as the average ofthe three mindset items included in the ‘school questionnaire’. These items are scored 1–6 with with 1 indicating ‘strongly disagree’
and 6 indicating ‘strongly agree’. The three items are the following: ‘you have a certain amount of intelligence, and you really can’t
do much to change it’; ‘your intelligence is something about you that you can’t change very much’; and ‘you can learn new things but
you can’t really change your basic intelligence’. The higher the mindset measure, the more fixed a pupil mindset is.

16

Changing Mindsets
Evaluation Report▪

three lesson observations of the final Changing Mindsets session to understand
implementation and, to some extent, fidelity;•

an end-of-project survey, administered to treatment schools from January to February 2017, to assess fidelity
and experiences of delivery;•

a separate fidelity survey, administered eight times to treatment schools over the course of the intervention
from September 2016 to February 2017; and•a control group survey, administered in June 2017, to gain an understanding of whether control schools werealso using Growth Mindset in their teaching and learning that might weaken the impact of the intervention.

All activities were carried out by NIESR with the exception of the fidelity and control group surveys, which were designed
and administered by Portsmouth University to collect information on each module. The control group survey took theform of a paper questionnaire and was completed by attendees at the training days that were provided after programme
implementation had finished at the end of academic year 2016/2017. This was forwarded to NIESR and incorporated
into the external evaluation. All other surveys were administered and completed by respondents online.
It should be noted that the views and experiences presented here are not necessarily representative of all participating
schools. The response rate to the fidelity survey decreased over the course of the intervention (from 49 to 9
respondents), while the end-of-project survey was completed by just over half of treatment schools (26 out of 50).
However, steps were taken to ensure that the treatment schools visited as part of fieldwork included a range by locationand intake which might be reflected in experiences of the project.
Schools were selected to include a range of key characteristics: Ofsted rating, proportion of FSM pupils, geographicallocation, and whether located in urban or rural settings. The project team provided NIESR with a full list of treatment
schools from which we drew up a profile according to characteristics of interest. These were Ofsted rating, proportionof FSM pupils, and area of the country. Using these main criteria, we aimed to achieve a range of schools within thesample of six. In two cases schools were not responsive to our request for participation and alternatives wereapproached from a reserve list.
The response rate to the control group survey was good in that it was completed by 37 out of 51 schools in this treatment
allocation.

Costs
The process evaluation collected additional data on costs by asking schools whether additional costs were incurred in
the delivery of the programme. Some teachers said they had sourced additional materials for teaching, for example
videos, but it was generally agreed that the project resources were both sufficient and very good. The intervention teamprovided information regarding the actual total cost to the school in order to implement the intervention.

Timeline
The timeline of the trial is summarised in Table 5. There were no changes to the timeline agreed at the protocol stage.

17

Changing Mindsets
Evaluation Report
Table 5: Timeline
Date

Activity

From Jan 2016

Schools approachedJan–May 2016

Schools recruited and agree to participate

May/July 2016

Pre-trial Mindset and MSLQ questionnaires

Jun 2016

Schools randomly allocated to treatment or control group

Sep 2016

Intervention group attended one-day training event

Sep–Dec 2016

Intervention delivered in schools by intervention group

Sep 2016–Feb 2017

Fidelity survey to treatment schools during implementation period

Dec 2016–Feb 2017

Fieldwork visits to six treatment schools

Jan–Feb 2017

End-of-project survey to treatment schools

Jun 2017

Control group survey

May 2017

Pupils sit Key Stage 2 exams

March/July 2017

Post-trial Mindset and MSLQ questionnaires

Jun 2017

Control group attended one-day training event

Autumn term 2017

Intervention delivered by control schools18Changing Mindsets
Evaluation Report

Impact evaluation
Participants
A flow of participants through the evaluation is presented in Figure 1.
The sample of interest is students in intervention and control schools in Year 6 in the academic year 2016/2017. These
pupils were assessed at the end of KS2 in May 2017 and were administered post-trial questionnaires between May andJuly 2017.
The precise number of pupils initially randomised into the trial is 5018. For these pupils the project team obtained theunique pupil numbers (UPNs) from the 101 schools participating in the study. Eleven pupils were withdrawn from thetrial by their parents and 132 UPNs could not be matched to administrative records of pupils in those schools. Of theremaining 4875 pupils randomised in the trial, 291 are lost as six schools withdrew from the study after the
randomisation (85 in two treatment schools and 206 in four control schools). The final number of pupils that could be
included in the primary analysis is 4584. There were some missing observations in the primary outcomes, specifically
147 in reading scores, 136 in GPS scores, and 130 in maths scores. However, to minimize potential bias from missingnot-at-random we allow the sample to vary by outcome instead of imposing a common sample of pupils where there isno missing data for any of the variables used in the empirical analysis.
In the secondary outcome analyses the samples were restricted to schools that sent back their pupils’ MSLQ responses
and to pupils that responded to the baseline and follow-up questionnaires; 79 schools (40 in the intervention group and
39 in the control group) provided the pre and post-trial responses of 2917 pupils. This low figure could be explained by
the lack of engagement of schools towards the end of the project, which spanned a long period of time. The highproportion of missing observations in the secondary outcomes can potentially affect the robustness of the results of thesecondary analysis, and, as a consequence, their external validity.
Table 6 shows the minimum detectable effect size (MDES) for the overall and FSM samples at the different stages of
the trial. At the time of preparing the trial protocol the MDES was 0.20 for the overall sample and 0.27 for the FSM
sample. At randomisation this was similar for the overall sample (0.19), but reduced from 0.27 to 0.24 for the FSM
sample reflecting a larger number of FSM pupils in trial schools than had been assumed when writing the protocol. For
the final analysis sample, the MDES was much lower: 0.11 (overall sample) and 0.15 (FSM sample) as the protocol
assumption for the intra-cluster correlations proved too conservative (assumed to be 0.25 for both samples but turned
out to be 0.09 and 0.13 for the overall and FSM samples respectively).

19Changing Mindsets
Evaluation Report
Figure 1: Participant flow diagram

Approached (school n=168)

Recruitment

Did not agree to participate(school n=67)
Agreed to participate(school n=101)
Excluded (school n=n/a)

Follow-up

Allocation

Randomised(school n=101; pupils=5018)

Allocated tointervention(school n=50; pupil
n=2502)

Lost to follow
up in
withdrawn
schools

Lost to follow
up in
withdrawn
schools

UPN not matched=76
Missing data:
KS2 Maths=62
KS2 reading=72
KS2 GPS=69

Pupilswithdrawn
from the trial
5 pupils

6 pupils
85 pupils in 2
withdrawn
schools

Not analysed

Analysis

Pupilswithdrawn
from the trial

Allocated to control(school n=51; pupil
n=2516)206 pupils in 4
withdrawn
schools.

Analysed
KS2 Maths=2273
KS2 reading=2263
KS2 GPS=2266

Not analysed
UPN not
matched=56

Analysed
KS2 maths=2181
KS2 reading=2174
KS2 GPS=2182

Missing data:
KS2 Maths=68
KS2 reading=75
KS2 GPS=67

20

Changing Mindsets
Evaluation Report
Table 6: Minimum detectable effect size at different stages
ProtocolRandomisation

Analysis

Overall

FSM

Overall

FSM

Overall

FSM

0.20

0.27

0.19

0.23

0.11

0.15

level 1
(pupil)

0.50

0.50

0.50

0.50

0.68

0.66

level 3
(school)

0.50

0.50

0.50

0.50

0.56

0.6

level 3
(school)

0.25

0.25

0.25

0.25

0.09

0.13

Alpha

0.05

0.05

0.05

0.05

0.05

0.05

Power

0.8

0.8

0.8

0.8

0.8

0.8

One-sided or two-sided?

2

2

2

2

2

2

Average cluster size

40

8

48

17

47

17

intervention

50

50

50

50

48

48

control

50

50

51

51

47

47

total

100

100

101

101

95

95

intervention

2000

314

2502

832

2335

782

control

2000

314

2516

857

2249

797

total

4000

628

5018

1689

4584

1579

MDES
Pre-test/
post-test
correlations

Intracluster
correlations(ICCs)

Number ofschools

Number of
pupilsAttrition
As previously outlined, the precise number of pupils initially randomised into the trial was 5018. For these pupils, theproject team obtained the unique pupil numbers (UPNs) from the 101 schools participating in the study. Eleven pupils
were withdrawn from the trial by their parents and 132 UPNs could not be matched to administrative records of pupils
in those schools. Of the remaining 4875 pupils, 291 are lost as six schools withdrew from the study after the
randomisation (85 in two treatment schools and 206 in four control schools). The final number of pupils that could be
included in the primary analysis is 4584. Table 6 presents the total rate of attrition for all the pre-test and post-test
outcomes included in the analysis. The total rate of attrition for primary outcomes is 11.58% for reading, 11.24% formaths, and 11.36% for GPS. As these rates of attrition are greater than the 5% threshold specified in the SAP, weinvestigate further whether the missing data is at random (MAR) or whether there is a pattern of missingness that canbe predicted by some available variables by using a logistic analysis. As Table 7 shows, attrition is a much bigger
problem for secondary outcomes, with the percentage of missing observations being above 40% for all four subscalesof the MSLQ. As for primary outcomes, patterns of missingness in secondary outcomes are investigated using a logistic
analysis.

21

Changing Mindsets
Evaluation Report
Table 7: Rate of attrition
Variable

Treated (in the analysis)

Controls (in the analysis)

Attrition rate

Maths score at KS2

2273

2181

11.24

Reading score at KS2

2263

2174

11.58

GPS score at KS2

2266

2182

11.36

Intrinsic value measure (post-treatment)

1508

1409

41.87

Self-efficacy measure (post-treatment)

1507

1409

41.89

Self-regulation measure (post-treatment)

1501

1398

42.23

Anxiety measure (post-treatment)

1507

1409

41.89

Pupil and school characteristics
The University of Portsmouth approached 168 schools; of these, 67 did not accept to participate in the study. Finally,
101 schools were included in the randomisation: 50 were allocated to the intervention group and 51 were allocated tothe control group. After the randomisation, six schools (two treatment, four control) withdrew from the study and were
not included in the final analysis.
Table 8 reports the main school characteristics for the schools that participated in the randomisation and their pupils bytreatment status. The most relevant difference at school level is the proportion of academy schools: 28% in theintervention group and 17.65% in the control group. Schools in the intervention group have also more pupils on roll thanschools in the control group (a mean of 362.52 in treatment schools and of 341.86 in control schools). Another small
difference concerns the proportion of EAL pupils in the schools—17.46% in the intervention group and 15.49% in thecontrol group. There are no other relevant differences between intervention and control schools in school-level
characteristics.
There are also few, and small, differences between the two groups of schools in pupil-level characteristics. For example,
there is a marginally higher proportion of white pupils (73.26%) and a lower proportion of black pupils (4.05%) in theintervention compared to the control group (respectively 69.78% and 6.84%). Although the difference in the proportionof black pupils is small in terms of percentage points it is significant in percentage terms given the low baseline
proportion. All the other characteristics and pupil pre-intervention outcomes are, on average, similar in the two groups,
as shown by the standardised differences. It is important to highlight that there is a good balance in all the pre-test
outcomes as all the differences found are very small and likely to be attributed to chance. In particular, the preintervention mindset measure, which is a measure of fixed mindset captured through an additional set of questions
administered to the pupils with the MLSQ questionnaire, is very similar, on average, in the two groups.
Table 8: Baseline comparison as randomised
Variable
School level (categorical)

Intervention groupn/N(missing)

Percentage

Control groupn/N(missing)

Percentage

Religiously affiliated12/50 (0)24.00

12/51 (0)23.53

Academy14/50 (0)28.00

9/51 (0)17.65

Community school

22/50 (0)44.00

29/51 (0)56.86

Voluntary or Foundation school

7/50 (0)14.00

3/51 (0)5.88Voluntary aided school

7/50 (0)14.00

9/51 (0)17.65

OFSTED: Outstanding

8/50 (0)16.00

9/50 (1)18.00

OFSTED: Good

37/50 (0)74.00

36/50 (1)70.59

OFSTED: Satisfactory

4/50 (0)8.00

5/50 (1)10.00

OFSTED: Inadequate

1/50 (0)2.00

1/50 (1)2.00

22

Changing Mindsets
Evaluation Report
VariableIntervention group

School level (continuous)n(missing)

Mean

Control groupn(missing)

Mean

Number of pupils

50 (0)

362.5251 (0)

341.86% of Free School Meal

50 (0)

14.54

51 (0)

15.56% SEN with support

50 (0)

12.93

51 (0)

12.15% SEN with statement

50 (0)

1.24

51 (0)

1.45% English Additional Language

50 (0)

17.46

51 (0)

15.49

VariableIntervention group

Pupil level (categorical)

Control groupn/N(missing)

Percentage

n/N(missing)

Percentage

Female1194/2420 (0)49.34

1190/2455 (0)48.47Ever FSM

832/2420 (0)34.38

857/2455 (0)34.91

Ethnicity: Asian359/2420 (0)14.83

397/2455 (0)16.17

Ethnicity: Black

98/2420 (0)4.05

168/2455 (0)6.84

Ethnicity: Mixed

148/2420 (0)6.12

108/2455 (0)4.40

Ethnicity: White

1773/2420 (0)73.26

1713/2455 (0)69.78

VariableIntervention group

Control groupn(missing)Meann(missing)

Mean

Std. diff.

Reading points at KS1

2288 (132)16.38

2324 (131)

16.37

0

Writing points at KS1

2288 (132)14.992324 (131)

15.21

-0.06

Mathematics points at KS1

2288 (132)16.18

2324 (131)

16.23

-0.01

Mindset measure2030 (390)

3.181814 (641)

3.15

0.02

Intrinsic value measure2038 (382)5.60

1827 (628)

5.66

-0.06

Self-efficacy measure2037 (383)

5.211827 (628)

5.28

-0.06

Anxiety measure2037 (383)3.801827 (628)

3.72

0.05

Self-regulation measure2034 (386)4.73

1820 (635)

4.82

-0.08

Pupil level (continuous)

Table 9 presents the baseline comparison at analysis and therefore does not include the six schools that dropped out
after the randomisation. This table is useful as it allows us to explore whether attrition was differential between thegroups. The baseline comparison at the analysis is very similar to the one at randomisation: the differences between
treatment and control group are very small and are likely to be attributed to chance. As the balance does not change
when attrition is taken into account in the baseline comparison, it is confirmed that differential attrition is not a threat tothe validity of the trial.
Table 9: Baseline comparison at analysis
Variable
School level (categorical)

Intervention groupn/N(missing)

Percentage

Control groupn/N(missing)

Percentage

Religiously affiliated

11/48 (0)22.92

12/47 (0)25.53

Academy

14/48 (0)29.17

9/47 (0)19.15

Community School

21/48 (0)43.75

25/47 (0)53.19

Voluntary or Foundation school

7/48 (0)14.58

3/47 (0)6.38

Voluntary aided school

6/48 (0)12.50

9/47 (0)19.15

OFSTED: Outstanding

7/48 (0)14.58

8/47 (0)17.02

OFSTED: Good

36/48 (0)75.00

34/47 (0)72.34

OFSTED: Satisfactory

4/48 (0)8.33

4/47 (0)8.51

OFSTED: Inadequate

1/48 (0)2.08

1/47 (0)2.1323Changing Mindsets
Evaluation Report
Variable
School level (continuous)Intervention groupn(missing)

Mean

Control groupn(missing)

Mean

Number of pupils

48 (0)

361.9647 (0)% of Free School Meal

48 (0)

14.87

47 (0)

16.6% SEN with support

48 (0)

13.06

47 (0)

12.59% SEN with statement

48 (0)

1.22

47 (0)

1.51% English Additional Language

48 (0)

17.51

47 (0)

15.58

Variable
Pupil level (categorical)

Intervention group

340.57

Control groupn/N(missing)

Percentage

n/N(missing)

Percentage

Female1150/2335 (0)49.25

1079/2249 (0)47.98Ever FSM

809/2335 (0)34.65

826/2249 (0)36.73

Ethnicity: Asian351/2335 (0)15.03

371/2249 (0)16.50

Ethnicity: Black

93/2335 (0)3.98

163/2249 (0)7.25

Ethnicity: Mixed

142/2335 (0)6.08

100/2249 (0)4.45

Ethnicity: White

1708/2335 (0)73.15

1546/2249 (0)68.74

VariableIntervention group

Control groupn(missing)Meann(missing)

Mean

Std. diff.

Reading points at KS1

2209 (126)16.39

2122 (127)

16.26

0.03

Writing points at KS1

2209 (126)15.00

2122 (127)

15.12

-0.03

Mathematics points at KS1

2209 (126)16.21

2122 (127)

16.16

0.01

Minset measure2030 (305)

3.18

1753 (496)

3.14

0.03

Intrinsic value measure2038 (297)

5.60

1766 (483)

5.68

-0.08

Self-efficacy measure2037 (298)

5.21

1766 (483)

5.29

-0.07

Anxiety measure2037 (298)

3.80

1766 (483)

3.72

0.05

Self-regulation measure2034 (301)

4.731759 (490)

4.83

-0.09

Pupil level (continuous)

In summary, there is a good balance between the treated and the control group in school and pupil characteristics andthe small differences found seem fairly consistent with the amount of imbalance that would occur due to chance.

Outcomes and analysis
This evaluation assesses whether supporting schools in encouraging ‘growth mindset’ in their pupils—that is, the belief
that intelligence can be developed rather than being a fixed trait—has an effect on pupils’ numeracy and literacy skills
and on a subset of non-cognitive skills (self-efficacy, intrinsic value, test anxiety, and self-regulation).
Analyses of primary and secondary outcomes are undertaken on an ‘intention to treat’ basis: all pupils for which there
is available data are included in the estimation of the effect size, regardless of whether the school completed theintervention or implemented it as agreed during the teacher training day.
We present now the findings of the statistical analyses of primary and secondary outcomes. The code used to calculate
the effect size in academic outcomes is provided in Appendix H.

Academic outcomes
The academic outcomes included in the analysis are KS2 scores in maths, reading, and GPS. Means and standard
deviations of post-test scores are very similar between the treatment and control group. The mean (and standard
deviation) of scores of KS2 scores in maths, reading, and GPS is, respectively, 103.82 (sd 7.42), 103.82 (sd 8.06), and105.87 (sd 7.51) for the treated group; and 104.06 (sd 7.06), 103.70 (sd 8.37), and 105.76 (sd 7.38) for the control
group. Histograms for the post-test scores are presented in Appendix I.
24

Changing Mindsets
Evaluation Report

We estimate three multilevel models to assess the effect of the intervention on KS2 academic outcomes. Table 10
reports the effect size for KS2 maths, reading, and GPS; Table 11 presents all parameters used for the calculations inTable 10. The intra-cluster correlations (ICC) were respectively 0.11, 0.07, and 0.12. The effect size for the impact of
the intervention on each of the three academic outcomes is zero. The confidence intervals of the effect of theintervention on KS2 maths, reading, and GPS were, respectively, -0.04 to 0.01, -0.02 to 0.02, and -0.03 to 0.03.
Therefore, there is no evidence of an impact of the Changing Mindsets intervention on literacy or numeracy skills ofYear 6 students.
As the effect is zero for all primary outcomes, we do not proceed with any multiple comparison adjustments in thereporting of the results.
Table 10: Primary analysis
Raw meansIntervention group

Control groupMean(95% CI)n(missing)

KS2 maths

2273
(62)

103.82(103.51;104.13)

2181
(68)

104.074454(103.77; 104.36) (2273; 2181)-0.01(-0.04; 0.01)

0.37

KS2 reading

2263
(72)

103.82(103.49;104.16)

2174
(75)

103.704437(103.35; 104.05) (2263; 2174)-0.00(-0.02; 0.02)

0.72

KS2 GPS

2266(69)

105.87(105.56;106.18)

2182
(67)

105.764448(105.45; 106.07) (2266; 2182)-0.00(-0.03; 0.03)

0.90

OutcomeMean(95% CI)n in model(intervention;
control)n(missing)

Effect size
(95% CI) p-value

Table 11: Effect size estimation in primary analysis
Intervention group

Control group

Outcome

Unadjusted
differences inmeans

Adjusted
differences inmeans

n (missing)

Variance ofoutcomen(missing)

Variance ofoutcomePooledvariance

KS2 maths

-0.25

-0.38

2273 (62)55.1

2181 (68)

49.9

52.56

KS2 reading

0.12

-0.13

2263 (72)64.98

2174 (75)

70.13

67.49

KS2 GPS

0.11

02266 (69)56.43

2182 (67)

54.56

55.5

Non-cognitive outcomes
The non-cognitive traits included in the analysis as secondary outcomes are the following subscales of the MLSQ:
intrinsic value, self-efficacy, test anxiety, and self-regulation. Means and standard deviations of the post-test subscales
are very similar between the treatment and control group. The mean (and standard deviation) of intrinsic value, selfefficacy, test anxiety, and self-regulation is, respectively, 5.54 (sd 1.01), 5.11 (sd 1.07), 3.44 (sd 1.74), and 4.76 (sd1.07) for the treated group; and 5.49 (sd 1.02), 5.17 (sd 1.07), 3.50 (sd 1.73), and 4.72 (sd 1.08) for the control group.
Histograms for the post-test scores are presented in Appendix I.
As in the primary outcome analyses, we estimate four multilevel models to assess the effect of the intervention on noncognitive outcomes. The estimates from a mixed model for the four subscales of the adapted MSLQ are used to
calculate the effect sizes of the intervention on non-cognitive outcomes. Table 12 reports the effect size for each
subscale, and Table 13 presents all parameters used for the calculations in Table 12. The results show that the effectsize for the impact of the intervention is not statistically different from zero for any of the four subscales. The effect sizefor intrinsic value is 0.09 with confidence interval -0.10 to 0.28; for self-efficacy, -0.05 (CI: -0.21; 0.10); for test anxiety,-0.01 (CI: -0.09; 0.06); and self-regulation, 0.05 (CI: -0.11; 0.21).

25

Changing Mindsets
Evaluation Report

Therefore, there is no evidence of an impact of the Changing Mindsets intervention on the tested non-cognitive
outcomes of Year 6 students.
Table 12: Secondary analysis
Raw meansIntervention group

Control groupn(missing)

Mean(95% CI)n(missing)

Mean(95% CI)n in model(intervention;
control)

Effect size
(95% CI)

Intrinsic
value

1508
(827)

5.54
(5.49; 5.59)

1409
(840)

5.51
(5.46; 5.56)

2917(1508; 1409)

0.09(-0.10; 0.28)

0.36

Self-efficacy

1507
(828)

5.13
(5.08; 5.18)

1409
(840)

5.21
(5.15; 5.26)

2916(1507; 1409)-0.05(-0.21; 0.10)

0.53

Test anxiety

1507
(828)

3.46
(3.37; 3.55)

1409
(840)

3.46
(3.37; 3.55)

2916(1507; 1409)-0.01(-0.09; 0.06)

0.74

Selfregulation

1501
(834)

4.77
(4.72; 4.82)

1398
(851)

4.74
(4.69; 4.80)

2899(1501; 1398)

0.05(-0.11; 0.21)

0.52

Outcomep-value

Table 13: Effect size estimation in secondary analysis

Outcome

Unadjusted
differences inmeansIntervention group
Adjusted differences innmeans(missing)

Control group

Variance ofoutcomen(missing)

Variance ofoutcomePooledvariance

Intrinsic
value

0.03

0.07

1508
(827)

0.98

1409
(840)

1.06

1.02

Self-efficacy

-0.08

-0.04

1507
(828)

1.14

1409
(840)

1.15

1.15

Test anxiety

0

-0.03

1507
(828)

3.05

1409
(840)

3.03

3.04

Selfregulation

0.03

0.05

1501
(834)

1.14

1398
(851)

1.21

1.17

Subgroup analysis
The statistical analysis plan proposed a separate analysis for disadvantaged pupils, identified by the indicator ‘Ever
FSM’ included in the NPD records. Tables 14 and 15 present the results for primary and secondary outcomes
respectively. In the primary analysis the effect size is zero for the FSM subgroup, as for the full sample. The confidence
intervals of the effect of the intervention on KS2 maths, reading, and GPS were, respectively, -0.03 to 0.03, -0.02 to
0.03, and -0.03 to 0.03.
In the secondary analysis, the effect size for the intrinsic value subscale is positive but not statistically significant. The
other effect sizes are very small or negative and not statistically significant. Therefore, there is no evidence of an impactof the Changing Mindset intervention on the tested non-cognitive outcomes of Year 6 students that have ever been
eligible for FSM.
We also carried out separate analysis for three other subgroups: females, males, and pupils with ‘fixed mindset’ in the
pre-trial tests. These analyses were not pre-specified in the protocol nor in the SAP and as such they are exploratory
attempts to understand whether the intervention had an effect on groups that showed promising results in other studies
The results of these exploratory analyses show that there is no evidence of an impact of the Changing Mindset
intervention on the tested cognitive and non-cognitive outcomes of Year 6 students in these subgroups. The results forthese additional subgroup analyses are reported in Appendix J.
26

Changing Mindsets
Evaluation Report

Table 14: Primary analysis for FSM subgroup
Raw meansIntervention group

OutcomeKS2 maths

KS2 reading

KS2 GPSn(missing)
782
(27)
780
(29)
779
(30)

Mean
(95% CI)101.81(101.28;102.34)
101.46(100.91;102.02)103.82(103.29;104.36)

Control groupn(missing)
797
(29)
794
(32)
797
(29)

Mean
(95% CI)102.24(101.75;
102.74)
101.40(100.82;
101.98)104.13(103.61;
104.64)n in model(intervention;
control)

Effect size
(95% CI)

1579(782; 797)

0.00(-0.03; 0.03)

0.92

1574
(780; 794)

0.01(-0.02; 0.03)

0.68

1576
(779; 797)

0.00(-0.03; 0.03)

0.92

p-value

Table 15: Secondary analysis for FSM subgroup
Raw meansIntervention group
OutcomeIntrinsic value

Self-efficacy

Test anxiety
Selfregulationn(missing)
565
(244)
565
(244)
565
(244)
563
(246)

Mean
(95% CI)
5.54(5.46;
5.63)
5.03(4.94;
5.12)
3.75(3.60;
3.89)
4.74(4.65;
4.84)

Control groupn(missing)

Mean(95% CI)5.46(5.38;
5.55)
5.12(5.03;
5.20)

n in model(int; control)

598
(228)
593
(233)

598
(228)
598
(228)

Effect size (95%p-valueCI)

1163(565; 598)

0.138(-0.08; 0.37)

0.23

1163
(565; 598)

-0.093(-0.30; 0.12)

0.38

3.65(3.51;
3.79)

1163
(565; 598)

0.049(-0.06; 0.15)

0.36

4.69(4.60;
4.78)

1156
(563; 593)

0.029(-0.17; 0.23)

0.77

Missing data analysis
In this project, missing data affects outcome variables. Primary and secondary outcomes present a high percentage of
missing data, mainly caused by schools dropping out from the project over time; as a consequence, we assess whether
the missing data is ‘missing completely at random’ (MCAR). To do so, we create an indicator variable for each primaryand secondary outcome in the impact model specifying whether the data is missing or not and use logistic regression
to test whether the missing status can be predicted from the following variables: all variables in the precise model plus
school average KS1 maths scores, gender, ethnicity, and eligibility for FSM (for secondary outcomes achievement at
KS1 is also included).
The estimates of the logit models are presented in tables K1 and K2 in Appendix K. The results for primary outcomes
show that the probability of cognitive test scores being missing is associated with ethnicity, previous achievement, FSM
status, and the block indicators. Since missingness can be predicted by some of the variables used in the logistic
regression, in particular FSM eligibility, ethnicity, and prior achievement, the implication is that the missing data could
be missing at random (MAR). To obtain valid estimates it is then necessary to carry out an additional analysis wherethe variables predictive of non-response are included in the estimated model. These additional estimates are presented
in Table K3: the results are not different from the ones based on complete data showing that the results are not biasedby the presence of MAR data but they are underpowered.
The missingness for all secondary outcomes is consistently related to pupil gender, block indicators, and previous
achievement, showing that missing data in these outcomes could be MAR. To reduce the bias introduced by missing
data we carry out an additional analysis where the precise model is augmented with indicators for pupil gender and27

Changing Mindsets
Evaluation Reportprevious achievement. Estimates from this additional analysis are reported in Table K4 and they do not diverge fromthe main results presented in Table 11, suggesting that there is no bias introduced by the presence of MAR data.
However, the high proportion of missing secondary outcomes in the data reduces the informative value of this additional
analysis.

Non-compliance
There was a low response rate to the fidelity survey administered to schools in the intervention group and therefore we
could not observe the level of non-compliance to the intervention. As a consequence, it was not possible to conduct theinitially planned analysis into the effect of non-compliance. Interestingly, the process evaluation and the control groupsurvey reveal that many teachers in both groups of schools were familiar with the growing interest in mindset theory
and had already adopted some of its principles in their everyday teaching style, such as rewarding their pupils for theireffort rather than the outcome achieved.

Costs
The Growing Learners team at the University of Portsmouth offers to deliver the training at a school during an inset
day.10 This delivery method allows all teachers in a participating school to be trained and, as it takes place during an
inset day, schools would not incur in additional costs for covers. The cost of this training is £1,800 per school plus travel
and accommodation costs for two trainers, if the school is located more than 20 miles outside of Portsmouth. This costcovers the mindsets intervention materials and a day of training in mindsets for all staff, which include:••••

an introduction to mindsets;
lesson plans for the six-week intervention for use with the children;
a memory stick containing ready-to-use printable materials (poster displays, learning materials) and a
PowerPoint presentation for other stakeholders; anda free Assessment Package for use with up to 300 pupils before and after the mindset intervention and 12
months’ access to the team’s advice and support service.

We derive the total cost over three years and the total cost per pupil over three years under the following assumptions:travel and accommodation costs for two trainers will be on average £500; teachers trained in the first year will stay inthe school for at least another two years; schools have an average of 60 Year 6 pupils on roll each year; and that all
pupils on roll in the school will benefit from the intervention. The cost for the school over three years will be £2,300 andthe cost per pupil over three years will be £4.26. The cost per pupil and cumulative cost are summarised in Tables 16
and 17.
Table 16: Cost of delivering Changing Mindset

Item

Training
Travel and accommodation
costs for two trainers
Total

Type of cost
Start-up cost perschoolStart-up cost perschoolCost

Total cost over 3 years£1,800£1,800£500£500£2,300Total cost per pupil per
year over 3 years£2,300/180/3 = £4.26

Table 17: Cumulative costs of Changing Mindset

Changing Mindset
Project

Year 1

Year 2

Year 3£2,300£2,300£2,30010 The full set of intervention packages offered by the University of Portsmouth can be found here: http://www2.port.ac.uk/department-of-psychology/community-collaboration/growing-learners/what-we-offer/#schools

28Changing Mindsets
Evaluation Report

Implementation and process evaluation
The purpose of the implementation and process evaluation is to provide insights into participants’ experience of
implementing and delivering the intervention, as well as to bring greater clarity to the quantitative research findings andto understand the reasons behind any impact, or absence of proven impact. Specifically, this process evaluation set
out to understand how the Changing Mindsets intervention programme was implemented and delivered in treatment
schools, and to what extent this differed (if at all) from the intended delivery model. It also aimed to shed light on
implementers’ as well as pupils’ responsiveness to the intervention, and the perceived impact the programme has hadon those involved. Finally, the process evaluation sought to gather information on the activity of schools within thecontrol group over the course of the trial period, and establish how far this differed from the intervention programme.
The research methods used included the following:•

fieldwork visits to six treatment schools conducted from December 2016 to February 2017, which included:▪

semi-structured interviews with nine Year 6 teachers, three deputy/head teachers, and four
teachers from other Year groups;▪

three focus groups with eight to ten Year 6 pupils receiving the intervention; and▪

three lesson observations of the final Changing Mindsets session;•

end of project survey, administered to treatment schools from January to February 2017;•

fidelity survey, administered to treatment schools over course of intervention from September 2016 to February
2017; and•control group survey, administered in June 2017.

All activities were carried out by NIESR, with the exception of the fidelity and control group surveys, which wereadministered by Portsmouth University. The control group survey took the form of a paper questionnaire and wascompleted by attendees at the training days that were provided after programme implementation had finished at theend of academic year 2016/2017. This was forwarded to NIESR and incorporated into the external evaluation. All othersurveys were administered and completed by respondents online.
It should be noted that the views and experiences presented here are not necessarily representative of all participating
schools. The response rate to the fidelity survey decreased over the course of the intervention (from 49 to 9
respondents), while the end-of-project survey was completed by just over half of treatment schools (26 out of 50).
However, steps were taken to ensure that the treatment schools visited as part of fieldwork included a variety of delivery
contexts. We selected schools to include a range by key characteristics: Ofsted rating, proportion of FSM pupils,
geographical location, and whether located in urban or rural settings. The project team provided NIESR with a full listof treatment schools from which we drew up a profile according to characteristics of interest and then aimed to achievea good mix of schools within the sample of six. In two cases, schools were not responsive to our request for participation
and alternatives were approached. The response rate to the control group survey was good in that it was completed by
37 out of 51 schools in this treatment allocation.

Implementation
The key factors for successful implementation of the Changing Mindsets programme are:•training in mindset theory and principles—structure of the programme and delivery;•

access to the training manual;•lesson plans with learning outcomes, activities, and materials including videos and posters;•commitment from the school; and29

Changing Mindsets
Evaluation Report•

space in the timetable.

Familiarity and engagement with growth mindset approaches
The extent to which staff in treatment schools were already familiar, and had engaged, with approaches similar to thosethat informed the Changing Mindsets project is important in understanding their expectations for this work andsubsequent experience of delivery. The findings from the process evaluation show that while many staff members were
aware of similar approaches prior to their involvement in the project, few said they had used these within their own
teaching practices. For example, findings from the survey of staff in treatment schools show that two thirds (21 out of
35 respondents) had read or heard about the concept of a growth mindset prior to the project by engaging with evidence
on the topic. This took the form of written evidence, such as articles or books, or videos they had viewed online. Thirteen
respondents had also attended meetings in which the theory of growth mindset was discussed. They included internal
meetings within the school with other teaching staff, and external meetings or training days for schools within their local
authority cluster.
However, only 7 out of 35 respondents said they had used, in their own teaching practices, approaches similar to thoseencouraged by the Changing Mindsets project in the past, while 11 reported that their school had. Where teaching staff
had engaged with similar approaches, these included the use of online resources, or the adoption of specific growthmindset principles in their own teaching style such as rewarding pupils for their effort, not the outcome achieved;
understanding that making mistakes is part of the learning process; and being knowledgeable of the difference between
a fixed and growth mindset, and how this can affect pupil progress in their learning.
Interviewees from several of the schools that took part in fieldwork visits commented that they were already familiar
with growth mindset approaches and were working to embed these principles into their school culture and/or value
system prior to the start of the project.
‘It was something we were already working on and something that we’d embedded into our set of school values.’
Deputy Head/Year 6 teacher, School 1.
Another school was already looking at how it could work these ideas and concepts into its teaching plans for thefollowing academic year.

Responsiveness of teaching and learning staff to the intervention
Among all of the treatment schools visited as part of fieldwork, there was a high degree of buy-in from interviewees for
the Changing Mindsets intervention programme. It was seen to address a recognised need among learners. Someinterviewees commented that the project tied in well with the introduction of the new national curriculum, which wasseen to be far more challenging for pupils at all levels of ability—that is, engagement with the project had helped pupils
to learn coping strategies to help them progress in this new environment. Others spoke more generally of how it wasnecessary to get all pupils, even the high achievers, to reflect on the process by which they learn and become familiar
with such coping strategies, given that all would encounter difficulties at some point in their education.
Staff from one school observed that their current cohort of Year 6 pupils was underperforming, and they were therefore
encouraged to get involved in the project to improve the perseverance and resilience in learning of this groupspecifically:‘We’ve worked with them throughout the whole of Year 5 to try and plug the gaps, if you like, and we felt that thisChanging Mindsets would be the extra layer to try and get them to think about their own learning.’
Deputy Headteacher, School 6.
Interviewees from another school commented that, after hearing the concept of growth mindset referred to by otherstaff, they undertook some background reading on this approach. They reported reading that while it had already been
tried in a number schools, it had not always been successfully implemented as not all staff had a clear idea of what it30

Changing Mindsets
Evaluation Reportwas. The school concerned was, therefore, motivated to become involved in this project as it felt it would give all staff
a clear understanding of the principles underpinning this approach, and also liked the fact the project would be evaluatedin terms of its impact.
Experiences of the training
The delivery team for this project delivered one day’s training for all treatment schools. This took place at the start of
the academic year (September 2016). The results of the end-of-project survey show that the majority of respondents
who attended the training (23 out of 30) were satisfied with the content of the training day, while seven were ‘somewhat
satisfied’. Responses to an open ended question about how the training day could have been improved included apreference for more practice with activities rather than the theoretical basis for growth mindset.
The findings from the fieldwork visits and end-of-project survey show some consistency in terms of how treatment
schools felt the training could be improved. Most commonly, respondents wanted the training to focus less on the
background and evidence base underpinning growth mindset approaches. As noted, many treatment schools had
already engaged with evidence on this topic and were somewhat familiar with the background to this approach, while a
few already had intentions to cultivate a growth mindset culture within their school and were therefore highly motivatedto take part.
‘They were preaching to the converted really […] people were there because they wanted to get involved.’
Year 6 teacher, School 5.
As an alternative, some attendees observed that would have liked the training to have spent more time going throughand discussing the intervention materials, such as the lesson plans, and to have had further opportunities to practicedelivering some of the suggested activities and reflect on how they may best be applied in the classroom. While sometime was spent during the training presenting the lesson plans for each session, some attendees did not feel that thiselement was very interactive.
In spite of this, most respondents (18 out of 30) to the end-of-project survey felt, to a great extent, that they wereprepared for delivering the Changing Mindsets intervention as a result of the training day and the material. Only four
said the training and materials had prepared them ‘very little’ and eight said they had done so ‘somewhat’.

Fidelity
The Changing Mindsets intervention programme was designed to be delivered by teachers in the form of eight sessions.
It was suggested that teachers deliver one session per week, over eight consecutive weeks, with each session lasting
approximately two hours. The delivery team at Portsmouth University made clear, however, that the intervention was
designed to be flexible, and treatment schools were free to deliver the sessions in shorter, more regular intervals if they
preferred. Schools were also encouraged to adapt and modify the materials to suit the needs of their classes; the only
consistency required was that the learning objectives set for each session were achieved by the children.
The findings from the fidelity survey administered by the delivery team show that, according to teacher self-reports,
many treatment schools were able to meet these requirements. As Table 18 shows, over the course of the programme,
most respondents were able to deliver sessions that were one to two hours in duration, although data is missing frommany schools for later sessions. Most commonly, respondents were able deliver sessions that lasted from one and ahalf to two hours, as recommended by the delivery team. Year 6 teachers were also asked to note the date and time ofeach session, and from this it appeared that the majority of respondents opted to deliver one session per week in asingle block, rather than splitting the eight sessions into shorter, more regular lessons.

31Changing Mindsets
Evaluation Report
Table 18: Length of session
Session no.
30 minutes–1 hour
1–1.5 hours
1.5–2 hours
2 hours or more
N

1
6
17
21
4
49

2
3
12
19
34

3
3
10
15
28

4
5
16
3
26

5
6
15
21

6
1
9
2
12

7
6
7
3
16

8
4
4
1
9

Among the treatment schools that took part in the fieldwork visits, while all were aiming to deliver a single, weekly
session, the length of these varied from between 40 minutes to close to two hours. For those delivering shorter sessions,the main reason was that pupils were used to working in 40- to 60-minute slots and teachers believed they would
struggle to maintain concentration over a longer period. In these cases, it was clear from the interviews and lesson
observations that fewer activities were being delivered than suggested in the lesson plan as a result of the shortened
session time.
Concerns over pupil concentration were also expressed among a few teachers delivering sessions that lasted longer
than one hour. However, they opted to deliver longer sessions so they could attempt to complete all the suggested
activities. Some interviewees who expressed this view did not feel it was possible to deliver more regular, shorter
sessions as an alternative due to pressures to deliver the necessary aspects of the school curriculum.
Those schools that participated in fieldwork had managed to deliver most of the sessions on consecutive weeks, thougha few mentioned that they had been unable to deliver sessions during some weeks due to day-trips or other events
within the school calendar in which all Year 6 pupils were participating. When this happened on more than one occasion,
schools had to complete the final few sessions after the Christmas holiday period, at the beginning of the Spring term.
In spite of variations in the reported length of the sessions, their regularity, and the number of suggested activities Year
6 teachers were able to get through, the results of the fidelity survey show that, for most sessions, the majority of
respondents agreed that the key learning outcomes provided in the training manual had been met. However, we cannot
be sure this is true of the final few sessions since the response rate to the fidelity survey fell during the final few weeks
of implementation: data was received from only a small proportion of the treatment sample. An alternative summative
measure of pupils’ achievement in learning through the intervention programme is provided by the end-of-project survey.
This was completed by just over half of all participating treatment schools. It found that the overwhelming majority of
respondents (21 out of 24) who were involved in delivering these sessions felt that pupils had understood the ideas andconcepts they had been taught.
As mentioned, the achievement of the stated learning outcomes for each session was the main aspect of the treatment
model that schools were asked to adhere to. These findings therefore suggest that the necessary conditions for trial
fidelity were largely met.
In the small number of instances where teachers did not agree that some or all of the learning outcomes for a given
session had been achieved, the reasons for this were not always made explicit by respondents. Where explanations
were provided, this was attributed to difficulties in accessing online resources or the time-consuming nature of some ofthe suggested activities, which left respondents with little time to get through all of the content they intended to cover.

View and use of project resources
As indicated in the preceding section, Year 6 teachers’ experience of delivering the Changing Mindsets intervention
programme was influenced by their use of the resources provided by the delivery team: namely, the training manual, in
which the lesson plans for the eight sessions were contained, and the online videos that accompanied the sessions
supplied by Positive Edge Education.
With regards to the training manual: as highlighted, treatment schools differed in how many of the suggested activities
they attempted to deliver for each session. The fieldwork visits showed that teachers based this decision on how longthey believed their pupils would be able to concentrate in lessons and progress through the suggested content, as well32

Changing Mindsets
Evaluation Reportas how much of this they felt they had to deliver to meet the requirements of the project in terms of fidelity and achieve
the stated learning outcomes.
In line with the guidance issued by the delivery team at Portsmouth University, teachers also altered the content andnature of the activities they chose to deliver. The end-of-project survey showed that most of these changes, however,were minor, with the majority of respondents (16 out of 24) involved in delivery reporting that they had changed thecontent of the lessons very little. In some cases, this involved small tweaks to the content to enhance its relevance topupils, or to make the activity more practical or engaging for pupils to complete. This included relating the content topupils’ upcoming SATS exams and turning whole class exercises into paired or group exercises where more children
had the opportunity to contribute their thoughts or ideas to the task at hand. In a few cases, teachers skipped some ofactivities entirely where they did not believe they were challenging or engaging enough for their class. One example
related to the multiplication and division tricks covered in session six, which the teacher believed were below the ability
level of their Year 6 pupils. Another found that pupils did not engage with the character animations that were used insome of the sessions, and so the teacher concerned found their own video clips and examples to illustrate the same
points in subsequent lessons.
While some treatment schools made small changes to the content of the lessons, the results of the end-of-project survey
showed that all respondents found the training manual overall to be useful. Several positive comments were made on
its content during the fieldwork visits with treatment schools. Some teachers felt that the manual was very
comprehensive, and all the materials and resources were there that they needed to deliver the sessions with very little
preparation required. Other teachers that were delivering shorter sessions than recommended (that is, 40–60 minutes)
commented that there was a wealth of content to choose from. This group were pleased that the lessons plans could
be used flexibly and that they could pick and choose what activities were most relevant and appropriate for theirstudents.
‘We always had the opportunity to select what we wanted to use. It was never under resourced.’Year 6 teacher, School 2.
‘The lesson plans are really clear, quite flexible as well, so quite easy to adapt to your own children.’Deputy Head, School 6.
While a few teachers criticised the amount of content included in the lesson plans and observed that there was toomuch to get through, even in a two-hour period, not all of these individuals had attended the training day delivered by
the delivery team. As a result, it may be some of these teachers may not have been made as acutely aware that theintervention was designed to be flexible, and that they were free to modify and adapt the materials as they saw fit.
Another resource that teachers were provided with access to as part of the intervention programme were the online
videos supplied by Positive Edge Education. This was paid content, which schools were provided with individual
licences to access. Year 6 teachers were required to activate these licences and log-in to an online system every timethey wanted to view the videos. They were referenced throughout the lesson plans, and teachers were encouraged tomake use of particular videos to help emphasise key points. The videos were made available on a timed release format:
teachers could only view the next video 24 hours after the previous one had finished. At the end of each video, a seriesof multiple-choice and long-answer questions were presented for teachers to work through with pupils to check theircomprehension of the ideas presented. Teachers needed to complete this form before they could start the timed releaseof the next video. Even if they did not wish to answer the questions, they were required to select ‘skip’ for those thatwere multiple choice or write some text in the box for the long-answer questions in order for the next video to bereleased.
The findings from the process evaluation show that, overall, the content of the videos was generally well received. For
example, the end-of-project survey showed that the vast majority (26 out of 29) of respondents found the videos to be
useful, with over half (18) rating them as very useful. Similarly, during fieldwork with treatment schools, several teachers
remarked how the pupils had generally enjoyed the content of the PPE videos. Some teachers felt they were a usefulin keeping pupils engaged with the content of the session and having the questions at the end helped to encourage
33

Changing Mindsets
Evaluation Reportpupils to maintain their concentration throughout the duration of the video. Others stated that the content of some of thevideos (for example, the story of Charles Darwin’s early life) tied in nicely with the cross-curricular topics they werestudying in Year 6, and provided real-life examples of where growth mindset approaches had worked, which pupils
related to better than the character animations:‘The famous failures videos […]—they can actually see that hugely successful people are using this, whether they callit growth mindset or not, that’s what they’re doing.’
Year 6 teacher, School 5.
However, a few teachers commented that the content of particular videos was too advanced and not age-appropriate
for Year 6 pupils (for example, the Einstein explainer video that discussed the structure of the brain). Another criticism
was that the videos could also be too long in duration, with some lasting up to 20 minutes. These factors combined
meant that teachers felt that pupils, on occasions, had struggled to pay attention and recall all of the content presented.
‘There was such a discrepancy between the pitch of those videos and then the pitch of some of the activities that actuallyfor the children to then be able to summarise what the film was about, you know, or answer the questions about what
was in the film, it was too long and they’d forgotten. I wasn’t even sure I knew the right answer.’
Deputy Head/Year 6 teacher, School 6.
The findings from the lesson observations showed that some Year 6 teachers were selective about how much of thevideo content they presented to pupils in order to get key points across. This again illustrates that Year 6 teachers
differed in how comfortable they felt in modifying the content of the intervention programme to suit the needs of theirclass.
By far the most common criticism of the PPE videos, however, was the process for accessing them. Teachers expressed
their frustration that they had to log-in every time they wished to view the videos, and that they were unable to get asense of their content at the beginning of implementation due to the time release format, which would have helped them
to better plan their delivery. Teachers also recounted how they had not always been able to access the videos theyneeded for each session as, in error, they had not started the timed release process. This was attributed to not having
completed viewing the previous video to the end of the running time or provided responses to the entire series ofquestions that followed. This impacted on teachers’ ability to get through all the content they intended for the session
and, as mentioned, in a few cases, their ability to meet the stated learning objectives.
Pupil responsiveness to intervention
The findings from the fidelity questionnaire, detailed in Table 19, show that pupils appeared to respond well to the
Changing Mindsets sessions with the majority of respondents agreeing that pupils were engaged throughout. Further,as highlighted in Table 20, only a minority of respondents noted that pupils were disruptive in these sessions to such a
degree that their behaviour negatively impacted on others.
Table 19: ‘Pupils were engaged throughout the session’ (fidelity questionnaire)

Session no.
Strongly agree
Agree
Mostly agree
Mostly disagree
DisagreeStrongly disagree
N

1
13
27
7
2
49

2
8
18
4
2
1
1
34

3
8
11
5
3
1
28

4
7
11
5
1
1
1
26

5
6
7
5
1
2
21

6
3
6
2
1
12

7
3
8
4
1
16

8
5
3
1
9

34

Changing Mindsets
Evaluation Report
Table 20: ‘Some pupils were disruptive to such a degree that the session was negatively impacted for others’ (fidelity questionnaire)
Session no
Strongly agree
Agree
Mostly agree
Mostly disagree
DisagreeStrongly disagree
N

1
2
1
1
13
32
49

2
3
3
6
22
34

3
4
1
7
16
28

4
2
2
1
5
16
26

5
4
1
6
10
21

6
1
1
6
4
12

7
1
1
4
10
16

8
1
1
7
9

Overall, in their delivery, several teachers and pupils noted during the fieldwork visits that the sessions were largely
discursive and reflective in nature. Pupils in one school commented that this format had made the sessions fun andinteresting.
‘After the video we have a massive talk and everyone just gives ideas and examples.’Year 6 pupil, School 5.
Year 6 teachers were similarly positive about this format, although one school did note that having a lot of classdiscussions and group work did mean that occasionally some children coasted in the session and did not make muchof a contribution to these activities. This behaviour was noted during a few of the lesson observations where group work
was being completed.
Teachers in one school felt that pupils also exhibited a high level of engagement due to the content, which was focusedon how they learn, and was therefore very practical for them. The interviewees contrasted this to pupils’ level ofparticipation in Personal Social and Health Education lessons, which followed a similar format, yet tended to be very
low. However, some pupils from a different school found the process of writing down or presenting their ideas on thecontent of the sessions each week quite repetitive and wanted to undertake more practical activities. Their Year 6
teacher supported this, arguing that while pupils had learnt the theory underpinning growth mindset through thesessions, not all had yet had chance to apply it. They commented that this applied in particular to the high-achievers intheir class who did not find their school work especially challenging. The teacher concerned suggested that the sessions
should, therefore, include more practical activities where pupils are encouraged to learn a new skill that none of them
have tried before. They themselves had taken the time to teach pupils how to juggle during the first few sessions, which
the class had enjoyed enormously.
Whole-school approach
As part of the training that treatment schools received in preparation for delivering the intervention, the delivery teamrecommended that in order for the programme to be most effective, it was necessary for staff to work embedding a
growth mindset culture or ethos across the whole school. As mentioned, a few treatment schools that participated infieldwork were already looking to cultivate a more supportive learning environment informed by the theory of growthmindset prior to their involvement in this project. Following the training day, most of this group had therefore used thisformal opportunity to cascade some or part of what they had learnt and the resources they had received to otherteachers within the school. The end-of-project survey supported this finding and showed that most respondents (19 outof 30) had disseminated the ideas they gained from the training day to other teachers in the school.
The most widespread change in practice that this encouraged was altering the nature of the feedback and praise thatteachers provided to pupils. This included omitting any fixed mindset language that is person- and/or ability-centred,
such as ‘clever boy!’ and focusing instead on praising the process by which pupils complete a particular task, and theirprogress in learning, as opposed to the end result. Some schools had also attempted to integrate these priorities into
their school value system by assessing pupils’ work according to the effort put into it.
Teachers also attempted to adopt a growth mindset in terms of how they viewed their own intelligence. It was recognisedthat teaching staff had to model the behaviour they wanted to see if they were going to avoid passing on fixed mindset
ideas and messages to pupils. To help encourage this change in attitude, interviewees from some schools spoke of35

Changing Mindsets
Evaluation Report

how they had put up growth mindset displays or the ‘my favourite mistakes’ board in the staff room to prompt teachers
to reflect on and challenge their own approach to learning.
With regards to the training manual and lesson plans provided to all treatment schools, Year 6 teachers spoke of how
some teachers had taken ideas for discreet activities that they then delivered to their own class. In other cases, teachers
remarked how staff from other year groups were in the process of slowly building up their own age-appropriate
resources for teaching these ideas and concepts to other pupils within the school. In a few instances, teachers also
spoke of plans to discuss these topics during school assemblies to further support their dissemination.
The end-of-project survey showed that, in the view of the majority (17 out of 24) of respondents, other teachers within
the school had been very willing to incorporate growth mindset approaches into their own practice. This was supportedby the findings from the fieldwork visits: teachers mentioned that while some members of staff were initially resistant tothese ideas, as they learnt more about this approach and came to see its potential benefit for pupils, they became
receptive to changing their working practices.

Outcomes
Impact on pupils
The fieldwork visits took place towards the end of the programme’s implementation when teachers were delivering thefinal Changing Mindsets lesson, or just after. At this point, many interviewees were uncertain about whether the project
could have a positive impact on attainment among Year 6 pupils. This was due to the short-term nature of theintervention delivered to pupils, and the ability of pupils to retain the key ideas and concepts they had learnt over the
rest of the school year after the sessions had ended.
A number of interviewees observed that the principles underpinning the programme could take time to embed, and citedseveral examples where, over the course of implementation, their pupils had struggled to make links between thecontent of different lessons and had to be prompted by teachers to help recall what they had covered previously.
Interviewees commented that for this approach to be effective pupils needed to be making these connections andapplying these concepts to their own learning spontaneously, with little or no input from teachers. They therefore
observed that it might be necessary for pupils to be taught the ideas and concepts contained within the programmeover a number of years in order for these changes, and any knock-on impact on attainment, to manifest.
Similarly, it was recognised that the key messages from the programme had to be reinforced at home by pupils’ parents
as well as at school if the intervention was to have any long-term impact on their progress and attainment. Teachers
recalled previous conversations with the parents of pupils in their class who had exhibited a fixed mindset. In their view,
this had the potential to undo all the school’s work to date to cultivate a different and more positive attitude towards
learning.
‘What happens is that we do stuff here and it gets undone again at home because parents haven’t changed theirapproach or attitude. I sit with children who are finding maths really hard and Mum says, “I can’t do maths either”. Those
sorts of things, […] that’s what you want to stop.’
Headteacher, School 5.

In spite of these challenges to the effectiveness of the intervention, most treatment schools visited did perceive softer,
short-term changes in pupils’ attitudes, enthusiasm, and perseverance in their own learning that they attributed to theprogramme. Some commented that this had the potential to eventually improve pupil attainment if it could bemaintained. For instance, a few Year 6 teachers noted how over the course of the intervention, when pupils were
presented with a difficult task in other lessons, they appeared more determined to complete it and responded positively
to the challenge, rather than giving up. Examples included being faced with a difficult set of problems within a maths
lesson or setting the pupils a code-breaker challenge in relation to the project they were completing on World War 2.
Comments from pupils showed that they also appeared to have internalised the messages about persevering with thetask at hand.
36

Changing Mindsets
Evaluation Report‘If I had given them [the code breaker challenge] last year, I honestly don’t think they would have been determined.’
Year 6 teacher, School 4.
‘Don’t give up. That’s the main lesson.’Year 6 pupil, School 5.
‘You’re not supposed to say you can’t do it because you need to learn how to say, “Okay, right, so I need to get on withthis and I don’t know how to do it, but I just need to keep trying.” [… or] “If you think someone is born with it, and you
give up, you’ll never be as good as them, but if you keep practicing they’ll just keep on practicing and then you’ll
eventually be as good as them.”’Year 6 pupil, School 1.
The interviews and focus groups highlighted how pupils were now more comfortable in making mistakes in class andacknowledging this in front of their peers. In the teachers’ view, the content of the sessions as well as the adoption of a
‘my favourite mistakes’ board in class had helped pupils to recognise that mistakes are a natural part of the learning
process. It was felt that this change in pupils’ attitudes had also been further reinforced by disseminating these ideas
throughout the whole school and encouraging other staff to alter the nature of the feedback and praise that they provide
to pupils.
‘I do think there has been a noticeable difference there: “We are okay to make mistakes, we are okay to fail.”’Year 6 teacher, School 4.
‘I’ve noticed with the children that if they make a mistake now, they are not afraid to say.’Year 6 teaching assistant, School 1.
‘I can see it in our numeracy lessons, especially the ones that didn’t like making a mistake and had that attitude of, “Oh,I’ve got it wrong.” But now they see it as okay: “I can improve, it’s my first step, it’s a challenge, I want to get further.”'Year 6 teacher, School 3.
‘Learn from your mistakes. It’s okay if you get it wrong. If you get it wrong, try again. People who never make a
mistake never try anything new.’Year 6 pupil, School 5.
Both teachers and pupils also identified examples of where pupils had repeated these messages to their family andfriends, demonstrating a good awareness of the difference between the concepts of a fixed and growth mindset, andthe language that cultivates these approaches to learning:‘Parents have been saying to me that their children have been coming home and telling them that they had to not giveup […] and mistakes make your brain grow. They’ve been getting a real lecture at home.’
Deputy Head/Year 6 teacher, School 1.
‘Whenever my brother says, “I can’t do it”, I always say that he can’t do it yet.’Year 6 pupil, School 5.

In terms of the groups of pupils for whom the intervention was judged to have worked best, some interviewees perceived
the most progress among those with previously low levels of attainment or those with special educational needs. As
37

Changing Mindsets
Evaluation Report

one interviewee highlighted, the way in which pupils had been taught to focus more on their individual progress inlearning and the amount of effort they put into a task as opposed to the end result had helped this group to stop
comparing their performance with that of their peers. As a result, pupils gained greater confidence in their ability to
improve in school and ‘grow’ their own intelligence.
In contrast, several teachers felt that the high-achievers in their class had found these concepts most difficult to grasp.
The delivery team at Portsmouth University had made teachers aware at the training day that this might be the case.
Interviewees commented that these individuals had found everything in school easy up to that point and did not see aneed for these approaches having not experienced failure very often. However, they emphasised that it was crucial thathigh-achievers still engage with and learn from the content of the sessions. This is to avoid the negative emotional
impact and damage to pupils’ self-confidence that they will experience when they do encounter a set of challenges thatare too great, which the Year 6 teachers perceived as inevitable.
Impact on teaching staff
Many Year 6 teachers also perceived a positive change in how they view their own intelligence and approach learning
as a result of participating in the programme, which they intended to carry forward. In teaching these sessions, several
remarked how they had become more aware of the areas of their life where they tended to adopt a fixed mindset. They
recognised the importance of challenging these ideas about their own intelligence and adopting more of a growthmindset if they were going to successfully convince pupils of the validity of this approach and encourage them to do thesame. Teachers spoke of how they had relayed their own examples to the class in order to model the behaviour theywanted to see—something that was encouraged at the beginning of implementation by the delivery team.
Involvement in the programme was also seen to have had a positive impact on how staff within treatment schools
provided praise and feedback to pupils. Some interviewees spoke of how they are now far more conscious of what they
say to pupils and whether their comments cultivate a view that intelligence is a fixed and innate quality, or whether it issomething that is malleable and can grow with the right learning strategies, effort, and perseverance. Teachers were
now far more aware of how their own feedback could put limits on what pupils believe they are capable of achieving
and how they respond to challenges. Some spoke of how they now focused on praising the process by which pupils
achieved a particular outcome, rather than the outcome itself, which does not always acknowledge pupils’ progress inlearning. In a few cases, treatment schools had updated their marking and feedback policy for all staff to reflect thisshift in focus.
‘I think we moved away from [rewarding] the end products and more on the effort and hard work and theperseverance, and the children are recognising that as well.’
Deputy Headteacher, School 6.
Impact on the delivery of other lessons

As part of the fieldwork visits, several Year 6 teachers spoke of how they had been bringing what pupils had learnt in
the standalone Changing Mindset sessions into particular subjects, most commonly, English or maths.
‘Within maths lessons we make lots of links […] the part about making mistakes […] in my numeracy lessons [we make]a big deal about this fantastic mistake.’Year 6 teacher, School 3.
Teachers commented on how it complemented the introduction of the new national curriculum and the introduction of
the concept of ‘mastery’ in mathematics (that is, that all pupils are capable of reaching the same level of achievement,
they just have to find the learning strategy that best works for them). Others had tried to encourage pupils to transfer
what they had learnt to these subjects, for instance, by placing pupils in the same pairings in both Changing Mindsets
sessions and maths lessons.
However, a few teachers were concerned that the time commitment required to deliver the Changing Mindsets sessions
had a detrimental impact on pupils’ progress through other elements of the curriculum. Variously, teachers spoke of38

Changing Mindsets
Evaluation Report

how their involvement in the intervention had taken a significant amount teaching time away from certain lessons, suchas the Year 6 cross-curricular topic, history, and ICT/computing. One teacher was particularly concerned about being
unable to catch-up on this content over the rest of the academic year and, ‘losing’ these aspects of the curriculum that
also needed to be taught and delivered. However, many were aware of the time commitment the intervention would
require when they first registered their interest to be part of the project and were accepting of this fact.

Formative findings
Treatment schools involved in the fieldwork visits and those responding to the end-of-project survey identified several
ways in which the intervention could be improved. By far the most widely discussed was the development of a schoolwide programme, with appropriate resources provided for each year group. As mentioned, Year 6 teachers felt that theideas and concepts taught as part of the programme would take time to embed. They observed that reinforcing andbuilding on this content year on year would therefore help enhance the effectiveness of the approach as the learning
strategies taught become more instinctive for pupils. Ideally, they believed that teaching should start at an early age,from nursery and reception, when pupils are potentially more receptive having limited learning experience and therefore
less of a fixed idea about their own intelligence.
‘I think it’s quite unusual with my four-year-olds for children to go, “I can’t do that”. Mostly, they will just have a go, butthat stops quite quickly so by the age of seven, probably, that approach has changed and they are back to, “well you
haven’t shown me how to do this so I can’t possibly and this is really hard”.’
Headteacher, School 5.
Other suggested improvements included greater linkages between the content of programme and the teaching ofparticular subjects, such as English or maths. As shown, while teachers were already bringing the content of the
Changing Mindsets sessions into other lessons, several wanted a more tailored programme that taught these concepts
in the context of other subjects as opposed to standalone sessions.
A few teachers also suggested that the sessions should be designed to be shorter in length, perhaps spread over thecourse of the academic year instead of a single term. Others felt that they could be broken down into short activities
delivered throughout the day, in 10- to 20-minute slots, or adapted and integrated with other lessons. As highlighted,
some teachers that opted to deliver longer sessions were concerned about pupils’ level of concentration and how muchinformation they were able to retain, as well as the potentially deleterious impact this was having on their ability todeliver elements of the school curriculum.
Further improvements put forward were an easier system for accessing the Positive Edge Education videos with notimed release format, corrections to some of the materials provided for delivery, which included spelling errors andincorrect answers, and using real life examples of individuals who have exhibited a growth mindset when faced withchallenges across a greater variety of cultural and religious/faith-based backgrounds.

Future plans
All of the treatment schools visited as part of fieldwork had plans to reinforce the key messages from the Changing
Mindsets programme over the rest of the academic year and for engaging pupils’ parents in this approach. As
highlighted, teaching staff were aware that the effectiveness of the project would be limited in the medium- to long-term
if pupils were unable to retain what they had learnt and/or if they continued to be exposed to fixed mindsets language
at home.
With regards to reinforcing the main ideas and concepts that informed the programme, Year 6 teachers planned tocontinue to link and refer back to the content of Changing Mindsets sessions in other subjects, such as English andmaths, so pupils had the opportunity to apply what they had learnt. Some teachers also planned to repeat a few of theactivities that featured in the lesson plans and tie them in with the cross-curricular topics pupils were covering thefollowing term. Others planned to continue to use classroom features developed for the programme, such as the ‘myfavourite mistakes’ board. A few teachers had found more resources online that they planned to work into individual
39

Changing Mindsets
Evaluation Reportlessons the following term, while some spoke of holding individual discussion groups (circle time) with pupils on the
topic.
Further ahead, some interviewees commented that they had plans to implement the approach more widely and holdindividual sessions on growth mindset approaches during the following academic year in other year groups. For someschools, this was conditional on the intervention having been shown to have a positive impact on pupil attainment, whileothers planned to press ahead regardless as engaging in the sessions, in their view, was not having an adverse impact
on pupils and was worth continuing.
As mentioned, teaching staff in treatment schools also spoke of plans to engage parents with the project over thefollowing term—also one of the recommendations that was made by the delivery team at the outset of the project toenhance impact. While a few schools planned to provide parents with more information on the theory underpinning the
project and why it is important in pupils’ learning, they had not yet decided what format this would take. Several schools,
however, had alternative approaches in mind for informing parents about the project that they felt would be moreinteresting for this audience. This included the delivery team’s recommendation that the school hold an ‘expert-led’workshop in which children lead the proceedings and tell parents what they have learnt. Other planned approaches
were more interactive and included parents completing some of the practical activities they had covered in the sessions
with pupils, as well as receiving background information on the topic.

Control group activity
Information was gathered from the control group of schools on their activity during the trial via a short questionnaire. As
this study used a waitlist control, the survey was issued at the training days held for control schools on how to deliver
the Changing Mindsets intervention, which were held at the end of the trial period (June 2017). It was completed by 96
respondents, representing 37 out of 51 control group schools.
Overall, the survey showed a presence of both knowledge of growth mindset approaches and the implementation ofrelated teaching strategies. Whilst just three participants reported that they were familiar with growth mindset
approaches ‘to a great extent’, the majority of respondents reported that they were ‘somewhat’ familiar (72 out of 96).
Half of the ‘somewhat’ respondents said they had previously attended training days where interventions based on
growth mindset approaches and/or the theories of Carol Dweck and others like Hymer, Duckworth, and Fleetham were
taught. Thirteen stated that they had engaged with independent research on the topic, including reading online articles
and participating in relevant Facebook group conversations with fellow teachers, and, in one case, buying members ofstaff a growth mindset pocketbook. Additionally, eight respondents claimed they were taught about growth mindset
approaches, or the work of Carol Dweck, at university or through Teach First training. The three teachers that reported
familiarity of growth mindset ‘to a great extent’ attributed this to multiple training days and personal research, including
one dissertation using early primary examples of Dweck’s theories.
The control group survey also asked respondents to what extent growth mindset approaches had been implemented intheir school. The three participants who previously reported a ‘great extent’ of familiarity with the approach also statedthe methods had been implemented ‘to a great extent’ in their schools and classrooms. Forty-three respondents said
growth mindset had been implemented ‘somewhat’ and 39 reported ‘very little’, highlighting a variation of activity. The
majority of participants responded that their school had used approaches similar to growth mindset in the past (44 outof 96) whilst just 18 responded that they had used these approaches in their own classroom. More than 20 other similar
approaches were named with Class Dojo, Purple Polishing Pens, and Guy Claxton’s BLP being the most common.
Finally, the questionnaire sought to gauge whether teachers were employing teaching strategies similar thoseencouraged in growth mindset theory and practice, regardless of their knowledge or awareness of this approach. The
findings show that a high number of respondents employed similar teaching strategies. Most commonly, respondents
reported that:•••they all set high expectations for learning effort, persistence, and eventual attainment (96/96 respondents);
they taught or encouraged pupils to see mistakes as part of the learning process (91/96 respondents); andthey praised pupils’ efforts rather than their intelligence (89/96 respondents).
40

Changing Mindsets
Evaluation Report

There was an almost even split amongst teachers celebrating mistakes in class with 41 employing this tactic against 51
who did not. A less frequently used approach by control schools was helping pupils to see that they can change how
intelligent they are, which had been employed by just 37 out of 96 respondents. These findings indicate that some of
the core messages of Changing Mindsets were also used by control schools. While control schools were not deliveringa specific growth mindset programme, it is likely that pupils were also being exposed to the same messages on a regularbasis and that it had a positive impact on their learning.

41

Changing Mindsets
Evaluation Report

Conclusion
Key conclusions
1.

Pupils in schools that received the intervention did not make any additional progress in literacy nor numeracy—as measured
by the national Key Stage 2 tests in reading, grammar, punctuation, and spelling (GPS), and maths—compared to pupils
in the control group. This finding has high security.

2.

This evaluation also examined four measures of non-cognitive skills: intrinsic value, self-efficacy, test anxiety, and selfregulation. The evaluation did not find evidence of an impact on these measures for pupils in schools that received Changing
Mindsets. A positive impact was found for the intrinsic value measure, but the impact was small and was not statisticallysignificant.

3.

Among pupils eligible for free school meals (‘FSM pupils’), those in schools that received the intervention did not make any
additional progress in literacy nor numeracy—as measured by the national Key Stage 2 tests in reading, GPS, and maths—
compared to FSM pupils in schools that did not receive the intervention.

4.

One explanation for the absence of a measurable impact on pupil attainment is the widespread use of the growth mindset
theory. Most teachers in the comparison schools (that did not receive the intervention) were familiar with this, and over athird reported that they had attended training days based on the growth mindset approach.

Interpretation
In this Changing Mindsets effectiveness trial we evaluate the causal effect of a greater scale intervention that—like thepilot studied by Rienzo and al. (2015)—partly differs from the ‘growing mindset’ interventions considered by Carol
Dweck and her co-authors: the subjects of the Changing Mindsets interventions are younger pupils who are introduced
to the idea of incremental intelligence by their teachers over several sessions. Unlike the other trials relating to growthmindset in the literature, our analysis found that the Changing Mindsets intervention had no impact on literacy andnumeracy overall, and that this applied across all pupils, including those eligible for FSM. Results from the analysis of
non-cognitive skills show that the intervention had a small, positive, but not statistically significant effect on a measure
derived from the Motivated Strategies for Learning Questionnaire, namely the ‘intrinsic value’ subscale. The magnitudeof this positive effect is higher among FSM pupils, however, none of these effects are statistically significant. The
positive effect overall is, however, encouraging since the intrinsic value scale measures attitudes towards learning frommistakes and enjoyment of classroom learning.
In theory, there are three possible reasons why the intervention’s measured impact was not statistically significant:•the programme was not delivered as intended, or was too short, so that pupils did not take on its messages
and change their attitudes, behaviours, and, consequently, their performance;•

control schools were also using growth mindset approaches, and the treatment schools had already been using
it to some extent; and•the pupils were too young and that older children are much better able to use growth mindsets to improve theirperformance, particularly as reflected in tests.

Explanation 1: The programme was not delivered as intended
We can rule out this explanation using evidence obtained at various points by the IPE. The evaluation indicates a goodlevel of fidelity with limited adaptations to the programme. The programme was very well received with a high degreeof buy-in from schools and teachers. It was seen to address a recognised need to change pupils’ views of their own
intelligence and approach to learning.
The programme was of relatively short duration, at eight weeks. However, each session was generally two and a half
hours long and involved repeated delivery of key messages. Teachers also said they reinforced the messages in otherlessons. Teachers reported good levels of understanding among pupils about the messages of the programme and thiswas evident in interviews carried out with pupils by the evaluation team. Despite the absence of a measurable impact,teachers reported softer, short-term changes in pupils’ attitudes, enthusiasm, and perseverance in learning that they42

Changing Mindsets
Evaluation Reportattributed to the programme. The interviews and focus groups for the IPE found evidence that pupils were comfortablewith the idea of making mistakes in class and acknowledging this in front of peers. Teachers also reported that the
programme had influenced their thinking about their own intelligence, which may have improved their own practice.
Such changes are difficult to measure.
The programme appears to have been popular among pupils and teachers. As an indication of its popularity, teaching
staff in treatment schools planned to implement the approach more widely, including for younger pupils, at whole-schoollevel, and to engage parents. It is possible that these longer-term strategies would increase the impact of the
programme. This could involve starting the programme in earlier year groups so that pupils incorporate its messages
naturally and consistently into their work.
Explanation 2: Control and treatment schools were already using growth mindset to some degree
Schools that had already delivered a growth mindset programme were excluded from the trial. Findings from the surveyof staff in treatment schools shows that few had used this approach in their own teaching practices, although some also
said their schools had done so. However, two thirds report knowledge of growth mindset prior to the project by engaging
with evidence on the topic. Interviewees from several schools that took part in fieldwork visits commented that they
were already familiar with growth mindset approaches and had been working to embed the principles into their school
culture and/or value system prior to the start of the project. These findings suggest that some teachers may have beenusing growth mindset approaches to an extent before the programme, thereby affecting the extent to which it could add
value.
In relation to control schools, the process evaluation found a high degree of familiarity with growth mindset theories withmany respondents having attended training days where interventions based on the theory or the work of Carol Dweck
and other exponents were taught. Other ways in which teachers in control schools had become familiar with growthmindset were through reading research material on the subject or engaging in discussions with colleagues. Moreover,around half of control group respondents said it had been ‘somewhat’ implemented in their schools, while a small
number also said it had been implemented to a ‘great extent’. A little over a third said it had been used ‘very little’ intheir school. The survey of control school teachers also found that schools operated growth mindset principles even if
they did not label them as such. These results suggest a high degree of awareness of the approach, which we cannot
be certain did not affect the teaching and learning practices of control schools. If this happened, then pupils in control
schools would also have been applying growth mindset principles to their work, for example, through being willing to try
difficult tasks. The implication is, therefore, that our study is likely to understate the overall effect of the Changing
Mindsets intervention because it is only a comparison of a ‘structured package’, as described in the intervention section,
to approaches based on the same theoretical underpinnings, and not a comparison of the intervention to thecounterfactual of no growth mindset approach at all.
We are also aware, more widely, from our visits to schools for other research projects, that Growth Mindset principles
and messages often form part of schools’ ethos, delivered in statements, posters, and other forms of messaging.
Explanation 3: The pupils were too young or that results are longer term rather than short term
A further explanation could be that the pupils were too young to embed and achieve results, and that these could only
be achieved in the longer term. Some support for this explanation is found in interviews with teachers who said theprogramme should start at a younger age before pupils take on a fixed mindset. It is also possible that, at age eleven,
pupils may be too young to self-direct their learning in a way which is possible for older pupils and that the programme
takes longer to impact on academic performance. There may be intermediate steps between adopting a growth mindset
and improving academic achievement which younger pupils are less able to utilise. These may include, for example,
understanding strengths and weaknesses, when to seek help, how to express the help they need, and then to listen
and process the feedback. These abilities may require a degree of maturity and self-understanding which may not be
possible for children of primary school age. This would then explain why other studies based on U.S. and Norwegiandata find positive effects of a growth mindset approach in older students.

43

Changing Mindsets
Evaluation Report

Teachers expressed the view that the principles underpinning the programme could take time to embed and cited
examples where pupils had struggled to make links between the content of different lessons and had needed to beprompted to recall their earlier learning. Some teachers were, therefore, of the view that it might be necessary for pupils
to be taught growth mindset concepts and messages over a number of years for an impact on attainment to be achieved.
It was also felt important that the messages are reinforced at home, which may not have been achieved during the lifeof the project.
Finally, it might be speculated whether good performance in SATS is less dependent on attitude and mindset and more
directly on the quality of teaching. Therefore, while, in theory, a child with a growth mindset might compensate for poorer
teaching than one without, this requires understanding of gaps in knowledge and skills which a child of ten or eleven isunlikely to have. Therefore, having a growth mindset, while benefitting learning, may not improve SATS results. Olderchildren, in comparison, are more likely to self-direct their learning in and out of school through reading and engagementin learning outside of the classroom in ways which improve test performance.

Limitations
The implementation and process evaluation provides insights into the limitations of the evaluation which are related tothe design of the intervention. The first is that the intervention might be found to have a longer-term impact on pupil
performance as the messages become embedded and influence attitudes, behaviour, and learning. In particular, growthmindset might have more potential to affect the learning behaviour of older pupils who take more responsibility for theirown learning, are more able to identify their strengths and weaknesses and seek help to improve their performance.
The experiences of implementing the programme, its effects on staff, and its perceived impact on pupils were found tobe fairly similar between schools and were very positive. This reflects the findings of the earlier project which also foundthat growth mindset was fully understood and well received. Therefore, on balance, we feel its limited impact is largely
due to the ubiquitous nature of either growth mindset or its messages. The trial excluded schools that had delivered agrowth mindset programme but was not able to exclude schools and teachers who were at least reasonably aware ofits messages, and had almost certainly been using them with pupils. It is possible that it might have a stronger impact
on schools where awareness of growth mindset is very low but, because of its popularity and appeal, these may be
hard to find.

Future research and publications
This is the second EEF project on growth mindset; neither of the evaluations found an impact on pupil attainment overall.
However, both were relatively short term interventions and pupil attainment was measured within a relatively shortperiod following the end of the programme. It is possible that a more prolonged and in-depth intervention, with regularreinforcement, may yield stronger impacts in the longer term. At the same time, growth mindset is now a well-known
theory which is likely to have been adopted by many schools and teachers, so that establishing a counterfactual is likely
to be difficult. Before further growth mindset projects are commissioned, it would be advisable to carry out a survey ofschools to establish the extent to which it is used either by teachers or in whole-school approaches.

44

Changing Mindsets
Evaluation Report

References
Allen, R., Jerrim, J., Parameshwaran, M. and Thompson, D. (2018) ‘Properties of commercial tests in the EEF
database’, London: EEF Research Paper No. 001, February 2018.
Aronson, J., Fried, C. B. and Good, C. (2002) ‘Reducing the effects of stereotype threat on African American college
students by shaping theories of intelligence’, J. Exp. Soc. Psychol., 38 (2), pp. 113–125.
Bettinger, E., Ludvigsen, S., Rege, M., Solli, I. F. and Yeager, D. (2018) ‘Increasing perseverance in math: Evidence
from a field experiment in Norway’, Journal of Economic Behavior & Organization, 146, pp. 1–15.
Crawford, C., Edwards, A., Farquharson, C., Greaves, E., Trevelyan, G., Wallace, E. and White, C. (2016) ‘Magic
Breakfast: Evaluation Report and Executive Summary’, London: EEF.
Blackwell, L. S., Trzesniewski, K. H. and Dweck, C. S. (2007) ‘Implicit theories of intelligence predict achievement
across an adolescent transition: a longitudinal study and an intervention’, Child Dev., 78 (1), pp/ 246–263.
Duncan, T. G. and McKeachie, W. J. (Department for Education, 2015a) ‘KS1 and KS2 test and examination pointscores used in the 2015 school performance tables: The making of the motivated strategies for learning questionnaire’,
Educational Psychologist, 40 (2), pp. 117–128.
Dweck, C. S. (2008) Mindset: The new psychology of success, Random House Digital.
Good, C., Aronson, J. and Inzlicht, M. (2003) ‘Improving adolescents’ standardized test performance: An intervention
to reduce the effects of stereotype threat’, Journal of Applied Developmental Psychology, 24 (6), pp. 645–662.
Heckman, J. J., Stixrud, J. and Urzua, S. (2006) ‘The effects of cognitive and noncognitive abilities on labor market
outcomes and social behavior’, Journal of Labor economics, 24 (3), pp. 411–482.
Moote, J., Williams, J. and Sproule, J. (2011) ‘Investigating strategies to develop self-regulated learning in young
science students’, presented at the Developmental Psychology Section Annual Conference, Northumbria University,
Newcastle upon Tyne, 7–9 September 2011.
Moote, J., Williams, J. and Sproule, J. (2013) ‘When students take control: Investigating the impact of the CREST
inquiry-based learning program on self-regulated processes and related motivations in young science students’, Journalof Cognitive Education and Psychology, 12 (2), pp. 178–197.
Paunesku, D., Walton, G. M., Romero, C., Smith, E. N., Yeager, D. S. and Dweck, C. S. (2015) ‘Mind-set interventions
are a scalable treatment for academic underachievement’, Psychological Science, 26 (6), pp. 784–793.
Pintrich, P. R. and De Groot, E. V. (1990) ‘Motivational and self-regulated learning components of classroom academic
performance’, Journal of Educational Psychology, 82 (1), pp. 22–40.
Rienzo, C., Rolfe, H. and Wilkinson, D. (2015) ‘Changing Mindsets: Evaluation Report and Executive Summary’,London: EEF.
Sisk, V. F., Burgoyne, A. P., Sun, J., Butler, J. L. and Macnamara, B. N. (2018) ‘To what extent and under which
circumstances are growth mind-sets important to academic achievement? Two meta-analyses’, Psychological
Science, 29 (4), pp. 549–571.
Yeager, D. S. and Dweck, C. S. (2012) ‘Mindsets that promote resilience: When students believe that personal
characteristics can be developed’, Educational Psychologist, 47 (4), pp. 302–314.
Yeager, D. S., Romero, C., Paunesku, D., Hulleman, C. S., Schneider, B., Hinojosa, C. et al. (2016) ‘Using design
thinking to improve psychological interventions: the case of the growth mindset during the transition to high school’, J.
Educ. Psychol., 108 (3), p. 374.
45

Changing Mindsets
Evaluation Report

Appendix A: EEF cost rating
Cost ratings are based on the approximate cost per pupil per year of implementing the intervention over three years.
More information about the EEF’s approach to cost evaluation can be found here. Cost ratings are awarded as follows:
Cost rating
Description£££££
Very low: less than £80 per pupil per year.
£££££
Low: up to about £200 per pupil per year.
£££££
Moderate: up to about £700 per pupil per year.
£££££
High: up to £1,200 per pupil per year.
£££££
Very high: over £1,200 per pupil per year.

46Changing Mindsets
Evaluation Report

Appendix B: Security classification of trial findings
Outcomes: KS2 Maths, KS2 Reading, KS2 Grammar, Punctuation and Spelling

Rating

Initialscore

Criteria for rating
DesignPower

Attrition11

Well conducted experimental
design with appropriate analysis

MDES < 0.2

0-10%

Adjust

Final
score

5

4

3

2

1

Fair and clear quasi-experimental
design for comparison (e.g. RDD)
with appropriate analysis, or
experimental design with minor
concerns about validity

MDES < 0.3

Well-matched comparison (using
propensity score matching, orsimilar) or experimental design
with moderate concerns about
validity

MDES < 0.4

Weakly matched comparison or
experimental design with major
flaws

4
11-20%

4
Adjustment
for Balance[0]

MDES < 0.5

21-30%

31-40%

Adjustment
for threats tointernal
validity[0]Comparison group with poor or
no matching (E.g. volunteer
versus others)

MDES < 0.6

41-50%

No comparator

MDES > 0.6

over 50%

0

•

•

•

•

Initial padlock score: lowest of the three ratings for design, power and attrition = This was a well randomised
design with MDES at randomisation of 0.19. Attrition rates for the primary outcomes were 11.24% for Maths,
11.58% for Reading, and 11.36% for Grammar, Punctuation and Spelling.
Reason for adjustment for balance (if made): baseline imbalance in pre-tests for the sample as randomised
were small ranging between 0 (Reading) and -0.06 (Writing). In the sample as analysed, these differences werealso small ranging between -0.03 (Writing) and 0.03 (Reading). These are accounted for analytically in the
ANCOVA model.
Reason for adjustment for threats to validity (if made): No substantive threats to validity are reported, butboth the control and intervention group were acquainted with the theoretical underpinning of the “Changing
Mindsets” approach, with the majority of the schools suggesting they had used similar approaches in the past
(See Section on “Control group activity”). This implies that the intervention was compared with a relatively active
business as usual. This does not affect the security of the finding, but should be considered carefully for its
interpretation (See Section on “Interpretation”)
Final padlock score: initial score adjusted for balance and internal validity = 4 padlock

11 Attrition should be measured at the pupil level (even for clustered trials) and from the point of randomisation to the point of analysis.

47

Changing Mindsets
Evaluation Report

Appendix C: Extract from the training material48

Changing Mindsets
Evaluation Report

Appendix D: Consent forms

Department of Psychology
King Henry Building, King Henry I st.
PORTSMOUTH, PO1 2DY

Principal Investigator: Professor Sherria Hoskins
Telephone:

023 9284 6321

Email:

sherria.hoskins@port.ac.uk

Dear Head Teacher,

Many thanks for your interest in the Changing Mindsets project, funded by the Education Endowment
Foundation (EEF) and run by a team of researchers located in the Department of Psychology at the
University of Portsmouth. As part of this project we are also working with a team of researchers at the
National Institute of Economic and Social Research (NIESR) who have been recruited by the EEF toevaluate the success of the project. Although you have already expressed your interest to take part, we
now need to ask for your formal, informed consent to progress with the project.

The Project
We are asking that you and some of your teachers attend one of a number of training sessions acrossthe UK. In these sessions, we will be training staff to understand the concept of ‘Mindsets’, pioneeredby Professor Carol Dweck. We are interested in whether a teacher training approach combined withspecially designed learning resources and activities can enhance pupils’ mindsets and increase
educational attainment.

Dweck suggests that it is not ability that predicts resilience and perseverance in the face of challenge
and failure; rather it is the individual’s belief about the nature of ability (their mindset). Research
identifies two types of intelligence belief, namely growth mindset and fixed mindset. Individuals witha growth mindset believe that they can develop their intelligence, while individuals with a fixed mindset
believe that their intelligence is innate (i.e. “I was born this way/this is what I am”).

49Changing Mindsets
Evaluation Report

The growth mindset approach has been widely researched in the US and has been linked to increased
attainment and improved pupil motivation, classroom behaviour and reports of life-satisfaction. It is very
popular in the US and widely implemented, but despite this, there has been limited systematic researchon mindsets in the UK. As such, we are interested to see whether an approach that is so widely usedin the USA can have an impact when delivered directly by teachers in the UK.

During the training session, teachers will be: introduced to mindset theory and evidence; shown how toembed the approach into their classrooms; and given materials to run an 8 week intervention
programme of weekly lessons (approximately 2 hours each).

Format of the Project
Half of the schools involved in the project will receive teacher training in September 2016, and carry out
the programme in the autumn term with their Year 6 pupils; they will act as the ‘intervention group’. The
remaining schools will be invited to attend training days in the Summer term of 2017; they will act as
the ‘waiting control group’ and carry out the programme in the summer term with their Year 5 pupils (orYear 4 pupils if mixed Y5/Y6 classes). All schools (regardless of group) will be collecting data from Year
6 pupils only. Schools cannot be given the option of attending the training in the autumn term or summer
term; this will be decided at random by NIESR and is critical to the scientific design of this trial.

Assessing the Project
To determine whether the project has been a success we will be comparing the difference between
pupils’ Key Stage 1 and Key Stage 2 SATs results in schools that have and have not received Mindset
training. For the attainment, we will be focusing on numeracy and literacy. Attainment data will begathered via an application to the Department for Education’s National Pupil Database (NPD).
We are also interested in the impact of the project on pupils’ mindset and approach to learning (e.g.
their motivation, effort and help seeking behaviours). To do this, we are using two questionnaires; youwill find a copy of these questionnaires in your Welcome Pack and we ask that you administer them toyour current Year 5 (next year’s Year 6) pupils now and in April 2017 and return them to us oncompletion. Please don’t worry about what to do when; we will guide you through this throughout the
project.

Data
All data will be stored and managed in accordance with Data Protection regulations. We ask that before
sending pupils’ questionnaires to us, you replace their names with their Unique Pupil Number. This willallow us to collate data without being able to identify individual pupils, ensuring confidentiality. Data will
be entered into a password-protected, encrypted file and hard copies will be stored in a locked cabinet
in a locked archive room within the Department of Psychology at the University of Portsmouth for 10
years after any publications associated with it. After this period, all raw data will be destroyed.
The research project team at the University of Portsmouth will have ownership of the data, while theproject evaluators (NEISR) and EEF will have access to it. Our grant agreement with the EEF requires
us to share data with them and NIESR. However, the data will be shared in accordance with Data
Protection regulations (as above) and will be moved to them in encrypted files. Data collected and any
50Changing Mindsets
Evaluation Reportother pupil data will be treated with the strictest confidence. No individual school or pupil will be identified
in any report arising from the research.

Along with NIESR, we will be making a shared application to the National Pupil Database to accesssome archive data (e.g. Key Stage 1 SATs, Key Stage 2 SATs, gender, free school meal status). Again,
this will not include pupils’ names. It will be matched to the questionnaire data by the UPN. This ensuresthat at all times the data will be confidential and unidentifiable to any pupil by a third party.

Parental Information
We require that parents are fully informed about the project and will be given the chance to contact the
research team or you with any concerns. They will also be given the opportunity to opt their child out ofdata collection and project evaluation. You will find a copy of this letter in your Welcome Pack; we ask
that you send this home to parents and sign below to confirm doing so. If a parent expresses concerns
about the project we will withdraw their child from the project evaluation and data sharing.

Support and Debriefing
You can discuss the project at any point with the Changing Mindsets team at the University ofPortsmouth. In addition NIESR will evaluate the project as it progresses and collect your views on the
project via interviews and questionnaires. Further, there will be a debriefing session open to all staff
after all data has been collected, in the form of a free day long wash up conference.

Many thanks for your assistance with this project.

Dr Sherria Hoskins
Changing Mindsets Lead (University of Portsmouth)
I …………………………………………...........................(print name and role) give consent for the
Changing Mindsets Project to take place at ……………………………………………………...........

(name of school) and do so knowing that:•

We are free to withdraw our school from the intervention at any point.

•

Pupil questionnaire data will not contain pupils’ names; the school will provide Unique Pupil
Numbers for each pupil, ensuring their data remains confidential and unidentifiable to a third
party.
51

Changing Mindsets
Evaluation Report•

All data will be stored and shared securely and will be handled in accordance with the Data
Protection Act.

•The research project team at the University of Portsmouth will have ownership of the data,
while the project evaluators (NEISR) and EEF will have access to it. This will be stored
securely for 10 years after any publications associated with it. After this period, all raw data
will be destroyed.

•

Parents will be fully informed about the assessments and data sharing and will be given the
chance to discuss their concerns and ultimately withdraw their child from the data collection
phase of the project if they wish.

•

A shared application from University of Portsmouth and NIESR will be made to the National
Pupil Database to access archive data such as Key Stage 1 and Key Stage 2 results, gender,and free school meals status. This data will be matched to questionnaire data via Unique
Pupil Identifier provided by us. At no stage, will University of Portsmouth, EEF and NIESR
identify any pupils taking part in this project.

Please sign if you give your consent:………………………………………………………………………………………………………….
Please return the signed informed consent to Dr Frances Warren at the postal address
above or via email (indicating in the email that you give consent) tochangingmindsets@port.ac.uk

52

Changing Mindsets
Evaluation Report

Date
Dear Parent
I am writing to you about research we are doing on the Growth Mindsets lessons that are taught at [name ofschool]. I am attaching a short description of the main ideas informing these lessons and the results they aim
to achieve.
Along with my colleagues at the National Institute of Economic and Social Research (NIESR) I am carryingout additional research to measure the impact of the Growth Mindsets programme. I am interested in how
pupils have understood and engaged with the content of these lessons, and what they liked and disliked about
them. We believe it is important that the research takes account of pupils' views and experiences of thelessons.
We are planning to talk to pupils in small groups called focus groups. The focus groups will involve a few,
simple activities to be completed in pairs, with follow-up questions. Your child has been selected throughrandomised methods to take part, along with other pupils from year 6. To keep their identity anonymous,
teachers have randomly selected pupils from the class register. At no point in the research will we ask for
pupils' full names and they will remain anonymous throughout. I am leading the focus groups in all the schools
involved in Growth Mindsets. The views of pupils across all of the schools will be combined with the otherfindings in a report to the Education Endowment Foundation who are funding the project.
At the start of the focus group, which will happen at the same time as a normal lesson, I will explain the
purpose to your child and the others in the group and check they are happy to take part. They can decide at
any time that they no longer wish to take part. I will also let them know that I will not pass on what they say toanyone in the school or to you their parent. I will use what they say to write the report to the Education
Endowment Foundation, but will never refer to them by name in anything I write.
If you have any questions about Growth Mindsets or the research, please contact me on 020 7654 1929 or[name of lead at the school].
If you are happy for your child to participate in the focus group for this part of the research please complete
and return the section below to your child’s form tutor.
Child’s name: ………………………………………………………………………
Child’s class Teacher: ……………………………………………………………………..
I agree to my child taking part in the focus group
Parent name (BLOCK CAPITALS) ……………………………………………………
Parent signature: ……………………………………………………………………
Date ………………………………………………
Yours sincerely53Changing Mindsets
Evaluation Report

Jonathan Buzzeo

What is the Growth Mindsets programme?

The Growth Mindsets project is to do with the theories that children hold about their intelligence, inparticular whether it is a "fixed entity" or has a "malleable" quality that can develop and improve.

It has been argued that teachers can help children to adopt a growth mindset by praising them for theireffort and persistence, rather than their innate intelligence. Research evidence mainly from the US has
found that the use of these techniques by teachers can have a positive impact on student test results
and performance.

This approach is now being tested in UK schools through the Growth Mindsets project. It was developedby a team at the University of Portsmouth and is being funded by the Education Endowment Foundation.
The aim is to see whether this change in the teachers' classroom practice will positively affect pupils’
attainment.

To enable them to deliver the project, Year 6 teachers in your child’s school have been introduced tothe theory and evidence informing this approach, supported to embed the approach in their classroom,and provided with material to run Growth Mindsets lessons in their school this term.

54

Changing Mindsets
Evaluation Report

Appendix E: Mindset measure questions, MSLQ questions and subscales
Scoring of the ‘Mindset measure’ – Section 1
All items of the ‘Mindset measure’ are scored from 1-6, with 1 indicating ‘Strongly disagree’ and 6
indicating ‘Strongly agree’
Mindset measureMax
possiblescore1) You have a certain amount of intelligence, and you really can’t do much to changeit.
2) Your intelligence is something about you that you can’t change very much.
3)You can learn new things, but you can’t really change your basic intelligence.
Maximum possible score on Mindset measure=

Scoring of the ‘School Questionnaire’ – Section 2
All items on the adapted version of the MSLQ are scored from 1-7, with 1 indicating ‘Strongly disagree’
and 7 indicating ‘Strongly agree’. (NB 3 items on the self-regulation scale need reversing before totallingthe sub-scale score and the total score).
Sub-scales (Intrinsic value, self-efficacy, test anxiety and self-regulation) are then calculated as a meanscore.
Intrinsic valueMaxpossiblescore1) I prefer class work that is challenging so I can learn new things.
4) It is important for me to learn what is being taught in class.
5) I like what I am learning in class.
7) I think I will be able to use what I learn in one class in other classes.
10) I often choose class and homework activities I will learn something from even if
they require more work.
14) Even when I do poorly on a test I try to learn from my mistakes.
15) I think that what I am learning in class is useful for me to know.
17) I think what we are learning in class is interesting.
21) Understanding school subjects is important to me.
Maximum possible score on intrinsic value=55

Changing Mindsets
Evaluation Report

Self-efficacy

Max
possiblescore2) Compared with other pupils in class, I expect to do well.
6) I’m certain I can understand the ideas taught in class.
8) I expect to do very well in class.
9) Compared with others in class, I think I am a good pupil.
11) I am sure I can do an excellent job on the problems and tasks given in class.
13) I think I will receive a good mark in class.
16) My learning skills are excellent compared with others pupils in class.
18) Compared with other pupils in class I think I know a great deal about the subject.
19) I know that I will be able to learn the information in class.
Maximum possible score on self-efficacy=

Test anxiety

Maxpossiblescore3) I am so nervous during a test that I cannot remember facts I have learned.
12) I have an uneasy, upset feeling when I take a test.
20) I worry a great deal about tests.
22) When I take a test I think about how poorly I am doing.
Maximum possible score on test-anxiety=56

Changing Mindsets
Evaluation Report

Self-regulationMaxpossiblescore23) I ask myself questions to make sure I know the information I have been learning.
24) When work is hard I either give up or learn only the easy parts.*
25) I do extra work and practice exercises even when I do not have to.
26) Even when lessons are dull and uninteresting, I keep working until I finish.
27) Before I begin school work, I think about the things I will need to do to learn.
28) I often find that I have been reading but I do not know what it is all about.*29) I find that when the teacher is talking I think of other things and do not reallylisten to what is being said.*
30) When I am reading, I stop once in a while and go over what I have read.
31) I work hard to get a good mark even when I don’t like a subject.
Maximum possible score on self-regulation=* Reversing required – the raw scores on these 3 items must be reversed before totaling the subscale or total score.

57

Changing Mindsets
Evaluation Report

Appendix F: Code for randomisation
gen randSeq=uniform()
gen block=.
replace block=1 if newLoc==1 & tertileAdj==1
replace block=2 if newLoc==2 & tertileAdj==1
replace block=3 if newLoc==3 & tertileAdj==1
replace block=4 if newLoc==4 & tertileAdj==1
replace block=5 if newLoc==5 & tertileAdj==1
replace block=6 if newLoc==1 & tertileAdj==2
replace block=7 if newLoc==2 & tertileAdj==2
replace block=8 if newLoc==3 & tertileAdj==2
replace block=9 if newLoc==4 & tertileAdj==2
replace block=10 if newLoc==5 & tertileAdj==2
replace block=11 if newLoc==1 & tertileAdj==3
replace block=12 if newLoc==2 & tertileAdj==3
replace block=13 if newLoc==3 & tertileAdj==3
replace block=14 if newLoc==4 & tertileAdj==3
replace block=15 if newLoc==5 & tertileAdj==3#delimit ;lab def block
1 "Midlands, lower"
2 "North East, lower"
3 "North West, lower"
4 "South East, lower"
5 "South West, lower"
6 "Midlands, mid"
7 "North East, mid"
8 "North West, mid"
9 "South East, mid"58

Changing Mindsets
Evaluation Report

10 "South West, mid"
11 "Midlands, upper"
12 "North East, upper"
13 "North West, upper"
14 "South East, upper"
15 "South West, upper";#delimit cr
lab val block blocklab var block "Randomisation block"sort block randSeq
gen T=randSeq>.5
replace T=1-T[_n-1] if _n>1
lab def T 0 "Control" 1 "Treated"lab val T T
lab val T "Treated"

59Changing Mindsets
Evaluation Report

Appendix G: Effect size from simple models

Table G1: Effect size from the simple model for primary outcomes
Raw means: standardisedvariables
Intervention
group
OutcomeKS2 maths

n (missing)2263 (72)103.82(103.51;104.
13)
103.82(103.49;104.
16)2266 (69)105.87(105.56;106.
18)2273 (62)

KS2 reading
KS2grammarpunctuation
spelling

Mean (95%CI)

n (missing)Control group
Mean (95%CI)

2174 (75)104.07(103.77;104.
36)
103.70(103.35;104.
05)2182 (67)105.76(105.45;106.
07)

2181 (68)n in model(interventio
n; control)

Effect size
(95% CI)

4454(2273;2181)-0.01 ( -0.04;
0.02)
0.46

4437
(2263;2174)-0.00 ( -0.02;
0.02)
0.87

4448
(2266;2182)

0.00 ( -0.03;
0.03)
0.96

p-value

Table G2: Effect size for secondary outcomes
Raw means: standardised variables
Intervention group
Outcomen(missing)

Mean (95%CI)

n(missing)

Intrinsic value

1714 (621)

Self-efficacy

1714 (621)

Test anxiety

1714 (621)

Self regulation

1711 (624)

5.55(5.50;5.59)
5.11(5.06;5.16)
3.45(3.36;3.53)
4.77(4.72;4.82)

1726
(523)
1726
(523)
1726
(523)
1716
(533)

Control groupn in model
Mean (95% CI) (intervention;
control)

5.50(5.45;5.54)
5.17(5.12;5.22)
3.50(3.42;3.58)
4.72(4.67;4.78)

3440(1714;1726)
3440(1714;1726)
3440(1714;1726)
3427(1711;1716)

Effect size (95%
CI)

pvalu
e

0.07 ( -0.12; 0.26)

0.47-0.06 ( -0.22; 0.09)

0.43-0.00 ( -0.08; 0.07)

0.9

0.03 ( -0.13; 0.19)

0.68

Table G3: Effect size for primary outcomes – FSM subgroup
Raw means: standardisedvariables
Intervention
group
Outcomen (missing)

Mean (95%CI)

n (missing)Control group
Mean (95%CI)

n in model(interventio
n; control)

Effect size
(95% CI)

p-value60

Changing Mindsets
Evaluation Report

KS2 maths

782 (27)

KS2 reading
KS2grammarpunctuation
spelling

780 (29)101.81(101.28;102.
34)
101.46(100.91;102.
02)779 (30)103.82(103.29;104.
36)794 (32)

102.24(101.75;102.
74)
101.40(100.82;101.
98)

797 (29)104.13(103.61;104.
64)

797 (29)

1579(782;797)

0.00 ( -0.03;
0.03)
0.97

1574
(780;794)

0.01 ( -0.02;
0.03)
0.59

1576
(779;797)

0.00 ( -0.03;
0.04)
0.92

Table G4: Effect size for secondary outcomes – FSM subgroup
Raw means: standardised variables
Intervention group
Outcomen(missing)

Intrinsic value565 (244)

Self-efficacy

565 (244)

Test anxiety

565 (244)

Self regulation

563 (246)

Mean (95%CI)

n(missing)Control groupn in model
Mean (95% CI) (intervention;
control)

5.54(5.46;5.63)
5.03(4.94;5.12)
3.75(3.60;3.89)
4.74(4.65;4.84)

598
(228)
598
(228)
598
(228)
593
(233)

5.46
(5.38;5.55)
5.12(5.03;5.20)
3.65(3.51;3.79)
4.69(4.60;4.78)

1163(565;598)
1163(565;598)
1163(565;598)
1156(563;593)

Effect size (95%
CI)

pvalu
e

0.085 ( -0.14; 0.30)-0.119 ( -0.31;
0.08)

0.45

0.047 ( -0.05; 0.14)

0.34

0.008 ( -0.18; 0.20)

0.93

61

0.23

Changing Mindsets
Evaluation Report

Appendix H: Code for the analysis of primary outcomesThis code was used to produce Table 5. It is a slightly modified version of the code provided by Crawford
et al. (2016) in their report "Magic Breakfast: Evaluation Report and Executive Summary."
glo in "data"
glo out "data\final"
glo log "log"use "$out\dataCM.dta", clear*Main tabletempname file
file open `file' using `"maintable_primary.csv"', write text replace
file write `file' "Table A: Headline academic outcomes" _newfile write `file' " , Raw means: standardised variables, , , , Effect size, , " _newfile write `file' " , Intervention group, , Control group, , , , " _newfile write `file' "Outcome, n (missing), Mean (95% CI), n (missing), Mean (95% CI), n in model(intervention; control), Effect size (95% CI), p-value" _new
loc k1 " k1attain2 k1attain3 k1attainmiss"foreach yvar in ks2_matscore ks2_readscore ks2_gpsscore {if "`yvar'" == "ks2_matscore" local name = "KS2 maths"
if "`yvar'" == "ks2_readscore" local name = "KS2 reading"if "`yvar'" == "ks2_gpsscore" local name = "KS2 grammar punctuation spelling"sum `yvar' if dropout==0 & treated == 1
cii mean r(N) r(mean) r(sd)local tN = trim("`: display %6.0f r(N)'")
local tmean = trim("`: display %6.2f r(mean)'")local tcilo = trim("`: display %6.2f r(lb)'")local tcihi = trim("`: display %6.2f r(ub)'")count if drop == 0 & treated == 1
local tmiss_temp = r(N) - `tN'
local tmiss = trim("`: display %6.0f `tmiss_temp''")sum `yvar' if dropout==0 & treated == 062

Changing Mindsets
Evaluation Reportcii mean r(N) r(mean) r(sd)local cN = trim("`: display %6.0f r(N)'")
local cmean = trim("`: display %6.2f r(mean)'")local ccilo = trim("`: display %6.2f r(lb)'")local ccihi = trim("`: display %6.2f r(ub)'")count if drop == 0 & treated == 0
local cmiss_temp = r(N) - `cN'
local cmiss = trim("`: display %6.0f `cmiss_temp''")local totN = trim("`: display %6.0f `cN'+`tN''")
local paro (local parc )local semicn ;local col2 `tN' `paro'`tmiss'`parc'
local col3 `tmean' `paro'`tcilo'`semicn'`tcihi'`parc'
local col4 `cN' `paro'`cmiss'`parc'
local col5 `cmean' `paro'`ccilo'`semicn'`ccihi'`parc'
local col6 `totN' `paro'`tN'`semicn'`cN'`parc'
mixed `yvar' `k1' treated i.block || ks2_urn_treatment : , reml /*1*/
loc cilo="`: display %6.2f _b[treated]-invttail(e(N)+e(df_m)+1,0.025)*_se[treated]'"di `cilo'
loc cihi="`: display %6.2f _b[treated]+invttail(e(N)+e(df_m)+1,0.025)*_se[treated]'"di `cihi'
estat recovariancemat V=r(Cov2)
loc tau=V[1,1]estat icc
loc icc=r(icc2)
di `icc'
loc es="`: display %6.2f _b[treated]/(`tau'/`icc')'"di `es'loc escilo="`: display %6.2f `cilo'/(`tau'/`icc')'"di `escilo'
63Changing Mindsets
Evaluation Report

loc escihi="`: display %6.2f `cihi'/(`tau'/`icc')'"di `escihi'
local epval = "`: display %6.2f 2*(1-normprob(abs(_b[treated]/_se[treated])))'"
if `epval' <= 0.10 local sigstars = "*"if `epval' <= 0.05 local sigstars = "**"
if `epval' <= 0.01 local sigstars = "***"
if `epval' > 0.10 local sigstars = ""di `epval'
local col7 `es' `paro'`escilo'`semicn'`escihi'`parc'
local col8 `epval'`sigstars'
file write `file' "`name'" "," "`col2'" "," "`col3'" "," "`col4'" "," "`col5'" "," "`col6'" "," "`col7'" "," "`col8'" _new}log close64

Changing Mindsets
Evaluation Report

Appendix I: Histograms of post-test outcomes

Figure I1: Distribution of KS2 maths scores for the pupils in the Changing Mindsets intervention
schools

Figure I2: Distribution of KS2 reading scores for the pupils in the Changing Mindsets intervention
schools

Figure I3: Distribution of KS2 GPS scores for the pupils in the Changing Mindsets intervention schools

65Changing Mindsets
Evaluation Report

Figure I4: Distribution of intrinsic value post-test subscale for the pupils in the Changing Mindsetsintervention schools

Figure I5: Distribution of self-efficacy post-test subscale for the pupils in the Changing Mindsetsintervention schools

66Changing Mindsets
Evaluation Report

Figure I6: Distribution of anxiety post-test subscale for the pupils in the Changing Mindsetsintervention schools

Figure I7: Distribution of self-regulation post-test subscale for the pupils in the Changing Mindsetsintervention schools

67Changing Mindsets
Evaluation Report

Appendix J: Additional subgroup analysis
In this appendix we present the results of separate analysis for three subgroups: females, males andpupils with fixed mindset in the pre-trial tests. These analyses were not pre-specified in the protocol
nor in the SAP and as such they are exploratory attempts to understand whether the intervention had
an effect on groups that showed promising results in other studies. The results of these exploratory
analyses show that there is no evidence of an impact of the Changing Mindset intervention on the tested
cognitive and non-cognitive outcomes of Year 6 students in these subgroups.

Table J1: Effect size of the intervention on literacy and numeracy among females
Raw means: standardised variables
Intervention group
OutcomeKS2 maths

n(missing)

1129 (21)

KS2 reading
1123 (27)KS2grammarpunctuation
spelling
1125 (25)

Mean (95%CI)

103.48(103.07;103.
89)
104.39(103.94;104.
84)
106.62(106.21;107.
04)n(missin
g)

1061
(18)
1058
(21)
1061
(18)

Control groupn in model
Mean (95% CI) (intervention;
control)

103.65(103.24;104.05)
104.52(104.01;105.03)
106.65(106.22;107.09)

Effect size (95%
CI)

pvalu
e

2190
(1129;1061)-0.00 ( -0.03; 0.03) 1

2181(1123;1058)-0.00 ( -0.02; 0.02) 0.852186(1125;1061)

0.00 ( -0.02; 0.03)

0.74

Effect size (95%
CI)

pvalu
e

Table J2: Effect size of the intervention on literacy and numeracy among males
Raw means: standardised variables
Intervention group
OutcomeKS2 maths

n(missing)

1144 (41)

KS2 reading
1140 (45)KS2grammarpunctuation
spelling
1141 (44)Mean (95%CI)

104.15(103.70;104.
60)
103.26(102.78;103.
75)
105.12(104.67;105.
58)

n(missin
g)

1120
(50)
1116
(54)
1121
(49)

Control groupn in model
Mean (95% CI) (intervention;
control)

104.46(104.03;104.89)
102.92(102.44;103.41)
104.91(104.47;105.35)

2264(1144;1120)-0.02 ( -0.05; 0.01) 0.262256(1140;1116)-0.00 ( -0.02; 0.02) 0.932262(1141;1121)-0.00 ( -0.03; 0.03) 0.86

Table J3: Effect size of the intervention on literacy and numeracy among pupils with fixed mindset

68Changing Mindsets
Evaluation Report

Raw means: standardised variables
Intervention group
OutcomeKS2 maths

n(missing)

504 (12)

KS2 reading
502 (14)
KS2
grammarpunctuation
spelling
502 (14)

Mean (95%CI)

102.33(101.69;102.
96)
102.27(101.60;102.
95)
104.45(103.81;105.
08)n(missing)Control groupn in model
Mean (95% CI) (intervention;
control)

436(10)102.91(102.27;103.56)940(504;436)-0.01 ( -0.05; 0.02) 0.52

433
(13)

101.99(101.23;102.74)935(502;433)

0.01 ( -0.03; 0.04)

436
(10)

104.70(104.05;105.36)

938(502;436)-0.01 ( -0.05; 0.03) 0.71

Effect size (95%
CI)

pvalu
e

0.75

Table J4: Effect size of the intervention on non-cognitive skills among females
Raw means: standardised variables
Intervention group
Outcomen(missing)

Intrinsic value864 (286)

Self-efficacy

864 (286)

Test anxiety

864 (286)

Self regulation

863 (287)

Mean (95%CI)

n(missing)Control groupn in model
Mean (95% CI) (intervention;
control)

5.64(5.58;5.70)
5.09(5.02;5.16)
3.79(3.67;3.91)
4.85(4.78;4.92)

853
(226)
853
(226)
853
(226)
849
(230)

5.57
(5.51;5.64)
5.14(5.06;5.21)
3.78(3.67;3.90)
4.79(4.71;4.86)

1717(864;853)
1717(864;853)
1717(864;853)
1712(863;849)

Effect size (95%
CI)

pvalu
e

0.110 ( -0.13; 0.34) 0.35
-0.008 ( -0.20;
0.19)
0.94
-0.013 ( -0.11;
0.08)
0.78
0.077 ( -0.11; 0.26) 0.41

Table J5: Effect size of the intervention on non-cognitive skills among males
Raw means: standardised variables
Intervention group
Outcomen(missing)

Intrinsic value

850 (335)

Self-efficacy

850 (335)

Test anxiety

850 (335)

Self regulation

848 (337)

Mean (95%CI)

n(missing)Control groupn in model
Mean (95% CI) (intervention;
control)

5.45(5.38;5.52)
5.13(5.06;5.21)
3.09(2.98;3.20)
4.69(4.61;4.76)

873
(297)
873
(297)
873
(297)
867
(303)

5.42
(5.35;5.49)
5.21(5.14;5.28)
3.23(3.12;3.34)
4.66(4.59;4.73)

83 (41;42)

Effect size (95%
CI)

pvalu
e

83 (41;42)

0.090 ( -0.11; 0.29) 0.39
-0.092 ( -0.27;
0.08)
0.31
-0.028 ( -0.13;
0.07)
0.59

83 (41;42)

0.047 ( -0.12; 0.22) 0.59

83 (41;42)69

Changing Mindsets
Evaluation Report

Table J6: Effect size of the intervention on non-cognitive skills among pupils with fixed mindset
Raw means: standardised variables
Intervention group
Outcomen(missing)

Intrinsic value375 (141)

Self-efficacy

375 (141)

Test anxiety

375 (141)

Self regulation

375 (141)

Mean (95%CI)

n(missing)Control groupn in model
Mean (95% CI) (intervention;
control)

5.51(5.41;5.61)
5.07(4.96;5.18)
3.65(3.46;3.83)
4.71(4.59;4.82)

341
(105)
341
(105)
341
(105)
338
(108)

5.45
(5.34;5.56)
5.06(4.94;5.17)
3.81(3.62;4.00)
4.64(4.53;4.76)

Effect size (95%
CI)

pvalu
e

716 (375;341)

0.180 ( -0.07; 0.42) 0.16716 (375;341)
716 (375;341)

0.067 ( -0.16; 0.29) 0.55
-0.029 ( -0.16;
0.10)
0.66

713 (375;338)

0.078 ( -0.14; 0.30) 0.4870

Changing Mindsets
Evaluation Report

Appendix K: Results of the missing data analysis and additional
estimates

Table K1: Logistic regression analysis of missing data in primary outcomesEver FSM

FemaleEthnicity group: Asian

Ethnicity group: Black

Ethnicity group: Chinese

Ethnicity group: Mixed

Ethnicity group: Unclassified

Ethnicity group: White

Treatment allocation==1

North East, lower

North West, lower

South East, lower

South West, lower

Midlands, mid

North East, mid

Maths score-0.479*(0.246)0.052-0.081(0.145)0.576
1.240(1.092)0.256
0.946(1.031)0.359
1.548(1.643)0.346
1.924*(1.118)0.085
1.815(1.402)0.195
2.329**(1.136)0.040-1.098*(0.640)0.086-0.223(0.416)0.592-1.201(1.128)0.287
1.611**(0.807)0.046
2.773***(0.982)0.005-0.227(0.447)

Reading Score-0.518**(0.236)0.028-0.059(0.137)0.669
0.925(0.881)0.294
0.710(0.845)0.401
1.988(1.371)0.147
1.506(0.940)0.109
1.422(1.244)0.253
1.989**(0.949)0.036-1.019(0.625)0.103-0.531(0.425)0.211-0.351(0.471)0.457
1.406*(0.765)0.066-1.486(1.147)0.195
2.493***(0.966)0.010-0.537(0.517)GPS score-0.451*(0.250)0.071-0.068(0.139)0.623
0.961(0.871)0.269
0.610(0.814)0.453
1.117(1.522)0.463
1.460(0.921)0.113
1.387(1.227)0.258
1.962**(0.929)0.035-0.983(0.634)0.121-0.540(0.484)0.265-1.316(1.102)0.232
1.566**(0.772)0.043-1.257(1.112)0.258
2.573***(0.989)0.009-0.347(0.518)71

Changing Mindsets
Evaluation Report

North West, midSouth East, midSouth West, mid

Midlands, upper

North East, upper

North East, upperSouth East, upperSouth West, upper

School-level KS1 mean achievement

KS1 (11-14.99 points)

KS1 (15-16.99 points)

KS1 (>=17 points)

KS1 missing

0.612-0.520(0.753)0.490
2.658*(1.605)0.098-0.859*(0.512)0.093-0.591(0.816)0.469
2.681**(1.172)
0.022
-

0.299
-0.823(0.753)0.274
2.330(1.593)0.144-1.148**(0.518)0.027-1.130(0.998)0.257
2.377**(1.166)
0.041
-

0.503
-0.657(0.752)0.383
2.456(1.610)0.127-0.985*(0.539)0.068-0.977(1.022)0.339
2.481**(1.190)0.037-

2.573*(1.517)0.090-0.413(0.953)0.665-0.297(0.523)0.570-1.506***(0.332)0.000-1.760***(0.355)0.000-1.733***(0.330)0.000
0.020(0.312)
0.948

2.157
(1.542)0.162-0.324(0.960)0.736-0.252(0.520)0.628-1.639***(0.323)0.000-1.904***(0.355)0.000-1.875***(0.320)0.000
0.105(0.297)
0.724

2.286
(1.562)0.143-0.182(0.986)0.854-0.223(0.538)0.678-1.547***(0.335)0.000-1.774***(0.363)0.000-1.734***(0.327)0.000
0.196(0.329)0.552

Observations
4,651
4,728
4,728
The table reports estimates, standard errors (in parenthesis) and p-values of logistic regressions.

72

Changing Mindsets
Evaluation Report

Table K2: Logistic regression analysis of missing data in secondary outcomesPre-test second tertile

Pre-test third tertile

Pre-test second missing

Ever FSM

FemaleEthnicity group: Asian

Ethnicity group: Black

Ethnicity group: Chinese

Ethnicity group: Mixed

Ethnicity group: Unclassified

Ethnicity group: White

Treatment allocation==1

North East, lower

North West, lower

South East, lower

South West, lower

Midlands, mid

Testanxiety-0.063(0.126)0.620
0.150(0.115)0.193
1.096***(0.338)0.001
0.022(0.133)0.871-0.148**(0.073)0.043
0.395(0.422)0.348-0.164(0.354)0.644
0.257(0.601)0.669
0.244(0.532)0.647
0.078(0.592)0.895-0.047(0.534)0.930
0.212(0.411)0.607-1.503**(0.652)0.021-1.082**(0.501)0.031
0.009(0.683)0.989-0.867(0.663)
0.191
1.017

Selfregulation-0.013(0.096)0.891
0.007(0.101)0.948
1.050***(0.321)0.001
0.036(0.133)0.789-0.136*(0.076)0.075
0.411(0.430)0.339-0.163(0.354)0.645
0.251(0.605)0.678
0.233(0.532)0.661
0.042(0.589)0.943-0.041(0.535)0.939
0.202(0.407)0.620-1.431**(0.657)0.029-1.116**(0.496)0.024
0.041(0.676)0.952-0.881(0.662)
0.183
1.041

Intrinsic
value-0.185*-0.1080.085-0.15-0.1330.259
0.953***-0.341
0.005
0.03-0.1340.821-0.128*-0.0730.081
0.36-0.4290.402-0.21-0.3480.546
0.208-0.611
0.734
0.197
-0.53
0.71
0.019
-0.5920.974-0.086-0.5340.872
0.213-0.4110.604-1.481**
-0.65
0.023
-1.081**-0.4980.03
0.009-0.6850.99-0.863
-0.66
0.191
1.029

Selfefficacy-0.079-0.0790.317-0.024-0.12
0.841
1.027***-0.3280.002
0.028-0.1350.838-0.132*-0.0740.074
0.377-0.4260.377-0.183-0.3480.598
0.232-0.6050.702
0.223-0.5280.672
0.043-0.5920.942-0.06-0.5290.91
0.214-0.4110.602-1.488**
-0.65
0.022
-1.087**
-0.5
0.03
0.011
-0.6830.987-0.883-0.664
0.184
1.024

73

Changing Mindsets
Evaluation Report

North East, mid

North West, midSouth East, midSouth West, mid

Midlands, upper

North East, upper

North East, upperSouth East, upperSouth West, upper

School-level KS1 meanachievement

KS1 (11-14.99 points)

KS1 (15-16.99 points)

KS1 (>=17 points)

KS1 missing(0.799)0.203-0.616(0.718)0.391-0.123(0.660)0.852
0.174(0.790)0.826-2.277***(0.521)0.000-0.555(0.745)0.457-0.175(0.681)0.797-1.438**(0.713)0.044-0.721(0.905)0.426
1.490(1.149)0.195(0.792)0.189-0.617(0.718)0.390-0.129(0.660)0.845
0.183(0.783)0.815-2.267***(0.518)0.000-0.527(0.735)0.474-0.169(0.681)0.804-1.437**(0.709)0.043-0.727(0.903)0.421
1.506(1.154)0.192-0.7980.197-0.607-0.7190.399-0.122-0.6570.852
0.179-0.7870.82-2.276***-0.5190-0.557-0.7450.455-0.168-0.6810.805-1.440**-0.7160.044-0.733-0.8990.414
1.502-1.151
0.192-0.7970.199-0.622-0.72
0.388-0.125
-0.66
0.85
0.175
-0.7880.824-2.271***-0.5220-0.563-0.7440.449-0.172-0.68
0.8-1.440**-0.7120.043-0.728-0.9030.42
1.498-1.153
0.194

0.189

0.188

0.185

0.188

(0.249)0.448-0.394**(0.161)0.014-0.353**(0.155)0.023-0.296*(0.153)0.053-0.052(0.180)0.772(0.248)0.447-0.384**(0.160)0.016-0.376**(0.156)0.016-0.331**(0.152)0.029-0.067(0.179)
0.708

-0.25
0.459
-0.393**-0.161
0.015-0.354**-0.1570.024-0.304**-0.1530.047-0.056-0.181
0.756

-0.25
0.452
-0.398**-0.1610.013-0.364**-0.1550.019-0.316**-0.151
0.037-0.063-0.181
0.728

4,875
4,875
4,875
4,875
Observations
The table reports estimates, standard errors (in parenthesis) and p-values of logistic regressions.

Table K3: Primary analysis with additional variables

74

Changing Mindsets
Evaluation Report
Raw meansIntervention group
OutcomeKS2 maths

n
(miss
ing)

2273
(62)

2263
KS2 reading(72)
KS2
grammarpunctuation2266
spelling(69)

Mean (95%CI)

103.82(103.51;104.13)
103.82(103.49;104.16)
105.87(105.56;106.18)

Control group
n
(miss
ing)

2181
(68)
2174
(75)
2182
(67)

Mean (95%CI)

104.07(103.77;104.36)
103.70(103.35;104.05)
105.76(105.45;106.07)

n in model(intervention;
control)

4454(2273;2181)
4437(2263;2174)
4448(2266;2182)

Effectsize (95%CI)-0.01 ( 0.04;
0.01)-0.00 ( 0.02;
0.02)-0.00 ( 0.03;
0.02)

pval
ue

0.3
2
0.6
6
0.8
7

Table K4: Secondary analysis with additional variables

OutcomeIntrinsic
value
Selfefficacy
Test
anxiety
SelfregulationRaw meansIntervention
group
n
n
Mean
(missi(missi(95% CI)
ng)
ng)
1508
5.54
1409
(827)
(5.49;5.5
(840)
9)
1507
5.13
1409
(828)
(5.08;5.1
(840)
8)
1507
3.46
1409
(828)
(3.37;3.5
(840)
5)
1501
4.77
1398
(834)
(4.72;4.8(851)
2)

Control groupMean(95% CI)
5.51(5.46;5.56)
5.21(5.15;5.2
6)
3.46(3.37;3.5
5)
4.74(4.69;4.8
0)n in model(intervention;
control)
2917 (1508;1409)

Effect size
(95% CI)
0.09 (-0.10;
0.29)

pvalu
e
0.39

2916 (1507;1409)-0.06 (-0.23;
0.11)

0.51

2916 (1507;1409)-0.01 (-0.09;
0.06)

0.72

2899 (1501;1398)

0.05 (-0.11;
0.22)

0.54

75

You may re-use this document/publication (not including logos) free of charge in any format or
medium, under the terms of the Open Government Licence v3.0.
To view this licence, visit https://nationalarchives.gov.uk/doc/open-government-licence/version/3 oremail: psi@nationalarchives.gsi.gov.uk
Where we have identified any third-party copyright information you will need to obtain permission fromthe copyright holders concerned. The views expressed in this report are the authors’ and do notnecessarily reflect those of the Department for Education.
This document is available for download at https://educationendowmentfoundation.org.uk

The Education Endowment Foundation
5th Floor, Millbank Tower
21–24 Millbank
LondonSW1P 4QP

https://educationendowmentfoundation.org.uk@EducEndowFoundn
Facebook.com/EducEndowFoundn