EUROPEAN COMMISSION Brussels, 31.1.2018 SWD(2018) 40 final COMMISSION STAFF WORKING DOCUMENT Mid-term evaluation of the Erasmus+ programme (2014-2020) Accompanying the document REPORT FROM THE COMMISION TO THE EUROPEAN PARLIAMENT, THE COUNCIL, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF REGIONS Mid-term evaluation of the Erasmus+ programme (2014-2020) {COM(2018) 50 final}



Table of Contents 1.Introduction 2.Background to Erasmus+ and its predecessor Programmes 3.Implementation of Erasmus+ until 2016 4.Method of the evaluation 5.Findings 5.1.Effectiveness 5.1.1.Outcomes for learners and practinioners………………………………........20 5.1.2.Outcomes for organisations and systems……………………………….......31 5.1.3. Transversal questions about effectiveness………………………………….40 5.2.Relevance 5.3.Coherence 5.4.Efficiency and simplification 5.5.European added value72 6.Conclusions Annex 1: procedural information 78

Annex 2: stakeholder consultation 82 Annex 3: methods and analytical models91 Annex 4: evaluation: criteria and questions103 Annex 5a: intervention logic of Erasmus+ and its predecessor programmes107 Annex 5b: factsheet on Erasmus+113 Annex 5c: specific objectives of Erasmus+115 Annex 5d: factsheets on predecessor programmes117 Annex 5e: legal basis indicators127 Annex 6: glossary 135



Executive Summary The 2014-2020 Erasmus+ programme and its predecessor programmes (2007-2013) have provided millions of people in Europe and beyond with opportunities to learn, volunteer or teach abroad. It is one of the EU's best-known successes. Its impact goes beyond individuals, as it also has a positive impact on education, training, youth and sport organisations or related systems and policies in developing cross-country cooperation. This mid-term evaluation of Erasmus+ looks at the current programme until the end of 2016. It also includes an evaluation of the long-term effects of the predecessor programmes. It is based on a very reliable methodology (see 5.3 for its strengths and limitations) and over a million responses from interested parties. It shows that at mid-term, the programme is on track to achieve or exceed the vast majority of the targets set in the Erasmus+ Regulation (EU) No 1288/2013. Effectiveness and European added value 1) The evaluation finds that the Erasmus+ programme is highly valued by the general public as well as by its stakeholders. This finding is linked to the available evidence that the programmes under evaluation – both under their intra-EU and external dimensions - achieve a broad range of concrete and positive impacts on their beneficiaries: learners (students, apprentices, volunteers, young people, etc.); practitioners (teachers, trainers, youth workers, staff, etc.); participating organisations (schools, universities, youth and sport organisations, providers of vocational education and training and of adult education, etc.). 2) Though less visible, the evaluation confirms the systemic effect of the evaluated programmes on education, training, youth and sport policies and systems, directly through the critical mass reached at least in the higher education sector or indirectly in funding policy cooperation (Open Method of Coordination). This systemic effect goes together with partial progress made in the area of dissemination of results of the programme. However, the evidence of the exploitation of project results by policy makers and the effective engagement of the latter when they are not included in the project itself is not always clear. In this sense, the evaluation found that the dissemination of results is one of the aspects of Erasmus+ where there is room for further improvement. The evaluation also noted that the impact of funded projects on national systems could be more systematic if there were more cooperation projects fit for mainstreaming, focussed on fewer priorities at EU level and further efforts made for mainstreaming these at national level. The systemic impact of the actions of a new kind introduced only in 2014 (KA3, alliances, etc) is meant to be evaluated at final stage, after 2020. 3) The evaluation considers there is potential for better definition of actions to maximise the programme's impact in Adult Education, sport, Jean Monnet activities and the Student Loan Guarantee Facility. Considering the funding available at EU level, evidence shows that: the contribution in the adult learning sector is diluted due to the wide size of the target population and the fragmented and diverse nature of the sector; in the field of sport, resources should not be spread too thinly to have meaningful result; the Student Loan Guarantee Facility has not yet lived up to volume expectations partly due to delays in its launch. Regarding Jean Monnet activities, there is a need to strengthen the youngest generation's (notably school pupils') awareness and understanding of European integration. 4) In light of the impacts achieved in all other areas, the evaluation highlights the strong European added value of Erasmus+ and its predecessor programmes, compared to what could have been achieved with similar programmes focused on separate geographical areas. This added value is the result of greater impact due to a much higher volume and wider scope of funded activities, fairer access to learning mobility, deeper EU integration, a clearer international dimension and mainstreamed best practices. However, the programme's potential for stimulating learning innovation seems to be lagging behind and could be further exploited in the future. 5) The evaluation concludes that, in the absence of Erasmus+ and its predecessors, there would be clear negative effects on learning mobility abroad, transnational cooperation among organisations, including with partner countries, integration between European countries as well as the attitude of participants towards the EU. Erasmus+ is more coherent, more relevant and only partly more efficient than its predecessors. 1) The main structural change of the Erasmus+ programme, compared to its predecessors, is its integrated nature which has contributed to enhance the programme's internal coherence. Erasmus+ covers learning in all its contexts – whether formal or non-formal, including youth work and sport – and at all levels of the lifelong learning continuum: from early childhood education and schools, vocational education and training (VET), adult learning to higher education, including its international dimension. The evaluation highlighted the positive effects of this integrated approach underpinned by the lifelong learning logic. The evaluation highlights the following positive consequences of the Erasmus+ design, which has resulted in: ·a sharp increase of cross-sectoral cooperation between education and training sectors, youth and sport; ·an improved geographical balance with small countries and countries from Central and Eastern Europe being better integrated; ·a simplified architecture in three key actions; ·a single brand name which has contributed to the programme's increased visibility and a progressively strong adherence by the sectors covered. 2) In terms of policy relevance, the evaluation shows that stakeholders see Erasmus+ as being more clearly aligned with EU policies and priorities than predecessor programmes. However, a majority of programme countries call for more flexibility at national level and the evaluation has shown that to maximize the impact of the programme priorities could be reduced and better focused. 3) The evaluation found a high complementary between Erasmus+ and other EU policies and programmes relevant to education, training, youth and sport (e.g. Europeran Social Fund, Horizon 2020). Although the level of synergies differs, it is notable that the evaluation detected very few overlaps. 4) When it comes to budget, the evaluation concludes that more is needed for the programme to reach a critical mass in sectors other than higher education. The demand largely exceeds the funding available including in higher education. The evaluation, including the public consultation, suggests that a further reconsideration of the programme's financial envelope is needed. Without prejudice to negotiations on the next Multi Financial Framework, the evaluation shows that the budget could be differently shared between the programme sectors, in particular at the advantage of sectors showing the highest performance, but which have received relatively less funding up until now, such as school education and vocational education and training. 5) The evaluation shows that Erasmus+ mobility actions are clearly cost-effective, especially learners' mobility (with a cost for the EU of 15€ per day/learner). Cost-effectiveness of other actions remains harder to quantify. The management costs (6% of Erasmus+ administrative and operational budget) are deemed reasonable, especially when compared to similar national schemes (14% in average). The overall efficiency stemming from the merge of 7 predecessor programmes is not yet clear. More efficiency gains are expected to materialise during the growing phase foreseen by the budget profile of Erasmus+ until 2020. This will have to be evaluated at final stage. 6) In terms of programme management, the division of responsibilities, as inherited from predecessor programmes, between the Commission, National Authorities, National Agencies and EACEA, is overall clear and fit for purpose. A majority of programme countries wish more flexibility in implementing the budget. A truly performance-based approach has been adopted, though some indicators need to be fine-tuned and less information collected while being better exploited. 7) However, there is clearly a repeated call for further simplification. Following a difficult transitional period/learning process, there is broad agreement that Erasmus+ has brought major improvements (e.g. simplified grants, digitalisation, VET Charter, linguistic support, etc.) but that procedures of application and reporting could be further simplified to reduce the administrative burden on beneficiaries. Applicants for small projects are too often expected to meet the same requirements as applicants for large ones. IT tools are not inter-operable and enough user-friendly. The application process could more clearly focus on those criteria that matter most for effectiveness. A specific challenge is to improve the efficiency of the decentralised international credit mobility action which includes more than 12 different budget envelopes for partner countries. Overall, the evaluation found that all the evaluated predecessor programmes were/are highly effective, whereas Erasmus+ is more coherent, more relevant and only partly more efficient than its predecessors.



1.Introduction The Union Programme for Education, Training, Youth and Sport for the period 2014-2020 'Erasmus+', and its predecessor programmes for the years 2007-2013, support learning mobility of individuals worldwide, transnational cooperation between organisations and promote Member States' reforms in the education, training, youth and sport fields. This is how the EU invests in people with a view to unlocking individuals' potential regardless of age or background, in support of Member States' efforts to develop human resources in Europe and beyond. Erasmus+ contributes to the objectives of the Europe 2020 Strategy and more specifically to the Strategic Framework for European Cooperation in Education and Training (ET 2020) , the European Youth Strategy and the EU policy in the field of sport . Erasmus+ also contributes to the EU's more recent overall political objectives, such as the European Pillar of Social Rights and the EU Global Strategy on Foreign and Security Policy. The Erasmus+ programme (2014-2020) has integrated all previously existing EU programmes in the domains of Education, Training, Youth and Sport, and includes an international dimension that is funded by different external action instruments. In the fields of Education, Training and Youth, the programme pursues its objectives through three types of actions: learning mobility abroad, transnational cooperation projects and policy support. Separately, the Jean Monnet activities promote teaching and research on the European integration , while the Erasmus+ supports transnational cooperation activities in the field of sport, focusing in particular on grassroots level sport. The Regulation establishing the Erasmus+ Programme stipulates that a mid-term evaluation report accompanied, if appropriate, by a legislative proposal to amend the Regulation, shall be submitted by the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions by 31 December 2017. Furthermore, the mid-term evaluation shall include the ex-post evaluation of the Erasmus+ predecessor programmes over the period 2007-2013 i.e. Lifelong Learning, Youth in Action, Erasmus Mundus, ALFA, Tempus, Edulink and sport preparatory actions, taking into account their long-term results and impact. Therefore, this report covers actions for the period 2007-2016 in all programme countries (Member States of the European Union, the Former Yugoslav Republic of Macedonia, Iceland, Liechtenstein, Norway, Turkey) and partner countries (neighbouring the European Union and other partner countries). The baseline for this evaluation is the period 2007-2013 unless otherwise specified. In total, the period under evaluation corresponds to a total budget of over EUR 15 billion. The objectives of the mid-term evaluation are to assess five evaluation criteria a) the effectiveness of the measures taken to achieve the Erasmus+ programme's objectives, including the contribution made to the realisation of the Europe 2020 strategy; b) the continued relevance of all of its objectives; c) the Programme's internal and external coherence; d) the efficiency of the Programme and the scope for simplification; and lastly e) its European added value. The results of the evaluation will be used to improve the implementation of the current programme until 2020 where possible. They will also be used to feed into the impact assessment for its successor programme. The mid-term evaluation assesses the performance of the different (Key) actions included in Erasmus+ and evaluates to which extent findings differ across fields and sectors included in the programme and across the different target levels (individuals, organisations and systems). It also assesses the extent to which the programmes have contributed to policy development and implementation in the participating countries. The evaluation does not cover actions that were discontinued during the previous programming period (2007-2013) as these were not to be evaluated, under the Terms of Reference. However, feedback from stakeholders on the interruption of actions discontinued with Erasmus+ was collected and analysed.

This Staff Working Document on the mid-term evaluation of Erasmus+ and predecessor programmes has been carried out according to a Roadmap published in January 2016 . It draws mainly, among other sources , on the National Reports submitted by the programme countries in accordance with Article 21(4) of Regulation (EU) No 1288/2013, previous evaluations and other studies and the final deliverables prepared by ICF Consulting Services Ltd, (hereinafter ICF), under contract to European Commission (DG Education, Youth, Sport and Culture) . The final report delivered by ICF provides answers to all evaluation questions defined in the Terms of Reference and related to the aforementioned five evaluation criteria. The final report of the contractor contains also recommendations addressed to the Commission.

2.Background to Erasmus+ and its predecessor Programmes Erasmus+ is the EU Programme in the field of education, training, youth and sport, with a budget of EUR 16.45 billion for the period 2014-2020 . It provides opportunities for people of all ages (university students but also school pupils, trainees, apprentices, etc.) to study, be trained, volunteer and learn in other countries. It also fosters the professional development of practitioners and supports cooperation on tangible results, networking and share of knowledge among organisation and institutions in the fields covered by the programme. Through cooperation in formal, informal and non-formal learning, the Programme aims to address the following challenges: economic recovery and high youth unemployment; skills' mismatches, low employability and education poverty; global competition for talents; Information and Communication Technology potential and digital divide; social exclusion and intolerance; lack of trust in the EU and low participation in democratic life; threats to the integrity of sport and, more generally, to common European values. The general objectives of Erasmus+ are to contribute to the Europe 2020 strategy for growth and jobs , including the headline education targets , as well as the strategic framework for European cooperation in education and training ('ET 2020'), including related benchmarks. Erasmus+ also aims to contribute to achieving the objectives of the EU Youth Strategy and the EU Work Plan for Sport, to promote the sustainable development of partner countries in the field of higher education and youth, as well as to foster European values . Specific objectives tackled by the programme include the improvement of the level of key competences and skills, with particular regard to their relevance for the labour market and their contribution to a cohesive society; the promotion of solidarity and participation in democratic life in Europe and the labour market; the improvement of quality, innovation, excellence (including in European studies) and internationalisation at the level of organisations and practitioners; support to the modernisation of education and training systems, in particular through evidence-based policy cooperation; the enhancement of the European/international dimension of its sectors, including with partner countries in complementarity with the Union's external action; the promotion of the Union's linguistic diversity and intercultural awareness; cross-border threats to the integrity of sport; support to good governance in sport and dual careers of athletes; and the promotion of voluntary activities in sport. Erasmus+ results from the integration of the following predecessor European interventions implemented during the period 2007-2013: Lifelong Learning (LLP) , Youth in Action (YiA), Preparatory Actions in sport (as of 2009), as well as in the sector of international higher education: Erasmus Mundus II (as of 2009, with the rest of the world), Edulink (as of 2008, with African, Caribbean and Pacific Group of States), Tempus IV (with neighbouring countries), Alfa III (with Latin America) and programmes of cooperation with certain industrialised countries. The current Erasmus+ programme has the following architecture: Key Action 1: Learning mobility of individuals (opportunities for students, trainees, apprentices, young people and volunteers, as well as for professors, teachers, trainers, youth workers, staff of educational institutions and civil society organisations to undertake a learning and/or professional experience in another country) Key Action 2: Cooperation for innovation and the exchange of good practices (transnational or international projects promoting cooperation, innovation, exchange of experience and know-how between different types of organisations and institutions involved in education, training and youth or in other relevant fields) Key Action 3: Support for policy reform (actions supporting national authorities and stakeholders in defining and implementing new and better coordinated policies in the field of education, training and youth). Jean Monnet activities (actions aimed at improving the quality of teaching on European integration studies, as well as projects and operating grants aimed at promoting discussion, reflection on EU issues and enhancing knowledge about the EU and its functioning) Sport (cooperation projects, events, studies and other initiatives aimed at implementing EU strategies and priorities in the field of sport) Erasmus+ relies on management modes inherited from predecessor programmes. The European Commission bears the overall responsibility for the supervision and coordination of the agencies in charge of implementing the Programme at national level. The European Commission manages the budget and sets priorities, targets and criteria for the Programme. Furthermore, it guides and monitors the overall implementation, and evaluates the Programme at European level after having received the National Reports from participating countries, as described under "method". It also manages directly few actions of the programme . At European level, the European Commission's Education, Audiovisual and Culture Executive Agency (EACEA) is responsible for the implementation of centralised actions of the programmes (direct management) which account for a small share of the total budget. The largest share of the budget is implemented through indirect management. The European Commission entrusts implementation and promotion tasks to National Agencies (NAs) established in each of the 33 countries participating to the Erasmus+ programme (named hereafter "programme countries" ) which implement those actions of the programme with the highest volume so as to bring the programme as close as possible to its beneficiaries and to adapt to the diversity of national education, training and youth systems. No direct support is given to individual beneficiaries ; all support is channelled through participating organisations, which distribute it to individual learners or practitioners. 2.1.Main changes in the programmes over the period under evaluation

In line with the subsidiarity principle and Articles 165 and 166 of the Treaty on the Functioning of the European Union, the EU intervention aims at supporting and supplementing the actions of Member States, while fully respecting their responsibility for the content of education, training, youth and sport policies. Meanwhile, more than its predecessors, Erasmus+ has been designed with a view to better aligning spending actions with the priorities set at EU level in each policy area. The integrated structure of the programme is the most noticeable change in scope brought by Erasmus+. Other important developments between the two programming periods have taken the form of an increased budget allocation (+40%); a simplified structure in 3 Key Actions; streamlined ways of implementation and systematic monitoring, as evaluated under "effectiveness", "coherence" and "efficiency". a) Erasmus+ does not only draw on the legacy of its predecessors but also brings them altogether into a single integrated programme, including its international dimension funded by the EU external action instruments (2% of all Erasmus+ student mobility activities have involved partner countries) . The suppression of (sub-) programmes per field (sector) contributes to the "effectiveness", "efficiency" and "internal coherence" of the programme, as evaluated below. b) Erasmus+ benefits from increased resources with over EUR 16.4 billion spread over seven years as opposed to a total budget of slightly more than a total of EUR 9 billion allocated to its predecessors over 2007-2013. Examples of changes with Erasmus+ • Opportunities for over 4 million people to study, train, volunteer or gain professional experience abroad (1.8 million learners and practitioners over 2014-2016) • A stronger focus on improving young people’s job prospects to tackle youth unemployment (more and shorter traineeships since 2014) • A more inclusive programme supporting people with fewer opportunities (11.5% of learner participants) • Opportunities for participants to study worldwide (12% of all higher education participating organisations) • Language learning support for mobility participants (14% of learners in long term mobility) • New funding for actions in the field of sport (1,8 % of the budget – heading 1) c) For simplification purposes, following the conclusions of the Impact Assessment for Erasmus+ , the current programme is based on three cross-cutting Key Actions which apply to all education and training sectors and to the youth field: Key Action 1 (KA1) supports learning mobility of individuals; Key Action 2 (KA2) fosters cooperation for innovation and the exchange of good practices, including collaborative platforms, and Key Action 3 (KA3) supports policy reforms. This integrated approach does not apply to the two stand-alone strands for sport and for European integration studies ("Jean Monnet"). The fact that the nature of the funded activities has not fundamentally changed with Erasmus+, as reflected below, makes it possible, however, to draw conclusions about the likely impact of the current programme based on what can be observed ex post from the impact of its predecessors. Main type of actions per Erasmus+ Key Actions Learning mobility of individuals (KA 1) Cooperation for innovation and exchange of good practices (KA2) Support for policy reform (KA3) -Mobility of learners and practitioners -Erasmus Mundus Joint Master Degrees -Student Loan Guarantee Facility -Strategic Partnerships -IT support platforms -Knowledge Alliances (HE) -Sector Skills Alliances (VET) -Capacity Building (HE and Youth) -Open method of Coordination (OMC) -Prospective initiatives -EU transparency and recognition tools -Dissemination & exploitation -Policy dialogue with stakeholders, third countries and international organisations In bold: most significant novelties of Erasmus+ d) The integrated nature of the programme means that cross-field approaches or tools can be used for the purpose of grant application, monitoring, audit or dissemination. This has materialised through on-line application, single audit system, generalisation of simplified grants (lump sums, unit costs), performance-based monitoring system, a single Project results platform, etc. In designing Erasmus+, specific attention has been paid to make it more result-oriented than its predecessors and more agile to adapt to policy evolutions notably in the light of the socio-economic challenges mentioned above. 2.2.Intervention logic The EU mandate in the education, training, youth and sport areas sets the scope for the intervention logic . An overarching intervention logic was designed to cater for all the programmes in scope of the evaluation, irrespective of the programming period. This approach was followed in order to assess and compare more swiftly results across programmes in order to make a more comprehensive assessment of the impacts achieved during the baseline period. As a matter of fact, although each evaluated programme had different architecture, most of the supported actions (mobility, cooperation and policy reform support) were transversally implemented – although with a different level of intensity - under all programmes. For that purpose, the programme's specific objectives and actions have been grouped according to the level at which the results are expected . Furthermore, considering the overwhelming dominance of the external factors of Member States' policy making and spending in these areas, it is not easy to attribute and quantify the specific effects of the EU intervention. Spill-over between intervention levels is expected, as shown in the graphic below, explaining why most of the specific objectives are expected to deliver results at more than one level. For instance, the mobility of learners and practitioners (KA1) can – in addition to individual-level results – improve the performance of the organisations and have an impact on systems, especially in terms of outcome recognition. Also, the performance of individual organisations (KA2) is expected to benefit from European cooperation in the fields of the programme, including through its modernising effects on national systems and reforms prompted by the open method of coordination and KA3. ·Systems, to improve policies

in programme and partner countries;

·Organisations, to promote cooperation

and positive changes in work methods; ·Individuals, to increase

participants’ competences

as well as change their attitudes,

practices and perceptions; Source: Roadmap for the evaluation, January 2016

3.Implementation of Erasmus+ and predecessor programmes until 2016 In the period 2007–2016, the programmes have funded mobility for more than 4.3 million learners and more than 880,000 practitioners. Many more have benefited from short-term blended mobility and/or other forms of transnational exchanges as part of cooperation projects. In the current programming period (2014–2016) alone, at the time of the mid-term evaluation (i.e. without taking into account fully 2016 data), the programme had already benefited over 1.4 million learners and 400,000 practitioners. Approximately 67 000 projects have been contracted in the same period, which represent a reduction in yearly number by around 50% compared to 2007-2013. Erasmus+ has indeed been designed to fund fewer, but more multifaceted projects with larger average numbers of participants. Comprehensive information is available from the Erasmus+ programme – Annual Report (taking into account fully 2016 data) . The Erasmus+ programme has been fully implemented as from 2015 without major difficulties . The international higher education strand and the Student Loan Guarantee Facility were launched only in the second year of the programme, when arrangements were ultimately made respectively with the External Relations instruments and the European Investment Fund. The implications of the newly created European Solidarity Corps on Erasmus+, more specifically on the implementation of the European Voluntary Service under the Youth Chapter, only materialised in 2017; it is, for that reason, out of the scope of this evaluation. Erasmus+ was allocated EUR 16.4 billion for the 2014-2020 period . Despite the overall significant budget increase compared to the previous programming period (+40%), Erasmus+ has only experienced a limited budget increase in 2014-2016 according to the programme's budgetary profile which concentrates the funding increase over 2017-2020 , as shown below. It is therefore too early at mid-term to evaluate the full impact of the additional resources allocated to the current programme. Source: Erasmus+ activity report 2015 Distribution of Erasmus+ budget across sectors (2014-2016) Source: ICF calculations based on EplusLink The most substantial part of the Erasmus+ budget is allocated to education and training and, within this field, to higher education, as shown in the graph above . In terms of actions, mobility has the largest share of the budget, as displayed below. Key data, 2014 - 2016 KA1 KA2 KA3 Jean Monnet Sport Projects received 108,904 34,298 4,456 2,425 1,325 Contracted projects 49,073 6,936 2,127 762 238 Amounts allocated (in million EUR) * 3,798.9 1,469.4 205.9 126.2 67.3 Participants 2,049,140 1,231,267 184,403 944,245 -

Source: Erasmus+ Programme Annual Reports 2014-2016 (taking into account fully 2016 data) *Amounts not only for project grants but including also operating grants

4.Method of the evaluation Considering the high quality of data collected through several complementary techniques from various sources, the sophisticated analysis and triangulation of evidence, the positive opinion of the Regulatory Scrutiny Board and the improvement stemming from its comments, as well as the significant feedback received (with more than a million responses processed in total), this evaluation can be considered very reliable and valid. 4.1.Evaluation framework To undertake the combined mid-term evaluation of Erasmus+ and the final evaluation of its predecessor programmes, as outlined above in section 1, the Commission has drawn upon a set of evaluation questions relating to five main criteria: effectiveness, relevance, coherence, efficiency and EU added value, as per the evaluation Roadmap . To ensure comparable approaches, the evaluation framework developed by the external evaluator classifies actions across the three levels of intervention of the programmes and uses that as the basis of comparison over 10 years, across all sectors and various target groups, namely: ·at System level ·at Organisation level ·at Individual level, divided into actions for learners and practitioners across all different areas (education and training, youth and sport) and – within education and training – different sectors (school education (SE), vocational education and training (VET), higher education (HE) and adult education (AE)). For the purpose of the evaluation, the overarching terms of Learners and Practitioners are used . In addition, the analysis of effectiveness has been based on the intervention logic represented graphically in annex 5a accompanied by a detailed elaboration of expected outcomes (results directly for individual beneficiaries; impact indirectly for a wider population, participating organisations or education, training, youth and sport systems) . All specific objectives of the programme (annex 5c) are reflected in the outcomes or impacts of the intervention logic. The purpose of the evaluation is to evaluate the outcome of the programme on target groups, not to audit each type of activities. Therefore this intervention logic is a powerful simplified model to develop questionnaires based on the most relevant levels of outcomes. It is worth noting that these three levels of intervention are not equivalent to the three Key Actions, as explained above . A given Key Action can indeed have an effect on more than one level of intervention (see 2.2). Key Actions have been set within Erasmus+ but these three levels of interventions apply to both programming periods . This unprecedented combination of the ex post evaluation of predecessor programmes and the mid-term evaluation of the current programme is a legal requirement. It implies to evaluate differently the two periods, as laid down in article 21(2) of the Erasmus+ Regulation: [The mid-term evaluation report] "shall also take into account the results of an evaluation of the long-term impact of the predecessor programmes (Lifelong Learning, Youth in Action, Erasmus Mundus and other international higher education programmes)." Whereas Erasmus+ is evaluated against all aforementioned criteria, its predecessors are mainly evaluated in relation to effectiveness and relevance . However, since the predecessors are systematically taken as a baseline for comparison with Erasmus+, they have been evaluated also for comparison purposes in terms of coherence, efficiency and European added value. Systemic impact as well as long term effects of projects should mainly be attributed to previous programmes, whilst short-term effects on individuals have rather been evaluated in the context of the current programme. The results at the level of individuals are unlikely to be different between the two programming periods as the types of actions are highly similar. What has mainly changed in that respect is the scale and reach of the programme as of 2014. The programme is designed so that the systemic impact of projects (Key Actions 2 and 3) should only be evaluated "by the end of the programme" as per article 21(1)b of the legal basis. At mid-term, systemic and long term effects can also be attributed by extrapolation to Erasmus+ due to the continued nature of basic activities (mobility; cooperation). 4.2.Data collection and analysis Evaluation findings have been based on a wide range of data sources systematically triangulated i.e. cross-checked against each other: programme documentation, programme data and views of the managing bodies, views of direct beneficiaries as well as non-beneficiaries, EU and national stakeholders, policy makers, programme countries, bodies in charge of implementation of other comparable programmes and the general public . The evaluation combined a number of techniques for data collection and analysis. Most results are reached by a mix of evidence combining a quasi-counterfactual assessment based on beneficiaries’ surveys (i.e. comparisons between beneficiaries and a control group), self-reported feedback of beneficiaries (i.e. monitoring surveys), as well as qualitative interviews and case studies. While monitoring surveys could only give findings as from 2014, evaluation surveys covered 2007-2016 and enabled the evaluators to compare the two programme generations. Monitoring data have been fully used and supplemented with surveys carried out by the external evaluator, providing more objective findings (with questions not based on self-perception), more comprehensive data (covering as well the 2007-2013 period and KA2 participants) and more comparable responses (as questions were harmonised across sectors and periods for comparison purpose). Collection tools presented below covered all programme countries, sectors and main types of actions (except where mentioned otherwise). The quality of the one million responses processed was in most cases exceptionally good. All quantitative data was statistically processed and presented using visualisation tools, while all qualitative data was systematically coded. As required by the Erasmus+ Regulation, programme countries have submitted their own evaluation reports on the implementation and impact of decentralised actions of Erasmus+ in their respective territories. Their findings are summarised in the National reports synthesis (NRS) annexed to the external evaluator's report . In line with Better Regulation rules, an Open Public Consultation (OPC) gathered the opinions of the general public and all interested groups on all evaluation criteria. Different questions were asked depending on the level and scope of knowledge of the programmes declared by each respondent. Consultation findings are presented separately in the synopsis drawn by the external evaluator . More widely, responses from all sources are presented in Annex 2.

Box 1 provides an overview of the data sources from which the evidence was drawn. A detailed description of the individual methods is provided in Annex 3. ·Literature review (131 sources about 2007-2016) ·Key informant interviews (190 interviews about 2007-2016) ·National reports (34 countries mainly about 2014-2016) ·Open Public Consultation (covering 2007-2016 with 4,786 responses of which 1,800 responses were fully exploitable; 24 position papers submitted) ·Social Media analysis (725,678 posts over 12 months) ·Benchmarking (18 comparable national/transnational programmes) ·Programme database analysis (all beneficiaries 2007-2016) including network analysis ·Monitoring surveys (over 955,000 respondents - KA1 since 2014) ·Online Linguistic Support (523,238 participants since 2014) ·Beneficiary surveys and control groups (over 47,000 respondents over 2007-2016) ·Survey of socio-economic organisations (947 responses over 2007-2016) ·Agencies survey (130 responses from National Agencies or the Executive Agency EACEA) ·Experts survey (1,122 responses from project assessors over 2007-2016) ·Case studies (38 organisations or policy cases; 233 respondents over 2007-2016) ·Review of selected projects’ outputs (386 project reports over 2007-2016) ·Expert panel assessment of projects’ outputs (100 projects) ·Jean Monnet activities (2,350 survey respondents; 5 interviews) ·Student Loan Guarantee Facility (SLGF) (219 survey respondents; 33 interviews)

4.3.Method strengths and limitations

Overall, the quality of available data for the predecessor programmes was good, while for Erasmus+ it was very good. This reflects improvements made in terms of monitoring since 2014. The evaluation used the wealth of the programme data available from the Erasmus+ management or monitoring tools, including some data on results (e.g. satisfaction-rate) since 2014. Although the period within the scope of the evaluation was 2007-2016, programme data was not fully available for 2007 . The main issues that affected the analysis of participation patterns include the following: i) the absence of a harmonised definition, across sectors, of people with fewer opportunities and from disadvantaged backgrounds (programme data were completed with other findings) ; ii) the lack of a complete picture on the type of participating organisations (some 40% of organisations were classified as "other" in the programme dataset limiting the analysis of participation per type of organisation). It should also be noted that the analysis over time of networking among participating organisations or repeated participation of individuals in the programme could not be fully pursued, as prior to 2014 the same organisations could be recorded under multiple names, which made such an analysis difficult to execute. These limitations have however remained marginal or have been overcome through the use of other sources. The monitoring surveys of beneficiaries carried out by the Commission concern all learners or practitioners taking part in mobility under KA1 (since 2014 only) . All are requested to fill in an online survey on completion of their mobility. Therefore the reliability of the data is strong. Given the sample sizes (first two years of the programme) the data is considered to be strongly reliable even though it was too early to include 2016 data. This being said the link between the monitoring questions and the indicators they are supposed to measure can be weak in some cases. Above all, most questions are based on the self-perception of beneficiaries in contrast to more objective measurements such as the assessment of language proficiency after mobility (OLS ). This is why monitoring data were supplemented with primary data collection, such as surveys carried out by the external evaluator, providing more objective and comparable responses. Series of surveys of beneficiary learners and staff and related control groups covered all programme target groups and predecessor programmes as well (2008-2016 in total), both for mobility and cooperation, over 2007-2016. Overall, findings could be generalised to the whole programme with a sufficient degree of confidence because of the sample sizes of all surveys and the distribution of various background variables within the survey samples. Given the limited data available on the background of beneficiaries, it nevertheless remains impossible to quantify the degree of comparability of the samples to the programme population . A partially counterfactual approach ("quasi-experimental") was also used to quantify the contribution of the programme to the results measured. This was done by assessing the difference in results between Erasmus+ beneficiaries and their "peers" who did not take part. The challenge is to attribute the difference in outcomes between both groups to the programme isolating any other underlying factor (attribution). Control groups of non-beneficiaries were therefore set up. In an ideal world, these should be selected on random basis. However, a random allocation is neither feasible in practical terms nor desirable for ethical reasons. By default, a ‘matched sample’ was sought where individuals are similar in some background characteristics such as gender, age, etc. There were no significant differences between the control group and the treatment group for most variables, and where there were, they concerned variables that were not likely to strongly influence the findings. Most control groups reached a significant size allowing comparison with beneficiaries, except in sport and Adult Education sectors where the control groups were too small. As a consequence, the control group of VET staff was also used for Adult Education and the control group of Youth staff was also used for sport, as these were found to be sufficiently comparable against a range of background characteristics. However this counterfactual approach cannot exclude the possibility that the difference in a given variable between beneficiaries and non-beneficiaries could be partially due to the selection into the programme rather than the effect of the programme. In addition to the above ‘post’ surveys (i.e. beneficiaries were surveyed after they took part in the programme), the external evaluator carried out pre-surveys in two sectors where short term mobility exchanges apply (i.e. school pupils and Youth exchanges). This was done as far as possible within the time frame of the evaluation assignment for the purpose of pre/post comparison. In spite of repeated recruitment efforts, the sample sizes suffered from a high attrition from the pre to the post survey, especially for the Youth sector where comparison of profiles prior to Erasmus+ mobility could not be done. Pre-post results for the Youth sector have therefore not been interpreted considering the high likelihood of sample bias. This allowed nevertheless for the school sector to draw conclusions on selection into the programme and to observe changes during the time of the mobility experience. To get a better understanding of their quality and dissemination potential, a review of selected project outputs, mainly collected from National Agencies, was carried out. Given that only relatively few Erasmus+ funded projects were completed at the time of this evaluation, the sample remained inevitably somewhat biased towards predecessor programmes. As for the Open Public Consultation (OPC), a large number of partial responses only contained very basic background information on respondents and had therefore to be marked as unsuitable for analysis. In any case, in contrast to surveys, public consultation findings can never be representative due to the selection bias inherent to any open recruitment. Lastly, some programme countries have delivered their national reports with a four month delay. This, in turn, has delayed the adoption of the Commission report to take into account all programme countries' views. The period of time between the deadlines set by the legislator respectively for the delivery of the national reports and the Commission report was reduced from 9 months (LLP) to 6 months (Erasmus+), which has not allowed the external evaluator to make a full synthesis of all findings .

4.4.Judgement on the validity and reliability of the findings The external evaluator (ICF) contracted for this assignment has carried out since May 2016 all tasks as required under the scrutiny of an inter-service group (ISG) and the daily steer of DG Education, Youth, Sport and Culture (EAC). The only significant change compared to the initial work plan was a two-month delay of the (sub)contractor in launching the beneficiary surveys . The ISG was consulted at each stage of the evaluation process and reviewed each deliverable produced by the contractor as well as this Staff Working Document . The Regulatory Scrutiny Board, responsible for the independent quality control of this evaluation, acknowledged in its positive opinion with comments the significant efforts of data and evidence collection and noted the good methodology used . This document has been improved following its comments. Based on the above elements of the evaluation method and as further described in Annex 3, the reliability and validity of the evaluation can be regarded as strong . Elements of the method Assessment of reliability and validity Validity of overall judgements Strong Definition of results and impacts Strong Validity of measurement tools to collect data on results and impacts Strong Reliability of overall evaluation design Strong Collection of qualitative data Strong Interpretation of qualitative data Strong Generalisation of findings to the whole programme Medium to strong Counterfactual assessment Medium Surveys Medium Programme data Medium Source: ICF, 3.5; see section above for assessment grounds

5.Findings

5.1.Effectiveness The assessment of the effectiveness of Erasmus+ is overall very positive, especially when compared to the situation before the integration of predecessor programmes . It has been informed by 15 evaluation questions that looked into the extent to which: ·outcomes (i.e. outputs, results and impacts) were delivered with particular attention to disadvantaged groups ·spill-over, sustainable or unintended effects took place ·certain fields or actions were more effective than others (including discontinued actions) ·results were disseminated and exploited. The programme reaches or exceeds the vast majority of its indicator targets set in the legal basis , except in the case of the Student Loan Guarantee Facility (SLGF) which is missing the target by a high margin, as detailed below . However, some output targets might have been set at a low level initially (sometimes below predecessor programme levels). The quality of the data on participation of disadvantaged groups although on target is unclear due to various definitions and practices (see below). Lastly, for Jean Monnet activities there is no exact data on numbers of students reached and this indicator is likely to be very overestimated. These limitations concern only a few indicators and the vast majority are without doubt on target. Incoming mobility of students from higher education international strand is also below target but the action started in 2015 only and the overall trend over ten years is positive. Evaluated programmes have been important for the EU's global outreach, as elaborated hereafter . The Open Public Consultation confirms that the programme is achieving its objectives to a (very) large extent. OPC respondents highlighted two elements as particularly helpful in terms of effectiveness: the new integrated structure of the programme and the overall increase in funding . One sustainable result of the programme and its predecessors is the cultural shift in the perception of mobility and its positive image, at least in the higher education sector and progressively in VET. These long-term changes are yet to fully materialise in the other sectors. 5.1.1.Outcomes for learners and practitioners Compared to the last three years of predecessor programmes, Erasmus+ has supported during the first two years a much higher volume of mobility within and outside the EU in higher education (the sector that has most beneficiaries overall), but also a higher number of practitioners in VET and schools . The repeated participation of individuals is kept reasonably low, indicating a continuous impact on renewed cohorts. In contrast, as intended with the design of Erasmus+ there has been a significant decrease in mobility for pupils - due to the fact that individual mobility of pupils was not embedded in the programme as such any more - and a slight decrease for VET learners . 5.1.1.1. Outcomes for learners Beyond contrasted evolutions in outputs across sectors, drawing on a variety of sources , the evaluation found positive and sustainable results for all individual learners taking part in mobility . Firstly, monitoring surveys outline that most learners themselves are satisfied with their mobility experience . The rate of formal recognition of participation in Erasmus+ is high but, more importantly for those concerned; the rate of recognition of learning outcomes is also rather high . The vast majority agree that the programme helped them improve more than 6 competences out of a list of 9 (see fig.) . To be noted, higher education students are comparatively least positive in their self-assessment . Share of learners who (strongly) agree that they developed transversal competence Source: ICF calculations based on Erasmus+ participant monitoring surveys. Multiple answer question. N = 730,254 In line with the intervention logic , all programmes under review are expected to contribute to changes in learners’ skills and competences as well as attitudes. In the longer term, they are expected to result in better completion rates and more positive employment outcomes. Both have materialised as detailed below. As far as the general indicators of Erasmus+ are concerned, trends are also positive and the Education and Training Monitor 2017 highlights a further decrease in the rate of early school leavers, and a tertiary attainment target that is within reach . Most result indicators set in the legal basis of Erasmus+ are on or above target . The evaluation found, using a variety of sources, clear evidence of results for mobile learners in: foreign language skills development, especially for those with low entry levels; feeling of belonging to the EU; willingness to be mobile and to work abroad; shorter transition from education to work ; positive perception of the value of education and positive feeling towards the sending organisation which are key precursors for retention and attainment in education and training. As evidenced for those benefiting from KA1 mobility, the programme is associated with stronger rate of completion of studies . However the result indicators set in the programme legal basis and related targets set in DG EAC's strategic plan have some limitations in that they are mostly based on the self-perceived contribution of KA1 reported by beneficiaries. Some of the targets were exceeded and others not reached (e.g. participants declaring that they have increased their language skills; young participants declaring being better prepared to participate in social and political life). However, there are concerns about the plausibility of the target values in both cases . The evaluation does not permit isolating how many of these results are the direct effect of the selection of the most motivated into the programme. For instance, Jean Monnet students do show better results than other higher education students when it comes to knowledge of the EU. This however can be expected given that the programme reaches mainly those who have shown an interest in the EU topics .

The most successful objectives of the Programme, which are also the most relevant, according to OPC respondents, have been a) developing the skills and competences of learners; and b) promoting the European dimension of education, training and youth activities . Other studies also confirm this in the relevant sectors . Higher education who have participated in the programme are not only more likely to be employed, but also more likely to secure management positions. On average, 64% of Erasmus students, compared to 55% of their non-mobile peers hold such positions within 5-10 years from graduation. This holds even more true for Erasmus students from Central and Eastern Europe, where around 70% of them end up in managerial jobs . The same can be said for the international dimension of higher education, with clear evidence of strengthened capacities and employability of individual students and scholars who have participated in mobility . Albeit mostly based on the self-reported perceptions of participants, most National Reports refer also to improvement of the level of key competences and skills of learners with a high level of similarity with the ones listed above. On top of technical/professional knowledge and skills , the impact of Erasmus+ on employability is underlined by several countries mostly in the case of HE and VET , transversal skills or soft skills are also frequently identified. These include communication skills , social and civic skills , and more widely personal development . The case studies confirmed the strongly positive influence the programme plays on personal development and maturity of young people. Both sources frequently outline that learners gain in confidence, independence, ability to cope with new environments and open-mindedness through their mobility. Participation in the programme also leads to development of learners’ social capital . In contrast to very positive effects of all forms of mobility, there is less clear evidence of impact of transnational partnerships on learners (excluding their mobility component) except in the school sector . For example, a survey conducted as part of the eTwinning evaluation, indicated that 55% of teachers believe that the platform is helpful in motivating their pupils . At programme level, the areas where learner surveys have identified strongest positive difference between the results of participants and non-participants include: willingness to move abroad permanently (+31% likeliness compared to non-participants); stronger feeling of being an EU citizen (+19%); shorter transition to employment (+13%) ; value attached to education (+8%) and also openness to immigration and minorities (+2.6%). The figure below gives an overview of all areas evaluated according to the degree of difference found .



Comparison of results for learner beneficiaries and control group Source: ICF Beneficiary surveys: only areas where the difference is statistically significant i.e. unlikely due to a sampling interference The share of beneficiaries who took less than three months to find a job stood at 68.5%, while the percentage of the overall control group was at 59.2%. This trend mirrors the positive expectation of most beneficiaries towards Erasmus+ . This is particularly strong for the higher education and VET beneficiaries, in contrast with the youth sector . Erasmus+ also leaves a positive effect on entrepreneurship . Overall, this confirms the findings of several previous studies about the effects of learners’ mobility on employability . The pre/post analysis for pupils confirms the positive contribution of Erasmus+ in five similar result areas, the order of which appears nevertheless specific to the school sector. There are statistically significant differences in results across pre- and post-survey of school pupils in EU citizenship (+7 score points), digital competence (+4), racism and xenophobia (+3), self-confidence in education (+4), civic participation and volunteering (negative difference). However, there is likely selection of already more performant pupils into the programme at the level of schools, since the pre-post survey also shows already a significant difference between participants and the control group at entry into the programme. Pupils participating in short-term mobility speak a foreign language more often than the control group (+13 percentage points or p.p.) and feel more often as EU citizens already before going on mobility (+9 p.p.). Cultural awareness (+7 points in index score or p.i.s.), positive attitudes towards immigration and minorities (+6 p.i.s.), digital competence (+5 p.i.s.) and perceptions of the value of their education (+4 p.i.s.) were also higher. This is confirmed by the case studies where some respondents claim that participation is seen as a certain form of reward. Foreign language skills development is more important in those sectors where learners tend to have lower entry levels (i.e. in VET and youth more than in higher education). The analysis of OLS foreign language tests (pre/post mobility) proved that the weaker the proficiency entry level, the stronger the linguistic improvement due to mobility . Comparison of levels of proficiency before and after mobility, by sector Legend: share of learners in each sector at a given level of proficiency; blue: before mobility (first language assessment), red: at the end of the mobility (final language assessment) Source: ICF calculations based on OLS data as of 31/08/2016. Higher education: n= 500,410; VET: n=19,441; EVS: n=3,387 Beneficiaries were asked in case studies to describe the most important contribution of the programme. The word-cloud below visualises the most cited testimonies.

What do you see as the most important contribution of the programme for yourself? Source: ICF case studies –interviews with learners: 200 mentions or expressions were collected; the size of the font captures the frequency with which a given result was mentioned. When asked about reasons hindering the achievement of programme objectives , respondents identified the following barriers during the public consultation, in descending order of frequency: linguistic barriers; financial difficulties and complex application process . The factors that enable positive results at the level of individual learners are related to their motivation to participate as well as the quality of the learning experience including foreign language learning support, the support received in the host organisation and their integration among other students and local communities . Lastly, levels of recognition of learning outcomes vary according to types of actions , but signing a learning agreement remains a strong predictive indicator for later recognition. Only a small share of learners does not sign one . Although financial barriers are a major obstacle to mobility , the Student Loan Guarantee Facility, which was an innovation in Erasmus+, has so far failed to attract financial intermediaries in sufficient numbers – especially for the incoming student segment . First beneficiaries are, however, satisfied. The Student Loan Guarantee Facility already shows signs of social fairness , but its visibility at this early stage is far from being sufficient throughout the supply chain . 5.1.1.2. Outcomes for disadvantaged learners The current programme is rightly perceived as paying more attention to the participation of disadvantaged people . To respond to the criticisms regarding the predecessor programmes – notably in the field of higher education – of being elite programmes, Erasmus+ has put in place actions to strengthen the participation of disadvantaged groups compared to predecessor programmes . In the period under review, 11.5% of participants under KA1 had special needs or came from a disadvantaged background. This share rose to 30% for learners in the Youth field between 2014 and 2016 . The number of participants with special needs or fewer opportunities has more than doubled since the predecessor programmes, i.e. more than proportionally compared to the overall budget increase. The indicator targets have even been exceeded , though data reliability is weak . This is why some countries call for more consistent definitions and rules for disadvantaged participants . According to case studies, the programme reaches out to those who are easier to reach among the disadvantaged, and not to those who are disengaged or at risk of marginalisation. This is not specific to Erasmus+ as other benchmarked programmes struggle with the same challenge. This is partly due to the fact that the participating organisations, even under an action as inclusive as the European Voluntary Service (EVS) , tend to select the most motivated and performing individuals. In spite of the specific measures mentioned above to foster social inclusiveness, the evaluation found that more specific support and adequate funding would be needed . The lack of spending flexibility limits the capacity to face unforeseen situations inherent to vulnerable groups (see Efficiency) 72% of respondents to the consultation indicated that there has not been sufficient progress in this area under the current Programme and that this aspect deserves more attention and funding . This was also flagged by the majority of countries in their National Reports . To mitigate the limitation of the related programme indicator mentioned above, beneficiaries were personally asked about their social background . Evaluation survey responses indicate that the self-reported participation of people from a minority background or with learning difficulties is rather small. Yet, responses highlight that disadvantaged learners show more positive results than the others, in particular in completing formal learning and in boosting their self-confidence in their education capacities. However, no significant difference was noted for them in terms of benefiting from shorter transition periods to employment following their mobility experience. These survey results are statistically significant , but the pre/post surveys of school pupils and the case studies indicate that they are likely due to selection into the programme, at least for a part and cannot be confirmed with strong evidence from data collected or the literature reviewed . Most interviewees in case studies believe in the potential of the programme to provide powerful results for disadvantaged groups. Several of them were clear that more than other groups, disadvantaged learners would not have gone abroad to study had it not been for the programme. Meanwhile, the participation of organisations with high shares of disadvantaged learners remains low. The case studies suggest that specific top-up points for organisations with high share of disadvantaged people should not have been discontinued in the current programme and that accompanying disadvantaged learners (e.g. in European Voluntary Service) requires a particular care and specific measures. On the external dimension, the picture is somewhat similar. All the predecessor programmes promoted inclusiveness in higher education from a gender perspective. Some promoted other under-represented, disadvantaged groups. However their specific evaluation noted that the mobility programmes only partly succeeded in achieving equitable participation . Beyond mobility, the other types of action have also inspired new ways of working with people from a disadvantaged background, especially in sectors other than higher education . This was reported by 44% of practitioner respondents , although only 19% stated that the activity they took part focused on integration of disadvantaged groups. This implies that there are significant spill over effects even where it is not the main purpose to target these groups (i.e. impacts beyond results). Lastly, certain actions perform better in terms of social inclusiveness. Studies in the Youth sector found that international youth projects or mobility have had significantly more impact on young people with fewer opportunities, especially in learning to learn, developing cultural awareness and building self-confidence (+3.6%), likely due to the lower starting points . The collaborative platform eTwinning seem slightly more able to reach disadvantaged groups than the rest of the programme . Some of the adult education actions are succeeding in engaging with hard to reach groups, but given the low budget share the scale of these actions have only been small. 5.1.1.3. Outcomes for practitioners Within Erasmus+, practitioner mobility has overall seen a significant increase compared to the past . It remains more common in higher education than in all other education and training sectors put together, although higher education practitioners have comparatively more opportunities abroad through other schemes. It should be noted that an important share of practitioners in the adult education sector teach primarily young people (aged below 24) . This could also be related to the limited effectiveness of the programme in reaching out to practitioners who are not employed in large organisations. In addition, collaborative platforms expand participation in the programme (eTwinning, School Education Gateway , EPALE ) and are associated with positive results for beneficiary practitioners . In particular eTwinning, the platform for schools, attracts a very large number of teachers and pupils, in the EU and beyond, as mentioned previously. In 11 countries, eTwinning seems to be reaching to more than 10% of all school teachers . As a stepping stone into the programme, as noted by some interviewees, it does reach to a broader audience than other types of action . After stagnation in 2011-2013, its audience has been sharply increasing under Erasmus+, pointing at a strong potential of this kind of intervention for the future. Practitioners are even more satisfied with their mobility experience (98% compared with 93% for learners) and very positive about the contribution to their competence development according to the monitoring surveys. For most of the competences the differences across sector are minor . Practitioners who (strongly) agree that they developed a given key competence or skill

Source: ICF calculations based on Erasmus+ participant monitoring surveys. This is a multiple answer question therefore the data does not add to 100%. N = 227,319 Most National Reports refer to results for practitioners with a high level of similarity with the benefits listed above, in particular for education and training staff: language skills , new teaching methods and job satisfaction ; as well as for youth organisations’ staff: professional development and new professional methods . Evaluation surveys found in addition, when comparing with a control group, that participation in the programme is associated with wider networking and cooperation, stronger attachment to Europe, and greater use of digital resources, as shown below : Main counterfactual results for practitioners Source: ICF beneficiary survey for practitioners From virtual collaboration on Erasmus+ funded platforms, beneficiary practitioners also observe positive results, though admittedly with smaller intensity than in the case of mobility or partnerships. Even though it is difficult to differentiate its contribution from other actions , in certain areas practitioners associate eTwinning with more positive results than the control group, such as international networking, but also digital skills (the collaborative platform might attract practitioners that are already digitally skilled, but can also enhance such skills). Findings of other sources support the positive results from ICF’s general surveys: another distinct evaluation of eTwinning revealed that a vast majority of teachers find the platform helpful to network across Europe (64%) or to improve relationships between teachers and learners (62%) . More generally, all practitioners interviewed cited positive results for themselves as well as their learners and organisations . They found their Erasmus+ experience as an opportunity to go ‘outside of the ordinary’ which can be important for retention in the teaching profession, a crucial issue in many countries . As practitioners appear mostly motivated by individual improvement, it is not surprising to find a clearer contribution to development of staff than hard evidence of evolution of pedagogical or institutional practices. Other studies point to positive impact on teaching practices, social competences and motivation for professional cooperation . Recognition is meanwhile less formalised for practitioners than for learners by nature. Only around half of the practitioner survey respondents recorded having received some form of recognition . While a majority of practitioners noted informal recognition by peers, effective recognition by the hierarchy is less common. At the same time, nearly half of respondents state that participation in the programme helped them achieve new roles or positions, which is very positive. It should also be noted that several of the results discussed in this section also have a spill-over effect from individual level to organisational level, as elaborated below. 5.1.2.Outcomes for organisations and systems As for mobility (KA1), the evaluated programmes reached or exceeded the vast majority of the targets when it comes to the number of projects funded (KA2 and KA3). Compared to predecessor programmes, this represents a strong increase in the number of higher education projects , but also a strong decrease in all other sectors. This is due to the lower budget available at the start of the programme but also to the fact that Erasmus+ aims to focus on funding fewer large-scale projects . Erasmus+ is nonetheless perceived as providing more opportunities for cooperation than it was previously possible . However, the fact that Erasmus+ and its predecessor programmes have produced a high volume of intellectual outputs is not necessarily an indicator of project quality . The panel experts have pointed out several issues with the selection process. They consider that the quality of applications selected is in general quite average. According to monitoring data, applications are frequently scored higher than the final outputs. Many projects (other than KA2 Alliances) lack a continuous quality review, external support, or solid needs analysis. National Reports refer to some enhancing-effectiveness factors (e.g. experienced staff on the applicants' side - a condition more often met by large organisations and therefore in higher education, and good cooperation between partners); they also outline challenges which hinder effectiveness (e.g. high-quality projects rejected due to lack of funding). On the contrary, the evaluators find that fewer outputs with clearer added value might possibly meet more clearly the innovation objective of the programme. According to the qualitative assessment of outputs carried out by the expert panel, an area for further improvement of Erasmus+ is the quality of KA2 (and to a more limited extent KA3 which cannot be fully evaluated at mid-term) outputs and especially their effective potential for mainstreaming beyond participating organisations. Although the programme often produces something new for participating organisations , this innovation does not enough follow the state of the art developments in a given sector. While the programme does fund good and innovative activities , these are relatively hidden among many standard projects. The vast majority of case studies identified some form of innovation or change in the organisations reviewed (e.g. use of materials or methods shared during the projects) . Yet often these innovations remain small scale or light touch and limited to a specific part of the organisation. For many funded projects, it appears that the main benefits arise from the participation process rather than the output itself. However, where there are instances of high quality outputs, these could be better supported upstream and thereafter mainstreamed building upon the appreciated precedent of the 2016 specific call in the area of social inclusion . 5.1.2.1. Outcomes for organisations Over 2007-2016, more than 940,000 organisations were contracted, at an average of approximately 115,000 organisations per year for the predecessor programmes but 80,000 under Erasmus+ . During the public consultation, 74% of respondents thought that the Programme improves the quality, innovation and internationalisation in education, training and youth organisations . When asked in the monitoring survey about expected changes to their institutions, more than 55% (up to 85%) of practitioner respondents stated that most such changes (10 out of 11) had indeed taken or were taking place . Later, when asked in the evaluation survey about changes that had happened after their participation , a vast majority of practitioners once again agreed with a high number of types of changes listed, such as the introduction of new teaching materials or new assessment methods . In the youth sector, 75% of practitioners agreed with most statements about organisational results. Lastly, most of other socio-economic actors (companies, public authorities, civil society bodies) stated that their objectives were (fully) met . For them, the most commonly cited barriers concern the complexity of the application procedure and the low success rates in applications, which is due to budget availabilities . Furthermore, the level of cross-sectoral cooperation increased under the current programme by 23 percentage points . Paradoxically, interviewees are often sceptical about the reality of this key evolution. This contrasts with OPC or agency respondents, whom, to a very high extent, recognise the effectiveness of cross-sector cooperation . In the same vein, the network analysis shows promising results in the form of an increase in the number of sectors participating per project. There are notably many more higher education institutions taking part in other strands of the programme than in the past. Cross-sector participation seems more challenging in other education and training sectors where organisations are smaller in average. Reciprocally, youth organisations strongly contribute to the international dimension of the programme and sport organisations take part in other sectors of the programme even more than in the their own strand of Erasmus+. This contrast between perception and observation suggests that there is a need for better communication to potential applicants about cross-sectoral opportunities. In terms of improved geographical balance, the centrality of countries in the programme network is changing. Small countries and countries from central and eastern Europe are better integrated . The main exception remains the country coverage of Jean Monnet activities, with about 25% of its grants going to two countries only .

The surveys also demonstrate a clear internationalisation of organisations, in and outside the EU . In the monitoring survey, practitioners in general most strongly believe that their mobility will lead (85%) or has already led (74%) to internationalisation of their sending institution, as shown in the figures below. For instance, 59% of concerned respondents believe that the Jean Monnet grant helped them to allocate additional funding for teaching or research about the EU. The fact that practitioners have developed strong networks, as demonstrated above, corroborates other findings on the internationalisation of their organisations. Participant organisations are significantly more likely than non-participants to be engaged in transnational cooperation, except in the sport sector . This holds true to the largest extent for higher education institutions and to the least for schools , as confirmed in case studies. Specific findings for the international dimension of the programme underline the strong contribution of the predecessor programmes to the internationalisation of higher education institutions in partner countries through curriculum development and the improvement of management practices of universities . But even in the case of school education, a vast majority of respondents believe, for example, that Comenius transnational partnerships strengthened the European dimension of schools and the survey conducted as part of the eTwinning platform evaluation has observed, as a result, a certain internationalisation of schools. It is particularly important for organisations in the youth and sport sectors which work on specific issues for which they have few counterparts in their countries . Given their scale and the number of organisations reached, it can be concluded that the programmes strongly contribute to strengthening the internationalisation of education, training, and youth sectors. Whether mobility has led or will lead to changes in the sending institution Source: ICF calculations based on Erasmus+ participant monitoring surveys. In contrast, the internationalisation effect is less clear-cut on sport organisations. Competitive sport is already much internationalised independently from any EU intervention, whereas grassroots sport organisations have fewer international opportunities. In the latter case Erasmus+ has not reached the required critical mass. Yet, sport organisations' knowledge and usage of the sport guidelines promoted by the EU (on dual careers or health enhancing physical activity ) suggest some alignment of sport actions with EU policies (see fig. below) . A total of 45% of organisations strongly comply with the good governance in sport principles . Usage of the EU guidelines in the field of sport Source: ICF beneficiary surveys (among practitioner respondents aware of these guidelines)) Only 4 out of 10 participant organisations have strong measures in place to combat doping and violence, while almost half have measures to combat discrimination . Share of respondents who comply with the anti-doping, violence and discrimination in sport principles (% of organisations) Principle Strong measures Medium measures Low measures Anti-doping 40% 27% 33% Violence 41% 38% 21% Discrimination 48% 33% 19% Note: Index (1 (strongly agree)-4 (strongly disagree), 1-1.5=strong measures, 1.51-2=medium measures, less is low measures More generally, another very positive monitoring finding is that overall 91% of practitioners state that they have shared their own knowledge with students and/or other persons after their participation in Erasmus+ . The programme is particularly seen as transformational for youth organisations, where members of staff are significantly more optimistic about the contribution of the programme to their organisations than in other sectors . Interestingly, 71% of practitioners in the sport sector state that the programme contributed to new cooperation with civil society . The literature review points as well to strong positive impact, particularly on: internationalisation (all sectors), teaching capacity (school education, VET), and staff development (VET, youth) , although evidence of benefits for organisations on sport, adult education and Jean Monnet activities is somewhat scarce. Other results are softer and there is less evidence of deep changes of institutional or pedagogical practices . The qualitative findings, however, suggest that a key contribution that was not measured quantitatively is the self-reflection and soft-benchmarking that happens through the opportunities for practitioners to see how other organisations tackle a given issue. In all case studies, concrete changes at organisational level were indeed cited . In many cases these are light adjustments rather than major structural evolutions; however an accumulation of light changes over time can result in a step change. Frequently noted mentions include not only internationalisation (most commonly cited) , but also soft benchmarking, implementation of new tools, materials, increased positive image and with that associated empowerment of practitioners and learners. Other effects for organisations include the development of national partnerships, cooperation with companies or quality frameworks specifically for youth work. On average, 58% of practitioners reported as an outcome the creation of quality assurance approaches . The National Reports confirm that Erasmus+ has three main impacts on participating organisations, namely their internationalisation , the quality of education and training they provide and the professionalisation of youth work . Both mobility actions and Strategic Partnerships allow for the development of new methods, often innovative at least for participating organisations, which can have a positive impact on quality. However, the evidence gathered at EU level in that respect is not sufficiently strong to assess whether the programme is transforming the practices of practitioners and organisations. Though practitioners are positive about the fact that they implement the lessons learnt and results from cooperation projects and mobility exchanges, there is only anecdotal evidence that the programme is changing institutions’ approaches to pedagogy and their target groups. Deeper qualitative work of a larger scale would be needed to see these types of results at final evaluation stage. 5.1.2.2. Outcomes for systems and policies The general literature shows little evidence of system level impacts of preceding interventions, in contrast to the abundant findings about individual impacts. Where such evidence exists, it is positive. But it is very fragmented in all sectors, although stronger in higher education, VET and youth. This is why, since 2014, Erasmus+ focuses through KA3 on funding actions that are specifically designed to have system level effects . Comparable actions existed under LLP and Youth in Action, but some new features were introduced under the current programme , enabling the Commission to address more rapidly emerging needs (e.g. the Paris Declaration, the New Skills Agenda for Europe). The systemic impact of Erasmus+ is only expected "by the end of the Programme" and therefore can only be evaluated at final stage . In particular, most KA3 projects were not finalised at mid-term. Nonetheless some National Reports outline KA3's policy effects although limited by the relatively small funding allocated to KA3 . Not all KA3 case studies could identify system level effects. Although KA3 has the highest potential to produce systemic impact , further involvement of decision-makers is required. It seems therefore too early to conclude on the additional systemic impact brought specifically by Erasmus+. However, even if effectiveness varies across instruments and countries (some use them much more strategically than others ), some policy networks financed through Erasmus+ have proven influential since 2007: the national contact points for EQF or EQAVET , the Bologna experts or the partnership between the Commission and the European Youth Forum . Studies and cooperation with OECD funded by Erasmus+ and LLP have provided Member States with stronger evidence, according to the evaluation of ET2020 .

On the international dimension, the general finding is that the projects funded by previous programmes have contributed to policy and institutional reforms in partner countries, through improved quality assurance, standardisation of higher education and a rapid extension of EU-supported Bologna principles. This has had an evidenced impact on the harmonisation of programmes and curricula in partner countries as well as on mobility and the recognition of studies and qualifications within and between non-European regions and Europe . Furthermore, the effectiveness of the programmes evaluated should also be assessed through their contribution to policies via the Open Method of Cooperation (OMC), which they have substantially supported over the period 2007-2016 . The evaluation of the OMC in education and training (ET2020) noted that according to national stakeholders the outputs of OMC activities have been used to influence national policies and practices. Therefore the programme plays a major role in supporting financially the OMC process, which in turn influences national education and training policies, including those priorities of ET2020 which are not easy to tackle through mobility and cooperation actions (e.g. policies on qualifications design or early school leaving policies). The programmes had also a certain systemic effect in the field of youth. The most recent evaluation of the European Youth Strategy concluded that the OMC was rather positive about the influence on national policies and structures in the youth field, while the results in the youth field might not always be measurable in a way that is easily comparable with the area of education and training. At the same time, the activities funded by Youth in Action and Erasmus+ were identified as strong pillars of the OMC, regarding the structural dialogue with young people and the increasing use of the Youthpass to promote learning outcomes of volunteers. Sport is too recent a programme sector to be evaluated at interim stage in relation to systemic impact . Overall, 88% of respondents to the consultation agreed that Erasmus+ has increased systemic impact compared to the predecessor programmes , whilst 46% estimated that the programmes had been effective in modernising education, training and youth systems . As shown in the figure below, around 80% of the key informant interviewees saw some positive influence of the programme on policy, of which half were able to cite concrete examples. This can be regarded as a good performance . Interviewees tend to be more positive about the systemic influence of the programme in smaller and less wealthy countries. Generally, most frequent examples cited dealt with EU tools related to qualifications. It seems that interviewees no longer think of systemic changes as resulting from the scale of funded mobility. Lastly, the examples remained particularly rare in the sport sector . Share of key informants interviewed who were able to cite concrete policy/ system changes linked to Erasmus+ or its predecessor programmes Legend: As positive were coded responses which named a concrete and influential project or a clear area of influence supported via one of the programmes. As neutral were coded interviews that thought there was some influence but did not give a concrete example. As negative were coded interviews which said they are not aware of any such influence. The interviewees mainly cited four types of changes at system level . Occasionally, even a single funded project can be used as a strategic input at national policy level (e.g. no alternative resource to design work-based learning legislation would have been available in Latvia). More frequently, examples were found where an effort has been made to pool together the findings of several projects to accompany a systemic development inspired by European priorities (e.g. the Polish reform based on ECVET). Thirdly, the scale of mobility actions can trigger a change in national policy to remove barriers (e.g. Erasmus+ beneficiaries are no longer charged a visa fee when going to Turkey). Lastly, programme funded activities can raise awareness of an issue at policy level (e.g. early development of Lifelong Learning Platforms at regional level in Spain). System level influence via individual cooperation projects remains an exception. This does not so much depend on the type of action but rather on the own merits of a given project. Effects at system level are more likely to happen if the project is aligned with country priorities . Even when small scale projects prove to be innovative, unless they had a high level influential stakeholder involved from the beginning they typically would not have had the capacity to reach out effectively to decision makers. It should nonetheless be noted that Knowledge Alliances and Sector Skills Alliances, two actions of a new kind and more ambitious within KA2, could not be evaluated at mid-term but might have a certain impact on systems at the time of the final evaluation of Erasmus+. The evaluation shows that a better way for the programmes to achieve system level impact is by reaching out to a critical mass of practitioners and organisations. This has been the case in higher education where mobility actions, by their volume, led to system level changes e.g. implementation at national level of European credit systems (ECTS) and joint degrees (Erasmus Mundus). Erasmus increased the employability advantage for Erasmus alumni over their non-mobile peers by 45% and half of all European graduates who studied or trained abroad benefited from it according to ex-post impact research . In several of the other sectors, the programme is also close to reaching a critical mass. KA1 alone supported in 2014 1.6% of higher education students, close to 1% of VET students and 0.36% of school staff . However the impact on VET and adult education systems is more difficult to detect because of the scale and the heterogeneous nature of these sectors, which would require more targeted actions considering available funding. In KA2, eTwinning is also instrumental in reaching out to a critical mass of teachers, as described above. Other types of activity do not reach the same volume, but can be influential in areas where system level changes are expected . For the youth sector, although an impact on systems is observed through other channels, it is difficult to evaluate to what extent the programme succeeds in reaching a critical mass . Other sectors such as adult education or sport remain too small to see system level results through this channel. Last but not least, Erasmus+ is expected, in line with its legal basis, to contribute to the Europe 2020 headline target in education , namely (a) higher education attainment and (b) tackling early school leaving. Though the EU countries seem well on target to deliver on these two dimensions, the contribution of the programme and its predecessor to their evolution is mostly indirect. The choice of these overarching indicators for the programme can indeed be questioned in terms of testability and plausibility. Though the programme recitals cover both targets, this is done together with many other priorities. It is true that, as seen above, some countries did make policy changes linked to these two benchmarks. The programme also reaches a high number of learners via KA1 and KA2, whose attitude towards education/training is more positive when compared to control groups. Such a positive attitude can be analysed as a precursor of education retention and as reducing the risk of drop-outs. However early school leavers or people not attracted by higher education are more likely to be found among disadvantaged people. As seen above, both the share of organisations targeting hard-to-reach groups and the share of individuals particularly at risk of exclusion are low within the programme. The programme does improve attractiveness of participating higher education institutions but it cannot be deduced that it attracts people who would not have otherwise studied at higher education level. Lastly, the review of selected projects reveals a weak alignment between the transnational partnerships funded and these two key priorities. Though some potential does exist, the causality link is only indirect. As regards other ET2020 European benchmarks mentioned in the general objectives of the programme, employability of young people has also improved in recent past (77% are employed i.e. +1 percentage point from 2014 to 2015). In contrast, other impact indicators more related to social inclusiveness have registered deterioration over the last years, with worse levels of basic skills and with lower rates of participation in youth out-of-school or physical activities , which could justify a strengthened focus on social inclusion. 5.1.3.Transversal questions about effectiveness When considering all objectives together, 71% of respondents to the consultation thought that the programme is achieving its objectives to a (very) large extent. Education and training providers who responded to the consultation claimed that lack of funding and insufficient language skills are among the barriers hindering the achievement of Erasmus+ objectives. 5.1.3.1. Most effective actions and fields The majority of respondents and countries believe that mobility is essential in ensuring the effectiveness of Erasmus+ . Actions cited as most effective by agency respondents are mobility ones and strategic partnerships, with the latter mentioned with more mixed views . Moreover, longer term mobility actions are associated with stronger results for learners in beneficiary surveys and by a few countries . For practitioners, what matters most seems to be participation itself, irrespective of the type of action . Lastly, KA2 Alliances are perceived by agencies as being less effective for organisations . It can meanwhile be noted that respondents tend to consider the actions closest to their interest as more effective. Comparing the relative effectiveness of Erasmus+ fields cannot be done without keeping in mind the diversity of needs they address and funding available at EU level. Nevertheless, 30 years of Erasmus with a larger and targeted budget allocation makes of higher education the strongest field in effectiveness. In contrast, less funded or more recent fields, such as adult education or sport, not only miss the critical mass but also a similar strong focus to guarantee an equivalent level of effectiveness. School education and VET fields appear in an intermediary position, indicating at this stage the highest potential for improving the overall effectiveness of an EU programme in the future . More generally, the respondents to the consultation have considered factors which could maximize the effectiveness of the programme, as follows:





To what extent do you think the following topics need to be addressed to maximize the impact of any successor to the Erasmus+ Programme? Source: Erasmus+ OPC survey, 28 February – 31 May 2017, ICF analysis. Base indicated next to each topic According to the public consulted, the programme should be better funded, more inclusive and user-friendly while its opportunities should be better known and its outcomes more widely exploited and publicised. 5.1.3.2. Visibility and dissemination of results Erasmus+ is very well-known across Europe and visible beyond. A majority of citizens have heard about Erasmus+ and student exchange programmes come third in the list of the EU's positive results . The dissemination and exploitation of project results are much more effective and systematic with Erasmus+ compared to the predecessor programmes, albeit with nuances according to the geographical level considered . The monitoring was much less performant in 2007-2013, when it was scattered over different tools covering only parts of previous programmes. Good practice examples are used to raise awareness of the general public; to guide potential applicants; to inform the work of practitioners, researchers, social partners, organisations and policy makers. This new approach has been recognised in the public consultation and by the experts . It addresses the objectives of transparency, accountability and visibility , drawing on the lessons learnt from the interim evaluations of predecessor programmes. At EU level, the establishment for the first time of an overarching dissemination and exploitation strategy with regularly reviewed indicators and a single project results platform represent clear improvements in comparison to the past . Specific steps have been taken both by the Commission and the Executive Agency to ensure that the exploitation of projects’ results effectively serves to inform policy making . The project results platform offers a common and accessible space for disseminating most project results whilst also highlighting good practices and success stories (including from LLP and Youth in Action). Its value in comparison to the past lies in structured and selected material. Insights gained from EU interviews suggest that the platform is actually used to inform the work of policy makers at EU level. Content-wise, the main limitation is the lack of complementary external reference materials that could inspire users and a lack of success stories in certain sectors (e.g. adult learning) due to the insufficient quality of reports. At national level, the assessment is also positive about the strategy, setting clear rules for the selection of good practice examples. However, the evidence of the exploitation of project results by policy makers and the effective engagement of the latter when they are not included in the project itself is not clear. The vast majority of National Agencies mainly disseminate examples of good practices to potential applicants or beneficiaries, and only rarely to inform policy making bodies. Moreover, though valued by over half of respondents as shown in the graph below, 37% of agency respondents do not agree that the dissemination platform is user-friendly. This sentiment, more mixed than at EU level, concurs with insights gained from national interviews and reports . Source: ICF experts’ survey Furthermore National Reports refer not only to the dissemination platform, but also to other dissemination channels (e.g. events, website