Commission staff working paper - Annex to the Communication from the Commission to the Council, the European Parliament, the European Economic and Social committee and the Committee of the Regions - Mid Term Evaluation of the Multiannual Programme for Action in the Field of Energy “Intelligent Energy Europe, 2003-2006” - Summary of the Evaluators’ Report March 2006 [COM(2006) 357 final]
Date: 2006-07-04
Institutions: European Commission
Subjects: EU programme, competitiveness, energy policy, environmental protection, job creation, renewable energy, security of supply, sustainable development













EUR-Lex - 52006SC0858 - EN






Important legal notice




|



52006SC0858

Commission staff working paper - Annex to the Communication from the Commission to the Council, the European Parliament, the European Economic and Social committee and the Committee of the Regions - Mid Term Evaluation of the Multiannual Programme for Action in the Field of Energy “Intelligent Energy Europe, 2003-2006” - Summary of the Evaluators’ Report March 2006 [COM(2006) 357 final]  /* SEC/2006/0858 */  








[pic] | COMMISSION OF THE EUROPEAN COMMUNITIES |Brussels, 4.7.2006SEC(2006) 858COMMISSION STAFF WORKING PAPER ANNEX TO THECOMMUNICATION FROM THE COMMISSION TO THE COUNCIL, THE EUROPEAN PARLIAMENT, THE EUROPEAN ECONOMIC AND SOCIAL COMMITTEE AND THE COMMITTEE OF THE REGIONS Mid Term Evaluation of the Multiannual Programme for Action in the Field of Energy “Intelligent Energy Europe, 2003-2006” Summary of the Evaluators’ Report March 2006 [COM(2006) 357 final]This report combines independent analysis of programme data, strategic rationale, views of Commission staff and views of the stakeholders that have received funds. The report examines the relevance, efficiency, effectiveness, added value and sustainability of the programme and concludes with recommendations as to how the evaluators feel the programme should go forward and be improved. The conclusions and recommendations of this report provide a valuable insight into the work being supported by the Commission now, and that planed for the future, to address the barriers to the uptake of energy efficiency and renewable energy.Programme DescriptionThe Intelligent Energy - Europe programme (IEE), established by Decision No 1230/2003/EC of the European Parliament and of the Council of 26.06.2003, is the main Community instrument for non-technological support in the field of energy. Its approach addresses the market barriers that still exist to the efficient use of energy and the increased use of new and renewable energies. The objectives of the programme have a clear match with a number of energy and other EU policy objectives. Within energy the programme contributes to the EU policy objectives: emission reductions, security of supply (by reducing the amount of energy used and by increasing the use of indigenous renewable energy resources), energy efficiency and the uptake of renewable energy sources and sustainable energy technologies. Other EU policy objectives that the IEE programme contributes too include the Lisbon Strategy - in both environmental terms and via the competitiveness benefits that a more energy efficient Europe offers.Through the programme, Community support is provided to actions within four fields :-  Energy efficiency: SAVE actions aimed at exploiting potential energy savings.-  New and renewable energy sources: ALTENER actions help to substantially increase the use of new and renewable sources of energy.-  Transport energy: STEER actions address all aspects of more sustainable energy use in transport.-  Co-operation with developing countries: COOPENER actions focus on capacity building and training in developing countries.The IEE budget is used in two ways: through grant agreements and through purchase of services. There are four types of action: Type one actions (the majority) concern for example capacity building, networking between market actors, promotion, education and training, pilot actions for market transformation, and analyses to prepare standards, policies and regulations; Type two actions for the creation of new local and regional energy agencies; Type three actions to support conferences and events; and Type four actions, also called Concerted Actions, to which a number of restrictions apply. For types 1, 2 and 3 funding is on a shared-cost basis with the EU providing up to 50% of the eligible costs. For Type 2 there is an additional limit on funds at €180.000 (now raised to €200.000) contribution per new agency during a three-year period. For Type 3 there is also an absolute ceiling (€40.000 - per event). Type 4 actions can only occur where the Commission and the Member States are directly concerned in relation to the implementation of Community policy. Costs are shared such that the purely national activity is borne by each participating Member State. All other costs resulting from performing these activities together with other countries in order to enrich the own activities (“Community added value”) are funded 100% by the EIE programme.With regard to the purchase of services, the Commission launches direct calls for tender and the activity is 100% funded.Programme Budget and Expenditure (€m) to December 20052003 Call (allocated) | 2004 Call (allocated) | Total to date | 2005 Call (plan) | 2006 Budget (Estimate1) | Total Budget1 |CALLS FOR PROPOSALS (grants) |Type 1 Actions |SAVE | 14.5 | 14.3 | 28.9 | 13.5 | 7.8 | 50.1 |ALTENER | 17.1 | 12.9 | 30.1 | 15.4 | 19.6 | 65.1 |STEER | 1.9 | 7.4 | 9.3 | 6.3 | 9.9 | 25.5 |COOPENER | 5.8 | 3.3 | 9.1 | 5.0 | 0.0 | 14.1 |Horizontal KAs | 5.3 | 10 | 15.3 | 6.0 | 6.0 | 27.3 |Type 2: Creation of new agencies | 1.9 | 4.5 | 6.4 | 4.0 | 4.0 | 14.4 |Type 3: Events | 0.4 | 1 | 1.4 | 1.0 | 0.9 | 3.3 |Type 4: Concerted actions | 0 | 2.18 | 2.2 | 0.0 | 0.0 | 2.2 |TOTAL GRANTS | 46.9 | 55.6 | 102.7 | 51.1 | 48.2 | 202.0 |Services | 7.8 | 6.6 | 14.4 | 8.2 | 7.3 | 29.8 |IE Executive Agency | n.a. | 2.77 | 2.77 | 5.13 | 5.06 | 13 |TOTAL | 54.7 | 64.9 | 119.8 | 64.4 | 60.5 | 244.8 |1 The split between actions for the 2006 call will not be confirmed until the call is prepared.The first call for proposals (2003/04)[1] under the 2003 budget appropriations resulted in 98 contracts being signed while the second call for proposals (2004/05)[2] under the 2004 budget appropriations call led to 126 contracts. The average cost of all the type 1 and 2 projects funded to date is €533,000 with €108m of support. More than 1,600 organisations are involved in the implementation of the signed contracts. Nearly 65% of these are private organisations. There is an equal share of private enterprises (energy service companies, energy suppliers, consultancies, manufacturers) and private non-profit organisations (energy agencies, industry associations, chambers of craftsmen). More than 80% of these private enterprises and organisations are Small and Medium Size Enterprises. Participating public organisations include authorities, research institutes and public agencies, including regulators and test facilities. The average number of partners per project is 9. For the majority of the contracts signed to date the duration is between 24-36 months.The Intelligent Energy Executive Agency (IEEA) was established by Commission Decision No 2004/20/EC of 23.12.2003. It began operations in late 2004 and is responsible for the administration of the programme. At beginning of December 2005 it had 18 technical, 10 financial and 12 administrative staff (which will increase to 25, 10 and 11 respectively when fully staffed) concerned solely with the programme operation[3].Within Directorate General Energy and Transport (DG TREN) there are 18 staff involved in the strategic operation of the programme, as well as other duties related to overall programme management and co-ordination with other initiatives.The Decision establishing the Intelligent Energy – Europe programme requires, in its Article 9(2), the Commission to arrange, at the end of the second year of the period of application of the programme, an external evaluation by independent experts of the overall implementation of the actions carried out under this programme. Since the first contracts related to the actions/ projects (co-)financed by the programme only started at the end of 2004 or in early 2005, it is still too early for evaluators to identify their long-term impacts. Therefore, this evaluation is designed to provide the following:-  To draw conclusions on the programme relevance, effectiveness and efficiency, including on its management, and to be in a position to integrate indicators into the monitoring of current and future actions.-  To allow the Commission to judge the suitability of renewing and extending similar activities.-  To recommend action, if necessary, to improve the programme management, architecture and the specific means of intervention, including those related to funding.Method and executionThe evaluation was carried out in close cooperation with a steering committee composed of representatives of the Commission services and the Intelligent Energy Executive Agency. The research comprised five main elements:1. A desk review of relevant programme and broader strategy documents to develop a sound understanding of the intervention logic and IEE programme activity to date.2. Compilation and analysis of available statistical data relating to the IEE programme.3. An on-line questionnaire-based survey of those who have applied for funding (successful and non successful). This was used to capture qualitative and quantitative views and information on the efficiency and effectiveness of the IEE programme.4. To add depth to the questionnaire responses, a series of follow-up interviews with individuals who responded to the questionnaire were carried out.5. Consultations were carried also out with Commission officials and IEEA personnel responsible for administering the IEE programme, key stakeholders and representatives of other EU institutions, such as the European Parliament.ConclusionsBased on the survey that has been carried out, the evaluators can conclude that the Intelligent Energy – Europe programme represents, in most aspects, a considerable improvement in relation to the previous Energy Framework Programme (EFP) and its specific programmes, be it in terms of concept and scope, in the type of implementation measures and as well as in management and administration. In the preparation and running of this programme, the Commission has taken into account many of the recommendations made by the external panel in the evaluation of the previous EFP, and has been successful in incorporating the advice into the IEE programme. This includes the recommendations relating to programme integration and improved co-ordination with other initiatives, as well as the externalisation of management.Programme RelevanceHaving reviewed the policy context for the IEE programme and consulted a number of stakeholders, programme users and Commission staff, the evaluators have drawn the conclusion that the need for the programme has increased since its start. This is informed by the strength of fit of the programme aims with the aims of the Lisbon Strategies. There is also increasing pressure to achieve these aims as evidenced by documents such as the Green Paper on Energy Efficiency and the increasing profile of security of supply and global warming issues.For the COOPENER field consultees felt that there remains a strong need for a programme of this nature which addresses two problems - access to energy and climate change. However, it is also recognised that there can be some conflicts between poverty alleviation and reductions in carbon emissions and that the hierarchy of objectives could have been better explained as being directly linked to those of the European Union Energy Initiative (EUEI).Programme EfficiencyThe majority of those consulted felt that given the current nature of the programme and the type of projects eligible for support, the budget could be considered adequate. The level of applications in comparison to the budget seems to support this, since the programme is not excessively under or over subscribed. However, the aims of the programme are a high priority for the EU, the problems being addressed are of a deep rooted nature and there is a low level of confidence among some stakeholders consulted in the ability of the programme as it stands to significantly influence these issues. Therefore many stakeholders (and the evaluators) agree that there is a good case for enlarging the programme to include support to new type of projects designed to encourage the mass deployment of best available technologies and best practices, and for increasing the budget accordingly in the future. Some of those consulted reported that they would like a more flexible approach to the level of funding available, particularly for COOPENER projects as the traditional donor funding model in international development projects is for 100% support. However for all the other aspects the majority opinion was that the 50% funding was sufficient, easy for applicants to understand and allows the demonstration of a genuine commitment to the project by the applicants.The projects funded are considered high quality. However in comparison to the predecessor Energy Framework Programme, the IEE programme is attracting fewer proposals though the budget (overall and per project funded) has increased. There has been some success in attracting new entrants and new types of market actor. However, there is also evidence to suggest that the pool of applicants could be larger. This is thought to relate to a number of barriers to participation to new entrants such as the complexity and time consuming nature of the application and appraisal process and a lack of knowledge of programme aims. There is evidence that new applicants take longer to arrange their consortiums and complete the forms than applicants who have been through the process before. This problem is observed in most programmes that have a relatively long track-record.When considering geographical spread of applicants the programme would ideally like a higher number of applicants from new Member States. This is for reasons of equity but also because there is a perception that they can benefit from adapting good practices of the old Member States. Generally it can be noted that all eligible countries are actively participating in the programme and its calls for proposals with a success rate equally balanced between the countries.Our survey suggests that applicants are generally satisfied with the level of information provided to them on the nature of the programme and the nature of projects which are required. They are also generally satisfied with the application forms and recognise that these have evolved and improved over successive calls for proposals. When considering this comment it should be borne in mind that the majority of the applicants have experience of previous and similar programmes so they have already overcome the barriers to participation, discussed earlier, faced by new applicants. There is no strong evidence from our consultations of a demand for the application process to become on-line.From our discussions with DG TREN staff the proposal evaluation process appears to be in line with their standard evaluation procedures. The use of external experts was generally agreed by those consulted to add value to the process. However there was concern among some DG TREN staff that the amount of internal staff resource required to complete the evaluation is high and could be higher than that required in the comparable 6th Framework RTD programme (FP6). However, one should bear in mind that, as far as DG TREN involvement in FP6 is concerned, there is an external service provider to assist the Commission in the evaluation process, there is a lower number of proposals and a dedicated staff resource in DG TREN Directorate A, i.e. the- RTD Co-ordination Cell.Previous external evaluations of the predecessor programme and internal reports on the evaluation process have suggested that a two stage application process could both reduce the barriers to entry to new applicants and potentially reduce the evaluation staff resource requirement. However, the Commission need to assess this recommendation against its own previous experience in using a two stage application. For example some Commission staff reported that this approach was not entirely successful in the RTD Framework Programme.Our survey of applicants revealed that a number of them felt that the negotiation stage after their application had been approved was inconsistent. The IEEA management of this procedure should help improve this. The idea of a facilitated meeting between the coordinators of related projects during the negotiation phase has been successfully trialled with a view to making it a more common approach.In line with the recommendations of earlier external evaluations, an Executive Agency, the IEEA, has been set up and staffed and is now in a position to formally carry out its duties. All of those consulted have been impressed with the quality of the staff attracted to the IEEA. Early indications are that even during the last year, when operating partially staffed under supervision from DG TREN, the IEEA had a positive effect on the efficiency of programme administration.Once contracts have been awarded applicants expressed concerns over the requirement to have a bank guarantee and the staging and delays to payments.Programme EffectivenessPrevious evaluations have highlighted the need for projects to develop and use indicators of success which show how the project outputs lead to measurable impacts in line with the aims of the programme. Our consultations have confirmed that the applicants are supportive of this need and are willing to address it. The project application form does now request information on indicators of success and the associated guidance does give some explanation of what is required. However many applicants have reported that they still do not fully understand what is required.In the view of the evaluators and those consulted, a simple impact indicator for all projects is agreed to be carbon saved as a result of improved energy efficiency or installed renewable energy. If the project ultimately achieves carbon savings it should also help to achieve the other objectives of the programme, e.g. improved security of supply and competitiveness. However there are a number of reasons why using carbon savings alone would be problematical. These include complexity, difficulty in ensuring consistency, time delay in impacts and the difficulty of isolating the importance of projects upon individual carbon saving decisions. For this reason a larger suite of output indicators directly related to the projects are required. In association with these the applicants should be required to show how these outputs (e.g. number of people attending a conference) lead to outcomes (e.g. improved understanding of sustainable patterns of energy use by those attendees) which in turn produce impacts in line with the programme objectives (e.g. installation of energy saving and/or renewable technologies).Previous evaluations have highlighted the need to improve the dissemination of project results, both individually and collectively. The application form now clearly stresses the importance of this aspect and accepted projects are often encouraged to improve their dissemination plans during negotiation. It is too early to say if the projects funded to date have successfully disseminated their results. There is a view among applicants and stakeholders that collective dissemination of results could be improved.Programme Added ValueIn the view of the evaluators an important aspect of added value that the IEE programme brings is the strength of the synergy between the fields. This can have the effect of bringing together actors in renewables, energy efficiency and transport. Evaluations of predecessor programmes and our consultations have pointed out that the projects funded have inspired member state policy changes and interventions. It is too early to say that this will occur as a result of the projects being currently funded. However, in the view of the evaluators as the projects are of the same nature as in the predecessor programmes it is reasonable to assume that this will occur. This aspect should be covered in the ex-post evaluation of the IEE programme.Furthermore, stakeholders and the evaluators agree that the programme does not duplicate the work of other funding streams, and in fact complements the work of other programmes such as the EC Framework Programme of RTD. The trans-national nature of the programme makes it unique in the field of sustainable energy promotion. This adds value to national, regional and local programmes and initiatives, enables fruitful exchange of knowledge and experience, promotes benchmarking and also helps to avoid duplication of efforts.Programme SustainabilityGiven that the programme has only been operating for two years it is not possible to assess the extent to which positive effects actually have lasted after the intervention. Therefore this evaluation has concentrated on the future of the programme. The future plans for the IEE programme intend to make it part of the larger Competitiveness and Innovation framework Programme (CIP) with a larger budget and the addition of funding for market replication projects. Furthermore, the Commission plans to relocate the external dimension of the programme (COOPENER) to an external aid thematic programme on environment and sustainable management of natural resources, including energy. These plans were positively received and were viewed by those consulted as an opportunity to raise the profile of the programme as long as certain aspects, mostly related to concerns over retaining visibility, budget and autonomy in management, were retained.If COOPENER is to be funded under the new external relations Thematic Programme, the opinion was raised that the Commission staff charged with its management should include those with a high level of energy expertise, and that there should be a close relationship between this part of the thematic programme and the future IEE, notably in order to help SMEs take advantage of the potential markets for intelligent energy which exists outside the EU. A coordinated/joint management of the two programme parts could help to achieve this, as is the case in the current IEE.RecommendationsDrawing on the conclusions described above this evaluation makes the following recommendations:1. The current IEE programme should continue as it has a strong rationale and is attracting worthwhile projects, which contribute to achieving the objectives of EU energy policy and to implementing the Lisbon Agenda.2. The majority of those questioned felt that given the nature of the programme and the type of projects eligible for support, the budget as it currently stands is adequate.3. In order to help attract new applicants and more applicants from the new Member States:-  Hold more targeted and themed information days.4. In order to further improve the quality of submitted proposals and reduce the evaluation workload-  The guidance to applicants is helpful and needs to continue to be developed, in order to continue accommodating the requirements and requests for information coming from applicants.-  The Commission should reflect on the effectiveness of having a two stage application process - either for the final call of IEE or in the successor programme. This consideration should take the form of an analysis of the adequacy and interest of introducing such a procedure in the light of previous experiences.-  If type 3 projects (support for events) are continued in any follow on programme to the IEE, the possibility of adopting a 'lighter touch' appraisal process for them should be considered.5. In order to improve the uptake and usefulness of project indicators of success.-  Continue to improve the guidance to applicants through additional and simple guidance with examples, explaining the output - outcome - impact chain.-  Continue the work initiated by the IEEA to improve the application form and the guide for proposers.6. In order to improve the quality of collective dissemination of project results.-  Improve the profile and ease of use of the Intellibase[4] resource and similar tools.-  Extend the practice of group meetings between co-ordinators of related projects during the negotiation stage.7. It has been felt by most of those consulted that DG TREN and the IEEA have made, during the 2 first years of the programme, a positive contribution to the efficiency of the programme. It is too early to formally comment on the effectiveness of the IEEA or the appropriateness of its staffing levels. This should be the subject of a formal evaluation at a later date, notably to determine the cost-effectiveness of delegating the management of future IEE-II programme, or of the CIP as a whole to this executive agency.8. With regard to the future of the IEE programme we have the following recommendations:-  There is very strong evidence of the need for the programme so it should continue with an enlarged scope - reinforced to include support to new project types aiming at enhancing the deployment of best available technology, achieving mass replication of best practice and attracting new applicants and key actors. Accordingly, the successor programme should have a larger budget.-  Moving IEE into the CIP may attract new actors to apply but it is important that it retains its identity and independence in order to build on its historical profile.-  Early indications are that the IEEA are improving the efficiency of the programme and as such they should be retained, and expanded, to run the IEE as part of the CIP and possibly the whole of the CIP (if formal evaluation of their performance justifies this).-  The addition of funding for replication projects requires a much closer co-ordination between CIP and the future FP7 in order to avoid overlapping benefit - this issue will need to be addressed in the drafting of the future programmes and the procedures for evaluation of bids for funding.9. With regard to the COOPENER element of the programme we have the following recommendations:-  There should continue to be a close integration with the EUEI institutional process, since COOPENER is part of the EUEI.-  The links with the EC delegation in ACP countries should be improved.-  In the context of the new architecture for external relations’ programmes in the period 2007-2013), moving the programme to an international development and economic cooperation instrument, the “thematic programme on environment and sustainable management of natural resources, including energy” (as proposed) would deal with the problems of co-financing and target country sub contractor status. However, it will be essential to ensure that COOPENER retains its specificity, autonomy and visibility within this new thematic programme.-  The management of the future COOPENER needs to retain energy expertise as well as international aid expertise, though the latter is the more important.[1] TREN/DIR D/SUB/04-2003 of 24.12.2003[2] TREN/DIR D/SUB/05-2004 of 23.12.2004[3] An organigramme of the IEEA is available at: http://europa.eu.int/comm/energy/intelligent/ieea/organigramme_en.htm[4] http://europa.eu.int/comm/energy/iebase/introduction.cfm