# EUR-Lex - 52012SA0025 - EN

Important legal notice

*|*

# 52012SA0025

**Special Report No 25/2012 ‘Are tools in place to monitor the effectiveness of European Social Fund spending on older workers?’ **

*

*

ABBREVIATIONS

AIR : annual implementation report

ESF : European Social Fund

EU : European Union

IT : information technology

MA : managing authority

OJ : Official Journal of the European Union

OP : operational programme

EXECUTIVE SUMMARY

I. The European Social Fund (ESF) is a key financial instrument at the disposal of the European Union (EU) to assist Member States in the field of employment. Over the programming period 2007–13 ESF spending will amount to over 75 billion euro, representing around 8 % of the total EU budget.

II. The goal of this audit is to verify whether and to what extent the Member States and the Commission developed and used the tools required to assess whether the objectives planned have been achieved. To this end, the Court selected one of the main disadvantaged groups, namely older workers.

III. The Court found shortcomings concerning the design of the operational programmes (OPs) as well as the monitoring and evaluation systems. Furthermore, it observed that the Commission does not have consistent data at EU level on operational goals, indicators and allocated funds.

IV. As a result, despite the fact that increasing the employment rate of older workers is part of the EU’s strategic goals, neither the Member States nor the Commission are in a position to establish how many older workers have gained new qualifications, or found or kept a job ,after having benefited from an action funded by the ESF. Furthermore, the amounts spent on this kind of action are also unknown.

V. The Court recommends that the Commission should:

(a) require Member States to design their OPs in such a way that the performance of the ESF funds can be measured. The target populations should be unambiguously defined and relevant, quantified operational goals and indicators should be defined to measure outputs, results and specific impacts at target population group level. Intermediate milestones should be set and a hierarchy of target values established;

(b) require Member States to design monitoring and evaluation systems able to measure and explain at appropriate intervals the progress towards all target values set;

(c) obtain consistent and reliable information from the Member States in order to be able to provide appropriate information on the means mobilised and the results achieved by the ESF;

(d) ensure an in-depth analysis of performance issues when assessing and monitoring Member States’ management and control systems; and

(e) improve the documentation of its checks by ensuring that there is an audit trail that allows the extent and consistency of these checks to be assessed.

INTRODUCTION

EUROPEAN UNION EMPLOYMENT GOALS AND THE EUROPEAN SOCIAL FUND

1. In 2000 the Heads of State or Government approved the Lisbon agenda [1] and in 2010 the Europe 2020 strategy [2], on both occasions setting the economic and social strategies of the European Union (EU) for the following 10 years. Both strategic documents include action plans for more growth and jobs and set goals in several fields, in particular regarding employment.

2. The European Social Fund (ESF) is a key financial instrument designed to help the Member States achieve EU employment policy objectives. Its aims are to promote the integration in the labour market of the unemployed and disadvantaged, primarily by supporting training activities. Over the programming period 2007–13 ESF spending will amount to over 75 billion euro, representing around 8 % of the total EU budget.

OPERATIONAL PROGRAMMES AND SHARED MANAGEMENT

3. The ESF is implemented by the Commission through shared management. This means that certain tasks are delegated to the Member States, while the Commission retains ultimate responsibility for all EU expenditure [3].

4. The ESF is put into action through multiannual operational programmes (OPs) prepared by the Member States, negotiated with the Commission and adopted in the form of Commission decisions [4]. An OP is a strategic document. It analyses the problems and needs of the geographical area covered; it identifies the disadvantaged groups which need assistance and it sets the goals to be achieved with ESF funds.

FIGURE 1

OPERATIONAL PROGRAMME: DIAGRAMMATIC OVERVIEW OF PROGRAMMING, IMPLEMENTATION AND REPORTING

+++++ TIFF +++++

Source: European Court of Auditors.

5. To manage and implement the OPs, the Member States designate managing authorities (MAs) having the key responsibility for making sure that the programme is effectively and correctly implemented. To this end, MAs establish and maintain adequate financial management and control systems to carry out the tasks of programming, controlling, monitoring and reporting. The MAs can decide to delegate part of the implementation work to one or more intermediate bodies. The projects are carried out by providers selected by the MAs or the intermediate implementing bodies. The provider supplies them with monitoring information on project progress and the MAs send aggregate information to the Commission in the form of annual implementation reports [5].

6. Figure 1 sets out in simplified terms the main aspects of the programming, implementation and reporting processes for ESF actions.

MEASURING PERFORMANCE

7. As for any item in the EU budget, the implementation of the ESF must comply with the EU’s Financial Regulation and in particular with the fundamental principles of sound financial management [6]: economy, efficiency and effectiveness [7]. This requirement has been incorporated into the Structural Funds regulation [8].

8. For the MAs to be able to report on the results achieved it is essential that the OPs specify, from the outset, quantified operational objectives (target values) and indicators to measure the progress made in the course of the OP. Furthermore, adequate systems need to be set up to monitor and evaluate the implementation of the OP.

AUDIT SCOPE AND APPROACH

9. The goal of the audit was to verify whether the Member States and the Commission have set up and implemented the tools required to assess the effectiveness of the ESF actions addressed to older workers during the programming period. It focused on the content of the OPs and in particular on the quantified operational objectives and indicators as well as on the monitoring and evaluation systems.

10. Older workers are one of the groups highlighted by the EU multiannual strategies. The Lisbon agenda set a specific employment target for this population and it is still a key population for the achievement of the Europe 2020 strategy employment objective [9].

11. There were 117 ESF OPs for the 2007–13 programming period, of which 63 addressed older workers in at least one of the following aspects:

(a) The OP explicitly identifies older workers as a target group.

(b) The OP defines specific indicators to monitor the progress made for this group.

(c) Funds were allocated for measures encouraging active ageing and prolonging working life, i.e. EU priority theme 67 [10].

12. For the purpose of the audit the Court selected six OPs [11] amongst these 63. They account together for 222 million euro, i.e. 22 %, of the total amount earmarked for EU priority theme 67 (1 billion euro) and cover four Member States (Germany, Italy, Poland and the United Kingdom).

13. The audit covered the 2007–13 programming period up to May 2011. For each OP selected, the Court analysed the management systems and carried out checks on their functioning, by examining on the spot a number of projects ranging from two to four, depending on the case. At the Commission, the auditors conducted interviews with the staff responsible and carried out desk reviews. In addition, for the 63 OPs including those visited on the spot, a survey was organised in order to obtain information from the respective MAs regarding the systems they use and how they assess the assistance given by the Commission [12].

OBSERVATIONS

CONTENT OF THE OPERATIONAL PROGRAMMES

14. This section deals firstly with the definitions of older workers used and the supporting information provided and secondly with the different elements necessary for performance assessment (targets, indicators and target values). The third part concerns the connection of the project goals with the OPs’ ones. The last one is about the Commission’s appraisal.

IN THE OPERATIONAL PROGRAMMES THE "OLDER WORKERS" DEFINITIONS ARE NOT COHERENTLY APPLIED AND THE SUPPORTING INFORMATION GIVEN IS DEFICIENT

INCOHERENT DEFINITIONS OF "OLDER WORKERS"

15. While the Lisbon agenda defined an older worker as any person of working age between 55 and 64 years old [13], the survey shows that many MAs use in their OPs different age groups, (i.e. 45-plus and 50-plus to 55-plus, with or without an upper age limit). That may be explained by different socioeconomic and demographic situations.

16. Nevertheless, the definition of older workers is not consistently applied throughout the whole document without justification. This was the case in four of the six OPs selected [14]. In particular, the age group used in the needs analysis differed from the age group used in the priority axes [15] and/or in the related actions and targets (see Box 1).

BOX 1

THE OPERATIONAL PROGRAMME DEFINITION OF "OLDER WORKERS" IS NOT COHERENTLY APPLIED THROUGHOUT THE DOCUMENT

(a) In the Emilia Romagna OP, the one target value set and the measures planned for older workers relate to the 45-plus age group but the description of the situation (including statistical data) and of the needs focuses on the 55–64 age group.

(b) In the Puglia OP, the operational objectives at OP priority axis level and the measures planned relate to the 45-plus age group. However, the description of the situation (including statistical data) and of the needs relates solely to the 55–64 age group.

(c) In the Saxony-Anhalt OP and in the Thüringen OP, the 55–64 age group is used in the description of the situation and for the target values set at OP level for older workers. However, the schemes targeting older workers are open to the 50-plus age group.

DEFICIENT SUPPORTING INFORMATION

17. The absence, to a greater or lesser degree, of quantitative data in the needs analysis made it difficult to apprehend the national or regional background. In particular, context indicators providing quantified information on the socioeconomic situation of the area (e.g. activity rate, unemployment rate) are not used in three of the six OPs selected.

18. The Court found that none of the OPs selected made explicit the causal link between the actions described and the objectives pursued. This hinders the verification of the programmatic consistency between the needs identified, the strategy chosen and the specific goals set at priority axis level [16].

19. Although identified in the selected OPs as a disadvantaged group and in line with the employment objective of the Lisbon agenda and the Europe 2020 strategy, it is an exception to find financial information on actions targeting "older workers" as a specific group [17]. The current regulatory framework provides for financial data at priority axis level only. None of the OPs selected has a priority axis specifically for "older workers".

20. The OPs also include an indicative breakdown of the ESF financial appropriations across the different EU priority themes [18]. However, no priority theme covers specifically the actions in favour of older workers. The EU priority theme 67 "encouraging active ageing and prolonging working lives" is by its wording the most likely to include actions focusing on older workers [19], but the different MAs have various interpretations of the content of the priority theme 67 (see Box 2).

21. Consequently, it is not possible to know how much funding has been allocated in the OPs to the increase of the employment of older workers.

THE TARGETS, INDICATORS AND TARGET VALUES IN PLACE ARE INADEQUATE

22. Targets, indicators and target values should enable the objectives of an OP to be understood and its achievement to be measured. In order to verify the effectiveness of their interventions, the MAs ought to define in their OPs output, result and specific impact indicators and quantified goals for the different populations they chose to assist with ESF funds.

BOX 2

VARIOUS INTERPRETATION OF THE CONTENT OF EUROPEAN UNION PRIORITY THEME 67

(a) In the Saxony-Anhalt OP, priority theme 67 covers the "Active to retirement" actions. Other actions aiming to improve older workers’ adaptability and their access to employment are presented under other priority themes.

(b) In the Emilia Romagna OP, priority theme 67 covers many different actions not necessarily focusing on older workers.

(c) In the England and Gibraltar OP, priority theme 67 does not cover specified actions. In fact, the amount allocated to this priority theme is an estimation of all costs linked to participants older than 50.

23. Figure 2 provides additional information on the different elements needed for a proper performance assessment together with illustrative examples.

FIGURE 2

OPERATIONAL PROGRAMME: OVERVIEW OF POTENTIAL TARGETS, INDICATORS AND TARGET VALUES

+++++ TIFF +++++

Source: European Court of Auditors based on the definitions withheld by the Commission in its indicative guideline on "evaluation methods: monitoring and evaluation indicators" (Working Document No 2 of August 2006).

LACK OF SPECIFIC INDICATORS AND TARGET VALUES FOR OLDER WORKERS

24. In the OPs selected, older workers, although identified as a group to be assisted, did not necessarily have their own indicators or target values. Consequently, it is impossible to assess the effectiveness of the actions addressing their needs (see Box 3).

BOX 3

LACK OF SPECIFIC INDICATORS AND TARGET VALUES FOR OLDER WORKERS

(a) In the England and Gibraltar OP, there are specific output target values regarding older workers in each priority axis addressing them (e.g. for priority axis 1, of all unemployed and/or inactive participants 18 % should be aged 50 or over). However, result and impact target values are set by priority axis for the participants as a whole and not specifically for older workers (e.g. for priority axis 1, 22 % of the participants should be in work on leaving the project).

(b) The Puglia OP mentions older workers explicitly in two priority axes. However, only one of them includes a specific target value, in this case a specific impact indicator (33,3 % of the older workers and migrants are to be in employment a year after the end of the project). All other indicators given are general and limited to outputs (e.g. number of projects).

(c) In the Saxony-Anhalt OP, unemployed older workers are a target group under the priority axis addressing the long-term unemployed and some disadvantaged unemployed groups at risk of becoming long-term unemployed (priority axis C). For older workers only two specific quantified indicators were defined under this priority axis, one output indicator (11 % of the participants are unemployed and 55 or over) and one macroeconomic indicator (reduce by at least one quarter the difference between the regional ratio and the national average ratio of the percentage of those unemployed in the 55-plus age group to all unemployed persons).Neither result nor specific impact indicators (in line with the definition given in Figure 2) were defined for them.

(d) In the Thüringen OP, older workers are an identified target group, falling under the priority axes A and C. Only priority axis A has specific target values for the older workers, notably two quantified output target values (12000 older participants are to be admitted to courses to gain qualifications; the coverage rate of the older worker population is 10 %) and a macroeconomic one (stabilisation of the labour force participation rate for the over-55s at 43,3 %). Neither result nor specific impact target values (in line with the definition given in Figure 2) are indicated.

25. The Court observed also that indicators refer mostly to outputs and results and not to specific impacts. Although finding a job can be seen as the ultimate proof of success of an action aimed at increasing the employment of its participants, such a goal is not systematically quantified and measured in the OPs selected for the different target populations (see Box 3).

GOALS AND INDICATORS NOT ALWAYS DIRECTLY LINKED TO THE EUROPEAN SOCIAL FUND ACTIONS

26. Some of the goals and indicators used in the audited actions have no direct connection with the ESF interventions, thus making it difficult to assess performance.

27. This is the case for the macroeconomic targets set in the OPs (e.g. obtain by 2013 a target employment rate for the 55–64 age group) because success or failure in respect of the achievement of these goals is beyond the control of the ESF actions, as it depends to a very large extent on external factors (e.g. the economic conditions, the social system and pension scheme in place, local investment). Two [20] out of the six OPs selected contain such global impact targets for older workers and 34 of the 63 OPs surveyed contain global impact indicators.

28. The same problem of relevance may arise when the MA uses indicators that encompass actions other than the ESF ones alone. This was observed in the Poland OP where an output indicator refers to all participants of all support measures regardless of the measures’ source of finance (ESF funds or other funds).

ABSENCE OF INTERMEDIATE MILESTONES AND HIERARCHY

29. The OP goals should be achieved by the time of the closure of the programme, i.e. 9 years later [21]. Setting intermediate (annual or biennial) milestones would allow the MAs to identify deviations in the implementation of the OP at an early stage and give them an opportunity to take timely corrective actions.

30. None of the OPs selected included intermediate milestones. However, in Poland the MA sets objectives to its implementing bodies to be reached at the end of the following year. In the England and Gibraltar OP there is provision for two periods for the conclusion of contracts with the providers (2007–10 and 2011–13) so that changes in the economic prospects and results already achieved can be taken into consideration.

31. Having a hierarchy allows the policymakers to make clear their highest priority and helps the implementing bodies to act accordingly. Without a hierarchy there is a risk of conflict between goals (e.g. open the training course to very disadvantaged people vs high percentage of participants finding a job) and of the implementing bodies making an inappropriate choice. None of the OPs selected established a hierarchy between the different quantified goals and only seven of the 63 MAs surveyed reported to have done so.

NO COMMON INDICATORS

32. Finally, in the absence of mandatory common indicators for all OPs and with the exception of the output data regarding the participants [22], the Commission has no standardised performance data at its disposal that could be aggregated at EU level. On this basis, the Commission is not in a position to report on the overall results and impacts of the actions funded by the ESF addressing the older workers.

THE RESULT AND SPECIFIC IMPACT TARGETS OF THE OPERATIONAL PROGRAMMES ARE NOT SYSTEMATICALLY INCORPORATED IN THE PROJECT CONDITIONS

33. For five of the six OPs selected, the Court found that projects usually contain only quantified output goals (e.g. number of participants). Obtaining a certain result or specific impact is rarely imposed on the providers. The exception here is the England and Gibraltar OP, where the result target values set in the OP are systematically incorporated in the project conditions [23].

34. When OP goals are not incorporated in the projects, the risk exists that these goals will not be met at the end of the programming period or that the projects chosen do not contribute to them at all, even when the projects are successful according to their own criteria (see Box 4).

35. Moreover, incorporating into the project agreements result or specific impact goals to be met by the provider open the way to a management alternative. It allows the MAs to link the amount of their payments to the providers to the performance of the projects and no longer solely to the actual costs. Of the six OPs selected, this technique is currently systematically applied only by the England and Gibraltar MA but two others used this approach in a limited way and one was considering a move in that direction [24].

BOX 4

INCONSISTENCIES IN TARGET VALUES BETWEEN THE PROJECTS AND THE OPERATIONAL PROGRAMMES

(a) For the Emilia Romagna OP, the invitations to tender do not contain any target values and the contracts with the providers only contain the number of participants to be met. The one outcome [25] target value quantified at OP level (64 % of the participants over 45 years are to find a job within a year of the closure of the intervention) has not been taken up in the contracts.

(b) For the Puglia OP, the one outcome target value set in the OP covering older workers and that can be incorporated in the project is the gross rate of integration of the participants in employment (33,3 % of immigrants and older workers participating in projects are to find a job within a year of the closure of the intervention). However, this has not been done.

(c) In Saxony-Anhalt, the participation rates laid down in the OP for the different target groups are not reflected in the project decisions. The explanation given was that all the projects have their own distinctive targets. Furthermore, the participation rates and placement quotas included at funding scheme level [26]are not binding for the implementing bodies and the provider because it is considered virtually impossible to forecast these with any precision over the 7 years of the OP.

THE COMMISSION DOES NOT SATISFACTORILY DOCUMENT ITS APPRAISAL OF THE CONTENT OF THE OPERATIONAL PROGRAMMES

36. The role of the Commission [27]is to verify that the OP contains all the information required and that it is in line with other strategic documents [28]. Once these analyses are completed, the Commission adopts the OPs.

37. The Court noted that the Commission verifies the submission and the quality of the data supplied in the OPs. In 2006, in order to ensure a consistent approach by its services, it established a manual [29] containing operational advice on all the main aspects relating to the 2007–13 programming. For transparency reasons, it made it available to the Member States.

38. However, from an audit trail perspective, the Commission does not document in a satisfactory way the controls it performs on the OPs. In order to verify that the required information is provided (called the admissibility check) the Commission’s officers use a common checklist. The checklist is succinct and in most cases there is not any underlying control programme or documentation of the controls carried out. Moreover, for the qualitative assessment of the data provided [30], no checklist exists.

39. As a result, the controls performed by the Commission’s services on the six OPs selected can be only indirectly identified by going through the recommendations and questions raised. Consequently, it is not possible to judge the extent of the verifications done, or that the same approach was systematically applied.

MONITORING AND EVALUATION SYSTEMS

40. This section deals firstly with the monitoring systems and then with the planning and the implementation of the evaluations. The last part concerns the Commission’s appraisal of the implementation of the OPs.

THE MONITORING SYSTEMS DO NOT ALWAYS ALLOW VERIFICATION OF THE PROGRESS MADE TOWARDS THE INTENDED OBJECTIVES

41. It is the role and the responsibility of the Member States to monitor the implementation of the OPs. In order to follow the progress made towards the goals set in the OP, the MAs need either to have access to complete, reliable and timely data at participant level that allows them to launch all the necessary queries or have at their disposal the necessary aggregated information provided to them by the intermediate implementing bodies.

42. The monitoring data are collected through the providers or come from other external sources (e.g. employment ministries or statistical offices). It is of the utmost importance for the sake of consistency that the MA defines upfront the content of the data to be collected and also ensures that the quality is or can, at least, be checked.

43. For the OPs selected, the MAs usually defined clearly the data they required, although cases were identified of incomplete monitoring data (see Box 5).

BOX 5

INCOMPLETE MONITORING DATA

(a) For the Saxony-Anhalt OP, it was only in the first quarter of 2009 that the MA defined the full content of the data to be collected by the providers regarding participants and projects. For the projects which started before, data are missing.

(b) The Thüringen OP includes two indicators regarding the gross employment effectiveness rate which cover older workers, amongst other groups: the first relates to the transition into employment of the participants within 4 weeks of finishing the project and the second to the sustainability of the participants’ jobs after 6 and 12 months. For the second indicator, there is no definition of the exact content and as such no data have been collected.

44. Shortcomings in the method used for data collection can have an adverse effect on the reliability of the data collected. This is the case when the provider does not have to document or justify its data or when the MA has no possibility even to cross-check their reliability. Data protection laws in particular may prevent employment agencies in Member States from divulging information about a participant in an ESF project, even to the MAs [31] (see Box 6).

45. When an independent third party has the responsibility to provide the monitoring data, the data collection method chosen by some of the MAs audited is inadequate, as the information is collected too late for the MA to obtain a timely update on the situation (see Box 7).

BOX 6

QUESTIONABLE MONITORING DATA

In Saxony-Anhalt and Thüringen, the participant in an ESF project is not legally obliged to deliver the correct information and the provider does not have to document or justify the employment status communicated by the participant at the end of the project. Because data protection laws prevent the MA and the intermediate bodies from having access to the employment agencies’ databases, they cannot check the reliability of this data.

BOX 7

SHORTCOMINGS IN THE EXTERNAL DATA COLLECTION APPROACH

For the Puglia OP, although the MA has an integrated IT monitoring system and the capacity to check the employment status of all participants in an ESF project against the employment database, it was decided that the data for the indicator "finding employment for ESF participants" would be collected through surveys organised at national level by the statistical office. Up to the audit visit on the spot (May 2011) no such data had yet been supplied to the MA.

46. The audit highlighted that not all IT systems have built-in controls checking the timeliness, completeness and plausibility of the monitoring data introduced. This increases the risk that the MAs do not have at their disposal the exhaustive and reliable information needed for monitoring the OP (see Box 8).

47. It was also observed that MAs did not put to their full use the data collected in a different context, thus missing an opportunity to enhance the monitoring information at negligible cost (see Box 9).

BOX 8

FLAWS IN THE ELECTRONIC MONITORING SYSTEM

(a) For the Poland OP, the use of spreadsheets by some intermediate implementing bodies to monitor their projects increases the risk of errors. The MA receives consolidated data limited to output. In addition, the two IT systems containing respectively the financial and the participant data are not interlinked.

(b) For the Saxony-Anhalt OP, the intermediate organisations use spreadsheets to follow up the individual projects, as they judge the data in the monitoring system to be incomplete and inaccurate.

BOX 9

AVAILABLE DATA NOT USED

The accreditation procedure in place for the Emilia Romagna OP stipulates that, in order to retain accredited status, providers must meet two predetermined threshold values set by the MA: a 70 % effectiveness rate (being the percentage of project participants who, 12 months after the end of the project, are in employment or are still studying) and a 50 % related employment effectiveness rate (being the number of participants working in jobs related to the training they have received as a percentage of the number of participants in employment). The providers send these data to the MA, which checks their reliability. However, the MA does not use these data at its disposal to follow up any employment effectiveness indicator because the OP has made provision for this indicator to be followed up using information from national surveys organised by the statistical office.

48. Finally, the MAs, in general, and the Commission are not in a position to establish what has been spent on either older workers or on other target groups. The MAs provide financial data in their annual implementation reports for the OP priority axes (amounts spent) and the EU priority themes (amounts allocated) [32], but not at the level of the target groups (see also paragraph 19). Moreover, the figures given for the EU priority themes are not of a consistent nature (see paragraph 20) and for some MAs allocated amounts means committed amounts, while for other MAs it means paid amounts.

SHORTCOMINGS EXIST IN THE PLANNING AND IMPLEMENTATION OF THE EVALUATIONS

49. It is the role and the responsibility of the Member States to evaluate the implementation of the OPs. Evaluation should assist the managers in understanding and explaining why and how specific actions have helped to contribute to the goals set or why they have failed to make any difference.

50. The continuous and timely assessment of a programme’s progress requires not only an ongoing evaluation activity that is built in from the start but also the selection of evaluation topics and approaches that enable an assessment of the effectiveness of the different actions for the different populations.

51. When an OP specifies target groups, quantified goals and indicators, the assessment of the effectiveness of the actions for these groups, targets or indicators should be a key evaluation topic. Otherwise, the MA misses the opportunity to obtain an in-depth feedback on its OP’s performance.

52. The regulatory requirements regarding the evaluations imposed on the MAs are very general [33]. They do not lay down a minimum set of topics to be covered, nor is it compulsory to produce an evaluation plan.

53. The Commission issues non-mandatory guidance notes to assist the Member States in implementing the OPs. With regard to the evaluation issue, the Commission has emphasised the importance of having an evaluation plan, using indicators, performing ongoing evaluations and collecting data [34]. Also, it has been promoting since 1997 the measurement of the net employment effects of the OP [35] as the ultimate tool to assess the effectiveness of employment actions. The updated guidance note mentioned above sets out step by step how to estimate the direct employment effects at programme level and this involves the use of monitoring data on projects.

54. According to the Commission [36], there was a variable use of its guidance across the Member States in estimating the employment impact of the 2000–06 programmes. Regarding the programming period 2007–13, it was observed by the Court that the MAs of the OPs selected opted for the minimum approach: what is not compulsory is not done. Consequently, inadequacies exist in the evaluation approach in respect of the OPs selected (see Box 10).

BOX 10

INADEQUACIES IN THE EVALUATION APPROACH

(a) For the Puglia OP and the Emilia-Romagna OP, no evaluation reports were available at the time of the Court’s audit visit in May 2011 (in the 5th year of implementation).

(b) Except for the England and Gibraltar OP [37] and the Polish OP [38], no evaluations had been planned or performed to assess the impact of the actions on older workers as a specific target group.

(c) Of the six OPs selected, the net employment effect of the actions carried out was only quantified for the England and Gibraltar OP [39]. For the other five OPs, the evaluators had no plans to make such a calculation.

THE COMMISSION PRIORITISES FINANCIAL ISSUES IN ITS APPRAISAL OF THE MANAGEMENT AND CONTROL SYSTEMS

55. The regulation [40] requires the Commission to check the existence and the proper functioning of the management and control systems put in place by the Member States, to verify the annual reports on the implementation of the OPs (AIRs) and to examine the progress made over the previous year.

56. For the examination of the implementation of the OP, the Commission has a multitude of information at its disposal. Its officers receive a copy of all finalised evaluation reports, participate in the meetings of the monitoring committees [41], carry out on-the-spot visits and controls and discuss with the MAs at least once a year the progress made. Finally, they verify the AIRs.

57. When assessing the management and control systems put in place in the Member States, the Commission’s emphasis is on the financial management and the legality and regularity of the actions and not yet on the performance monitoring and evaluation processes. The main goal is clearly the reduction of the financial impact of the errors. The same orientation is evident in the guidance given by the Commission to the MAs regarding their responsibilities in this domain.

58. The Commission is aware that analysing the added value of EU funding, instead of the input or costs incurred, requires a stronger emphasis on non-financial monitoring data [42]. However, this has not been observed for the 2007–13 programming period.

59. Furthermore the documentation by the Commission of its appraisal of the implementation of the OPs suffers from the same main weakness as that of its appraisal of the content of the OP (see paragraphs 38 to 39), namely the lack of a sufficient audit trail. In particular, as the Commission did not develop a standardised format to register the follow-up given to its written observations on the AIRs that were sent to the MAs, it is not possible for the Commission to demonstrate the quality of the process.

CONCLUSIONS AND RECOMMENDATIONS

60. Despite the fact that increasing the employment rate of older workers is part of the EU’s strategic goals, neither the MAs nor the Commission are in a position to establish how many older workers have gained new qualifications or found or kept a job after having benefited from an action funded by the ESF. The necessary tools to provide relevant and reliable information in this respect have not been put in place by most MAs. Furthermore, the amounts spent on this kind of action are also unknown.

61. In the OPs selected, the inability to demonstrate the performance of the ESF funds has two main reasons. Firstly, although output target values were generally set, feasible and relevant results and specific impact goals were not systematically set at OP and at project level. Secondly, shortcomings existed in the collection and registration of the monitoring data and in the organisation of an ongoing evaluation activity. Not having access to reliable, verifiable and timely performance data and assessments of the different actions impedes the drawing of conclusions for current and future decision-making.

62. Several of these weaknesses regarding the design of the OP and the monitoring process were already identified by the Court in its Special Report No 17/2009 "Vocational training actions for women co-financed by the ESF" covering the previous (2000–06) programming period.

RECOMMENDATION 1

The Commission should require Member States:

(a) to design their OPs in such a way that the performance of the ESF funds can be measured. The target populations should be unambiguously defined and relevant, quantified operational goals and indicators should be drawn up to measure outputs, results and specific impacts at target population group level. Intermediate milestones should be set and a hierarchy of target values established. The OPs’ result and specific impact goals should be incorporated at project level, as this would assist achievement of the OP target values and enable payments to be linked to performance;

(b) to design their monitoring and evaluation system in such a way that the progress towards all target values set can be measured in a timely and understandable way, at appropriate intervals, thereby allowing corrective actions to be taken and lessons learned for future decision-making. Such a design requires the timely collection of relevant and verifiable data, the proper functioning of electronic data processing systems, ongoing evaluations, also at target group level, and, for the ESF actions aimed at increasing employment, the measurement of the net employment effect.

63. With regard to the Commission, it was found that it does not have at its disposal consistent and reliable performance data on target values and funds that could be aggregated at EU level, thereby providing overall information on the achievements of the ESF and a fortiori on the attainment of the EU’s strategic employment objectives. The information gap is mainly caused by three facts. Firstly, the legislator has not yet defined the precise content of the EU priority themes, so MAs give a different interpretation to their content; secondly, every MA currently has the possibility to define its own indicators and target values; thirdly, none of the selected OPs provides data on the funds allocated to the different target groups.

RECOMMENDATION 2

For the Commission to provide appropriate data on the means mobilised and the results achieved by the ESF it must obtain consistent and reliable information from the Member States. More specifically, the Commission should issue mandatory common indicators to be included by the Member States in their OPs; the EU priority themes should be in line with EU strategies and their content clearly defined by the legislator.

64. The Commission’s assessment of the management and control systems in the Member States is currently oriented towards payments and financial compliance and not yet towards performance. The main goal of the controls is the reduction of the financial impact of the errors and the same orientation is contained in the guidance given by the Commission to the MAs regarding their responsibilities in this domain. Furthermore, the verifications carried out by the Commission should be better documented. The lack of a sufficient audit trail makes it very difficult to judge the extent and the consistency of the controls.

RECOMMENDATION 3

The Commission should:

(a) analyse in depth performance issues when organising the assessment of the management and control systems; and

(b) improve the documentation of its checks by ensuring that there is an audit trail that allows the extent and consistency of these checks to be assessed.

This Report was adopted by Chamber II, headed by Mr Harald NOACK, Member of the Court of Auditors, in Luxembourg at its meeting of 12 December 2012.

For the Court of Auditors

+++++ TIFF +++++

Vítor Manuel da Silva Caldeira

President

[1] The aim of the Lisbon agenda was to make the EU "the most competitive and dynamic knowledge-based economy in the world capable of sustainable economic growth with more and better jobs and greater social cohesion" by 2010.

[2] Europe 2020, launched in 2010, replaces the Lisbon agenda. It is about delivering smart, sustainable, and inclusive growth. The strategy focuses on five goals to be achieved by 2020 in the areas of employment, innovation, education, poverty reduction and climate/energy.

[3] Article 17(1) of the Treaty on European Union and Article 317 of the Treaty on the Functioning of the European Union (OJ C 115, 9.5.2008, p. 1).

[4] The OP is a document that is produced following a lengthy and detailed negotiation process involving stakeholder consultations, an ex ante evaluation and bilateral meetings between the Commission and the Member States.

[5] The institutional set-up is completed by a certifying authority, whose function is to certify to the Commission that the expenditure being declared for reimbursement is accurate, results from a reliable accounting system and is compliant with applicable EU and national rules. An audit authority audits the effectiveness of the functioning of the control systems and is responsible for the closure declaration and the report to the Commission.

[6] Article 27 of Council Regulation (EC, Euratom) No 1605/2002 of 25 June 2002 on the Financial Regulation applicable to the general budget of the European Communities (OJ L 248, 16.9.2002, p. 1).

[7] The principle of economy requires that the resources used by the institution for the pursuit of its activities shall be made available in due time, in appropriate quantity and quality and at the best price. The principle of efficiency is concerned with the best relationship between resources employed and results achieved. The principle of effectiveness is concerned with attaining the specific objectives set and achieving the intended results.

[8] Articles 14 and 60 of Council Regulation (EC) No 1083/2006 of 11 July 2006 laying down general provisions on the European Regional Development Fund, the European Social Fund and the Cohesion Fund and repealing Regulation (EC) No 1260/1999 (OJ L 210, 31.7.2006, p. 25)

[9] The EU 2020 strategy does not include a specific target value for the employment rate of the older worker population. However, to reach the sole EU quantified employment objective, namely 75 % overall of the 20–64 year-olds to be in employment, it will be necessary to have a higher proportion of the population remaining in work until a later age. Furthermore, the employment rate of older workers is being monitored by the Commission as an element contributing to the Europe 2020 targets in the annual activity report of DG Employment, Social Affairs and Inclusion.

[10] Annex IV to Regulation (EC) No 1083/2006 provides a list of numbered themes, ranging from employment in number 62 ("Development of life-long learning systems and strategies in firms; training and services for employees to step up their adaptability to change; promoting entrepreneurship and innovation") to number 74 ("Developing human potential in the field of research and innovation, in particular through post-graduate studies and training of researchers, and networking activities between universities, research centres and businesses"). Priority theme 67 is titled "Measures encouraging active ageing and prolonging working lives". The regulation requires the MAs to provide in their OPs a breakdown of the ESF funds they plan to spend on these themes, and in their annual implementation reports the cumulative sums committed for these themes. The estimate of the funds to be spent is indicative, meaning that the MAs are not bound by it.

[11] The OPs covered are Emilia Romagna (Italy), England and Gibraltar (UK), Poland, Puglia (Italy) and Saxony-Anhalt and Thüringen (Germany).

[12] The survey was launched in May 2011 and concluded in September 2011.

[13] Definition used by the 2001 Stockholm European Council complementing the Lisbon agenda.

[14] The exceptions are the OPs of England and Gibraltar and of Poland.

[15] In each OP, the MA defines the priorities on which it will spend ESF funds (e.g. adaptability of the workforce, access to employment, technical assistance). These priorities are called priority axes and comprise, at a lower level, operations and actions that will be implemented by providers through projects.

[16] The current regulatory framework requires the setting of specific goals only at priority axis level.

[17] Of the six audited OPs, this was only possible for the England and Gibraltar OP.

[18] According to the model set out in Annex XVI to the Corrigendum to Commission Regulation (EC) No 1828/2006 of 8 December 2006 setting out rules for the implementation of Council Regulation (EC) No 1083/2006 laying down general provisions on the European Regional Development Fund, the European Social Fund and the Cohesion Fund and of Regulation (EC) No 1080/2006 of the European Parliament and of the Council on the European Regional Development Fund (OJ L 45, 15.2.2007, p. 3).

[19] In a 2012 European Commission publication (Demography, active ageing and pensions, Social Europe guide,Volume 3, p. 31), active ageing is described as "encouraging older people to remain active by working longer and retiring later, by engaging in volunteer work after retirement and by leading healthy and autonomous lives". This definition clearly goes beyond any employment goal for the 55–64 age group.

[20] The OPs Saxony-Anhalt and Thüringen.

[21] The current OPs run from 2007 to 2013. As the MAs have an extra 2-year wrap-up time at their disposal, the real time span of the OPs is 9 years.

[22] Corrigendum to Regulation (EC) No 1828/2006 sets out in Annex XXIII the quantitative data on participants the MAs need to communicate to the Commission by priority axis. The data relate to the number of participants, to be broken down by gender, by status in the labour market, by age, by vulnerable group and by educational attainment.

[23] In Poland job effectiveness rates were established in 2011 for some measures to be introduced at project level. However, at the time of the audit visit in May 2011, no projects had yet been launched under this new procedure.

[24] Poland is considering introducing such an approach in the future. At the time of the audit visit in May 2011, however, no projects had been launched yet under this new procedure. For the Emilia Romagna OP, the payments for some training courses — not falling under EU priority theme 67— are in a way linked to the successful participation of the trainees. For the Puglia OP in some projects covering +/- 390 women, payments were dependent on 75 % of the participants finding a job.

[25] Outcome is used here in the sense of result or specific impact.

[26] In Germany, ESF OPs are implemented through regional (Land) administrative funding schemes. These define the conditions for funding, the kind of actions and the administrative procedures to follow. It is not necessary for the MA or the implementing body to launch their own ESF funding schemes; existing ones can be used.

[27] Article 32 of Regulation (EC) No 1083/2006.

[28] The other strategic documents in question are the national strategic reference framework and the Union strategic guidelines on cohesion. These two documents, the first prepared by the Member States and the second by the Commission, link the Structural Funds and cohesion policy to the achievement of the Lisbon, and now Europe 2020, goals.

[29] European Commission, DG Employment, Social Affairs and Inclusion, "Programming period 2007–13: Aide-Mémoire for the desk officers", November 2006.

[30] It should be noted however that for the Saxony-Anhalt OP and the Thüringen OP, the Commission wrote a report with comments on the different sections of the OP.

[31] This is the case for the MAs for Poland, Saxony-Anhalt and Thüringen. A possibility, currently not explored by the MAs, might be to obtain from the employment services aggregated (hence anonymous) data for the ESF participants as a group.

[32] Article 67 of Regulation (EC) No 1083/2006.

[33] Articles 37, 47 and 48 of Regulation (EC) No 1083/2006 stipulate that the OPs must contain a description of the evaluation system and that evaluations need to be carried out before, during and after the programming period by experts or bodies functionally independent of the ESF certifying and audit authorities and that the Member States should provide the resources necessary for carrying out evaluations, organise the production and gathering of the necessary data and use the various types of information provided by the monitoring system.

[34] Working paper No 2 — Indicators for monitoring and evaluation (August 2006) and Working paper No 5 — Evaluation during the programming period (April 2007).

[35] Working document No 6 — Measuring structural funds employment effects (March 2007).

[36] Working document No 6 — Measuring structural funds employment effects (March 2007, p. 3).

[37] The evaluators of the England and Gibraltar OP discuss in their report findings at an overall level and at the level of the different populations targeted by the actions, including the older workers group. This is particularly the case in the gender equality and equal opportunities study, the cohort studies and the "Early impacts of the European Social Fund 2007–13".

[38] For the Poland OP, two reports were issued by intermediate implementing organisations after the audit visit of May 2011 regarding the impact of certain actions on the professional activity of older workers (in one report the 45-plus age group and in the other report the 50-plus age group). A third report covering the 55-plus group is planned to be carried out in 2012.

[39] The evaluators for the England and Gibraltar OP published in May 2011 a calculation of the ESF net employment effects (‘Early Impacts of the European Social Fund 2007–13), albeit using a different method from the one recommended by the Commission.

[40] Articles 14, 67, 68 and 72 of Regulation (EC) No 1083/2006.

[41] The Member States set up for each OP a monitoring committee. Its main task is the supervision of the implementation of the OP by the MA.

[42] Annual report concerning the financial year 2010 (OJ C 326, 10.11.2011, p. 131).

--------------------------------------------------