Special Report No 10/2012 ‘The effectiveness of staff development in the European Commission’
Date: 2012-07-28
Institutions: European Court of Auditors
Subjects: European Commission, European official, continuing vocational training, job mobility, professional career, work productivity













EUR-Lex - 52012SA0010 - EN






Important legal notice




|



52012SA0010

Special Report No 10/2012 ‘The effectiveness of staff development in the European Commission’  








GLOSSARYCareer development : The management of careers by individuals and organisations (e.g. through job moves and promotions).Career guidance : Career guidance aims to help staff manage their careers (e.g. by providing advice on how to make best use of their skills in line with their aspirations). Career guidance in the Commission is provided both at central level by the Central Career Guidance Service (SCOP) or at local level by the Local Career Guidance Officers (ReLOPs) appointed by each directorate-general.Certification : Procedure allowing officials from the Assistant (AST) function group to become members of the Administrator (AD) function group following successful completion of training and examinations.Community of practice : A group of people with a common interest who interact regularly to share learning.Competency : Skill or ability to carry out a task proficiently.E-CV (electronic curriculum vitae) : A module of the Commission’s human resource management information system where staff can input data on their work experience, education and skills.E-learning : Computer-enabled learning.EPSO (European Personnel Selection Office) : EPSO organises and conducts selection procedures on behalf of the European Union Institutions. The decisions to recruit successful candidates are taken by each Institution.Formal learning : Intentional learning that is organised in terms of objectives (e.g. instructor-led training or structured e-learning).HR scorecard : Commission document showing a range of indicators concerning staff in post and vacancy rates.Informal learning : Learning which is not formally organised or structured (e.g. coaching, on-the-job learning, sharing experiences with colleagues).Inter-DG mobility : Mobility from one directorate-general (DG) to another.Intra-DG mobility : Mobility within the same directorate-general (DG).Job and competency planning instrument : Tool currently being developed to help identify future job needs in the Commission.Job information system : System showing, for each post in the Commission, the experience, education and skills required to carry out the functions related to the job.Knowledge : Familiarity with facts and information.Learning and development : The process of acquiring or improving knowledge and skills.Learning and development framework : Commission document identifying strategic training needs and planned training activities.Learning environment : An organisational climate that encourages staff to participate in learning activities and supports them in applying new skills in the workplace.Mobility : The movement of Commission officials from one job to another within the same directorate-general (DG) or from one DG to another.Performance level : One of five categories of performance resulting from the annual appraisal of the jobholder’s efficiency, ability and conduct.Promotion : Advancement to the next higher grade.Screening exercise : Annual analysis of the balance between Commission administrative and operational staff.Skills : Ability to carry out tasks proficiently.Staff development : Includes training, formal and informal learning, job mobility and all other aspects of human resource management linked to improving the knowledge and skills of employees.Syslog : The Commission’s training management information system.Training : Transfer of knowledge and skills.Training coordinator (COFO) : In each directorate-general, the member of the unit in charge of learning and development who is responsible for the design and implementation of the policy on learning and development. COFO meetings are regularly organised by the Directorate-General for Human Resources and Security (Human Resources and Security DG).Training map : List of training courses to be attended in the coming year for each member of staff agreed with their Head of Unit.Training path : A series of courses designed to allow learners to progressively improve their knowledge and skills.Underperformance : Continued failure of a staff member to meet the requirements of their job — including meeting targets (efficiency) or providing a satisfactory service (abilities and conduct). In the context of the appraisal in the Commission, underperformance is defined as less than 9,5 points before 2008, or performance level IV from 2009 until 2011.Workforce planning simulator : A tool which shows the probabilities, based on past records, of staff leaving a DG.EXECUTIVE SUMMARYI. The Commission depends upon its 33000 staff in order to achieve its objectives. To perform effectively, staff need to acquire and maintain up-to-date skills through training, informal learning and job moves. This is particularly important in the Commission because of the long career and low turnover of its permanent staff (paragraphs 1 to 5).II. In order to examine how effectively the Commission enables its staff to develop the audit addressed the following four questions (see paragraphs 6 and 7):(a) Does the Commission align staff development with organisational needs?(b) Does the Commission provide opportunities for staff to develop?(c) Does the learning environment motivate staff to develop their skills and to apply them in the workplace?(d) Does the Commission evaluate the effectiveness of the actions taken to develop staff?III. The Commission does not have sufficient consolidated information on the existing skills of its staff or the skills which they need. The introduction of a new appraisal system in 2012 and further development of the electronic curriculum vitae (e-CV) aim to improve the information available on staff skills. The Commission’s strategy for developing staff does not convincingly demonstrate how development actions will contribute to achieving the objectives of the organisation. Likewise, individual training maps were not sharply focused on them. Some skills gaps are not sufficiently addressed and managers consider that some development actions, such as some language courses and job moves, do not result in greater workplace effectiveness (see paragraphs 8 to 21).IV. The Commission provides a wide range of opportunities for staff to develop through training, informal learning and job moves. In 2010, staff attended an average of 6,9 days formal training and spent an estimated 4 days on informal learning. Each year between 2005 and 2010 on average 6 % of staff moved to another DG. Staff also have considerable opportunities to change jobs within the same DG (see paragraphs 22 to 28).V. The Commission has not created a sufficiently strong learning environment to capitalise on the extensive learning offer (see paragraphs 29 to 42):(a) Commission systems do not closely monitor whether staff participate in planned development actions. In 2010, staff attended only 35 % of the courses planned in their training maps.(b) Although Commission staff participated in an average of 6,9 days training in 2010, 30 % of staff participated in less than 2 days’ training. Older staff on higher grades generally participate in less training.(c) There are high absence and dropout rates from language courses.(d) The Commission’s own staff and managers deliver some training courses (16 % of general and IT training) but not enough to demonstrate that the organisation attaches a high value to staff development.(e) There is only limited support to apply new skills in the workplace.(f) The appraisal and promotion system in place until 2011 does not sufficiently distinguish between good performers who develop their skills and poor performers who do not. The new system introduced in 2012 aims to make a clearer distinction and not to promote those whose performance is below average.VI. The Commission measures the satisfaction of staff with development actions. However, it does not assess whether staff have attended necessary training. Nor does it assess whether they have acquired new skills (with the exception of language and certification training). There are some attempts to evaluate the utility of development actions in the workplace. However, these are mainly based on the opinions of staff. Managers are rarely asked for their opinion on the effectiveness of training undertaken by their staff and there is little use of objective indicators. The Commission does not evaluate the impact of development actions on organisational results. Consequently, it does not have the information necessary to demonstrate the contribution of development actions to achieving organisational objectives or to inform decisions on where to target learning and development resources (see paragraphs 43 to 58).VII. On the basis of these observations the Court’s main recommendations are that the Commission should (see paragraphs 59 to 68):(a) ensure it has sufficient consolidated information on existing staff skills and on those needed to meet future challenges and prepare a strategy which convincingly demonstrates how learning and development will contribute to the achievement of organisational goals;(b) support this process through improvements to the systems for planning training and job moves;(c) develop its systems for monitoring participation in development actions;(d) address the issue of underperformance and encourage greater participation in the wide range of development opportunities available while recognising staff who develop their skills and those of others;(e) test and certify the acquisition of new skills where practicable, and support their application in the workplace by providing follow-up activities;(f) evaluate how effectively development actions provide staff with new skills which they are able to apply in the workplace.INTRODUCTION1. The Commission depends upon its 33000 staff [1] to achieve its objectives. In order to perform effectively, staff need to acquire and maintain up-to-date skills. This is particularly important in the Commission because of the long career and low turnover of its permanent staff who represent 68 % of the workforce. The Commission can bring in fresh expertise in the form of certain types of staff on fixed-term contracts [2]. However, in the current context of staff reductions continuous personal and professional development is essential for staff to make their most efficient and effective contribution to meeting the objectives of the Commission.2. Staff maintain and develop skills through training, informal learning and job moves. The annual budget of the Commission on external trainers and learning materials in 2010 was 26,6 million euro [3]. In addition, the annual investment in learning and development activities in 2010 included 230000 staff days participating in training and the equivalent of 310 staff administering and delivering training and career development activities.3. The directorate-general in charge of human resources (Human Resources and Security DG) is responsible for identifying the strategic learning and development needs of the Commission as a whole. It is also responsible for managing the central training offer (training courses proposed to staff) including language training. Based on the central learning and development strategy, directorates-general (DGs) develop their own learning and development frameworks and manage the local training offer aimed at their own staff. Middle managers (Heads of Unit) are responsible for identifying the development needs of their staff.4. Individual staff members are expected to play an active role in meeting the needs identified and in developing their personal potential. Responsibility for training is shared between the individual and the institution [4].5. The 2000 White Paper on reforming the Commission [5] emphasised the importance of learning and development and led to the following key strategic documents:(a) the 2002 decision on staff training which aimed to increase the annual number of training days from 2,5 days in 2000 to 10 days in 2005;(b) the 2002 Guidelines on Mobility [6] which highlighted the importance of job moves both for the development of the individual and for contributing to the achievement of organisational objectives;(c) the new Staff Regulations of 2004 [7] which introduced a career structure intended to offer staff clearer incentives for good performance.AUDIT SCOPE AND APPROACH6. The audit examined how effectively the Commission enables its staff to develop through training, informal learning and job moves and how these actions are aligned with organisational objectives. The audit addressed the following four questions:(a) Does the Commission align staff development with organisational needs?(b) Does the Commission provide opportunities for staff to develop?(c) Does the learning environment motivate staff to develop their skills and to apply them in the workplace?(d) Does the Commission evaluate the effectiveness of the actions taken to develop staff?7. The audit was based on:(a) a review of documentation relating to the Commission’s staff development policies, procedures and tools;(b) structured interviews with Commission staff in the Directorate-General for Human Resources and Security (Human Resources and Security DG) and five selected directorates-general: the Communication DG, the Development and Cooperation DG — EuropeAid, the Environment DG, the Information Society and Media DG and the Regional Policy DG. These interviews were followed up with written notes of the meetings agreed with the DGs concerned. The five DGs were selected from different areas of Commission activity in order to provide a representative cross-section of systems and opinions;(c) a survey of 227 middle managers (mainly Heads of Unit) in the five selected DGs. The survey asked their views on various aspects of staff development. 127 managers (56 %) replied to the survey, representing 10 % of all Commission middle managers. In addition the audit examined the results of the Commission’s own staff opinion survey, carried out every 2 years, most recently in 2010 [8];(d) an analysis of statistics generated from Commission HR systems on various aspects of staff development;(e) a review of previous evaluations relating to aspects of staff development in the Commission.OBSERVATIONSTHE COMMISSION HAS INSUFFICIENT CONSOLIDATED INFORMATION TO CLOSELY ALIGN STAFF DEVELOPMENT WITH ORGANISATIONAL NEEDS8. In order to align staff development with the needs of the organisation the Commission needs reliable information on existing skills and on those needed to meet current and future challenges. This information is necessary to produce a strategy which links development actions with policy objectives. Such a strategy provides a framework for planning the development of individual staff. The audit therefore examined whether the Commission has sufficient information on staff skills enabling it to produce a staff development strategy and individual development plans in line with the needs of the organisation.THE COMMISSION IS GRADUALLY IMPROVING SYSTEMS TO PROVIDE CONSOLIDATED INFORMATION ON EXISTING AND REQUIRED SKILLS9. The Commission has three main systems for providing information on the existing skills of its staff:(a) Recruitment assessments: as part of the recruitment process all new Commission staff are assessed by the European Personnel Selection Office (EPSO) against eight core competencies. The results of the assessment are recorded on a competency passport;(b) Annual appraisals: throughout their career the conduct, ability and efficiency of staff in carrying out their job is assessed in the annual appraisal [9]. A new appraisal system was used from January 2012 onwards;(c) Electronic curriculum vitae (e-CV): staff can also record knowledge, skills and experience acquired from current and previous jobs on their e-CV. This provides information on their potential to work in other roles.10. These systems do not combine to produce a reliable, consolidated picture of the skills of Commission staff. The EPSO competency passport is available for staff recruited since 2010. It examines skills in a different structure to that of the annual appraisal in use until 2011 (see Figure 1). As a result it is not updated or followed up during the appraisal process. The Commission will use the same framework of eight competencies as EPSO in the new appraisal process introduced from 2012, allowing easier alignment between the two tools.FIGURE 1STRUCTURE OF SKILLS ASSESSED BY ANNUAL APPRAISAL IN USE UNTIL 2011 AND EPSO COMPETENCY PASSPORT FOR NEW RECRUITSSource: European Court of Auditors (ECA) analysis of criteria used in Commission annual appraisals and in the EPSO competency passport.Annual appraisal until 2011 | EPSO competency passport for new recruits |Efficiency | Organising and planning work | Prioritising and organising |Performing work and ensuring quality | Delivering quality and results |Ability | Technical skills | |Oral and written communication | Communicating |Communication in meetings |Negotiation skills | |Analysing problems and applying solutions | Analysis and problem solving |Awareness of the working environment | |People management | |Conduct | Teamwork | Working with others |Service culture | |Commitment to the job | |Personal development in the context of work | Learning and development |Leadership | Leadership || Resilience |11. The annual appraisal and promotion system in use until 2011:(a) awarded points to staff which they accumulate in order to achieve promotion (advancement to the next higher grade) when they have reached a predetermined threshold of promotion points. Considerations relating to promotion dilute the performance focus of the appraisal [10];(b) does not reliably identify underperforming staff [11]. Since 2004 only 37 cases of underperformance have been identified through the appraisal system (relating to 28 different people). In 2010, there were three cases of underperformance out of some 21700 permanent staff appraised. An indication that underperformance is in reality more widespread is the fact that 32 % of middle managers responding to the audit survey stated that they had faced situations of underperformance.12. The e-CV can be used to make informed decisions on staff moves and so enable staff to work in jobs where they can apply or develop their skills. Such a tool is particularly important because only half of staff movements take place following published vacancies [12]. It can complement the existing information from informal networks and personal contacts. Although it has been the Commission’s human capital database since 2007 [13], the Commission has not developed a simple and effective search tool for the e-CV and has not formally launched it or asked staff to complete it. By the end of 2010, only 25 % of staff had completed their e-CV. Of these, only 20 % allowed all Commission managers to access it. Consequently, managers can see the e-CVs of only 5 % of staff.13. The Commission has a number of systems to help DGs to identify the skills they need to meet current and future challenges. These include the workforce planning simulator, the HR scorecard, the screening exercise and the job information system (see the Glossary for a brief explanation of these tools). However, the systems currently focus on staffing numbers, and DGs lack systems to specify the future skills they require. The Commission is developing a job and competency planning instrument (not yet in production) which is intended to help DGs develop strategic HR plans which identify the skills needed to achieve policy priorities.THE CENTRAL TOP-DOWN STRATEGY DOES NOT CONVINCINGLY DEMONSTRATE HOW STAFF DEVELOPMENT WILL CONTRIBUTE TO ACHIEVING THE OBJECTIVES OF THE ORGANISATION14. The last multiannual strategic document on learning was the 2002 decision on staff training. A strategy for 2012 to 2015 is currently being prepared. In the meantime, the Commission produces a learning and development framework annually. These annual frameworks emphasise the quantitative target of 7,5 days formal learning [14] and 2,5 days informal learning. They do not make a strong link with the Commission’s policy objectives. For example, they do not articulate the objectives of language training, which accounts for one third of the proposed formal training offer, in terms of the working needs of the organisation. The annual learning and development frameworks produced at DG level make a stronger link between training and the objectives of the organisation. They explain how the decentralised training provided at DG level aims to contribute to meeting the DG’s objectives.15. Moving jobs is a way for staff to develop and learn new skills. The 2002 Guidelines on Mobility stressed that job moves should benefit both the organisation and the individual. The Commission has recognised for some years [15] that a more active mobility policy is required so that job moves address organisational needs and not only the considerations of the individual staff member [16].16. However, there is no multiannual plan for developing staff through job moves in order to contribute to meeting organisational objectives. There are, however, some initiatives which aim to better align mobility with organisational objectives. For example, the Regional Policy DG has plans to establish a Professional Development Committee which aims to align staff moves with policy objectives by filling vacancies in a way which best meets the needs of the service.BOTTOM-UP PLANNING THROUGH TRAINING DIALOGUES, TRAINING MAPS AND CAREER GUIDANCE IS NOT SHARPLY FOCUSED ON ORGANISATIONAL OBJECTIVES17. The middle manager has an important role to play in ensuring individual requests for learning and development correspond to organisational needs. Managers discuss learning needs with their staff annually, in the context of the annual appraisal, resulting in individual training maps (see Figure 2). Training paths exist for some posts, which provide guidance on the recommended and optional training.18. The audit survey of middle managers found that 90 % of respondents thought that training maps took into account the needs of both the individual and the organisation. In terms of individual needs, 75 % of staff stated that the learning offer met their needs in the 2010 staff opinion survey [17]. However, in terms of organisational needs, the audit survey of middle managers found that respondents considered that training in staff appraisal was more helpful in assessing past performance than in identifying future development needs. Only 44 % of respondents considered the training in staff appraisal helped them to define training maps for their staff. The 2010 staff opinion survey found that only 42 % of staff agreed that their manager supported them in helping to identify training and development needs [18].FIGURE 2SYSTEMS FOR PLANNING TRAINING+++++ TIFF +++++Source: ECA on the basis of Commission procedures.19. There is evidence of some training which is not sharply focused on organisational needs:(a) Language training represents one third of target training days (see paragraph 14). The audit survey of middle managers found that 36 % of respondents did not consider that language training helped staff to do their job better (compared with 12 % of respondents for IT training and 14 % for general training). Figure 3 shows the number of participants in different language courses between 2004 and 2010. The most widely used languages in the workplace are French and English, and these accounted for 35 % and 18 % of all language courses respectively. Other languages are generally less directly or immediately useful in the workplace although they may have medium to long-term benefits.FIGURE 3PARTICIPANTS IN LANGUAGE COURSES 2004 TO 2010+++++ TIFF +++++Source: Human Resources and Security DG.(b) Although the training offer in the form of the training catalogue is intended to respond to overall training needs, staff may choose more attractive courses rather than those which correspond to their real needs (e.g. training in mind-mapping or improving collaboration rather than financial procedures). From 2012, the Commission stopped using training maps. It has replaced them with a section on learning in the annual appraisal report. This identifies useful training for the future, rather than selecting specific courses from the catalogue to be attended in the coming year. This is intended to strengthen the link with the objectives of the organisation.(c) In two of the five DGs interviewed, managers sometimes use training as a form of soft reward and to compensate for limited career prospects.(d) Only 56 % of staff in delegations considered that the learning offer met their needs [19], compared with 75 % in the Commission as a whole (see paragraph 18). The Court’s Special Report 1/2011 highlighted a lack of expertise in delegations in the areas of macroeconomics, public finance management, health and education despite their importance for the millennium development goals and poverty reduction [20].20. In the absence of a strategy for developing staff through job moves, managers and career guidance services have an important role to play in aligning job moves with organisational goals. However, although the annual appraisal provides an opportunity for staff and their line managers to discuss future plans, there is no record of career development plans. Such plans could be used in conjunction with the e-CV to make informed decisions on staff moves.21. Although staff who consult the central career guidance service (SCOP) are generally satisfied with the quality of the advice [21], most staff are not aware of the services offered [22]. The Commission also runs a training programme in preparation for changing posts which was attended by 150 staff in 2010 (whilst some 3000 staff move jobs each year). Some DGs, for example the Communication DG (see Box 1), have an active local career guidance function (ReLOP). However, the local career guidance function in three of the five DGs interviewed has minimal staffing (e.g. the equivalent of 0,1 full-time staff in the Development and Cooperation DG — EuropeAid and the Information Society and Media DG and 0,2 in the Environment DG), lacks information on supply and demand, deals mostly with problem cases and is not intended to support mobility. As a result, Commission staff generally plan and manage their own careers through job changes and there is a risk that job moves may not be aligned with organisational needs.BOX 1CAREER GUIDANCE IN THE COMMUNICATION DGThere are two career guidance officers in the Communication DG who proactively provide a range of advice and seminars. The career guidance service aims to be part of a general process of career development and not only to be used to help resolve urgent problems following conflicts. Around 90 Communication DG staff members used the career guidance service in 2009 and 80 in 2010.The career guidance service offers impartial, confidential advice and guidance on:(a) how to make better use of skills or how to strengthen them;(b) how to identify personal strengths and weaknesses;(c) how to face new challenges;(d) how to present a CV;(e) how to write a letter of motivation;(f) how to prepare for interview with a selection panel.As well as services to individuals, the career guidance service organises events for groups of staff, for example, career guidance seminars for staff concerned by rotation (compulsory mobility) and seminars for women interested in becoming managers.The career guidance service has also put in place a "Welcome team" for newcomers. This contacts newcomers and provides them with information at the time of their arrival, and then follows this up with conversations a few months, and then one year, after their arrival.Fourteen months before the rotation date, the career guidance service contacts those affected by compulsory mobility to find out their job preferences. It encourages these officials to update their e-CVs and tries to meet them when they are in Brussels. The career guidance service also promotes, in headquarters, the future vacant posts in the representations. To evaluate the effectiveness of the rotation exercises the career guidance service contacts staff a short time after rotation and then 9 months later to ask them about their motivation and integration.Source: ECA on the basis of interviews with the Communication DG.THE COMMISSION PROVIDES A WIDE RANGE OF OPPORTUNITIES FOR STAFF TO DEVELOP22. To enable staff to develop, it is necessary for them to have access to appropriate training and opportunities to move jobs. The audit therefore examined whether the Commission provided sufficient learning and development opportunities for its staff.THE COMMISSION’S LEARNING OFFER IS EXTENSIVE23. The Commission offers an extensive range of formal learning opportunities. The Human Resources and Security DG organises language training centrally for all Commission staff. It also aims to organise general training (e.g. on HR policies, personal development and financial procedures) where this is the most cost-effective or appropriate approach [23]. In addition, DGs organise training locally mainly for their own staff. The Informatics DG (DIGIT) is the directorate-general responsible for IT training and the European Administrative School (EAS) [24] is responsible for training in induction, certification and management. Figure 4 shows the number of trainer days and participant days for the three main categories of training (general, IT and language training) [25] organised at central and local levels.FIGURE 4TRAINER DAYS AND PARTICIPANT DAYS 2010+++++ TIFF +++++Source: Human Resources and Security DG.24. E-learning is being developed at both central (Human Resources and Security DG) and local levels (other DGs), often blended with coaching or a classroom-based exchange of best practice. The Human Resources and Security DG has established a team of three staff to provide central guidance on e-learning. Across the Commission, e-learning represented 0,7 % of participant days in 2010 (1700 out of a total of 236000 participant days). The Human Resources and Security DG offers a variety of e-learning courses on soft skills and has also introduced an e-learning option for some language courses. The Development and Cooperation DG — EuropeAid has developed a number of e-learning courses for staff serving in delegations, and e-learning represented 6 % of its participant days in 2010, up from 3 % in 2009.25. Training is coordinated by the community of training coordinators (COFOs), which holds formal meetings every six weeks. DGs frequently open their courses to other DGs (for example, 20 % of the participants in the Regional Policy DG’s courses come from other DGs). The Development and Cooperation DG — EuropeAid makes learning available not only to other DGs, but also to other institutions and international organisations (see Box 2). However, the extent of coordination varies and there are practical constraints. For example, the Communication DG restricts its locally organised and financed courses to its own staff, despite the interest of other DGs in attending. Also, DGs do not have access in the Commission’s IT system for managing training (Syslog) to details of all courses organised by other DGs. They are therefore not able to check in Syslog if a course which they are planning to develop already exists.BOX 2COOPERATION WITH OTHER ORGANISATIONSThe Development and Cooperation DG — EuropeAid cooperates with other international organisations through the Joint Donors Competence Development Network (Train4Dev). The network was established in 2003 and consists of some 30 major development aid agencies. The network aims to improve aid effectiveness through enhanced donor cooperation in competence development and training. It designs and delivers joint training in areas critical to the development agenda. Members open some of their courses to other members’ staff.Source: ECA on the basis of interviews with the Development and Cooperation DG — EuropeAid.26. From 2005 to 2009, Commission staff participated in an average of almost 8 days formal training per year (see Figure 5). In 2010, the average was 6,9 days (consisting of 3,5 days general training, 2,7 days language training and 0,7 days IT training). The drop in 2010 reflects the recent tendency to reduce the duration of individual courses in order to minimise absence from the workplace.27. To complement the formal learning offer for general, language and IT training, the Commission also encourages informal learning (e.g. coaching, on-the-job learning, sharing experiences with colleagues and self-study in the Learning Centre [26]). The target of 10 training days per year includes 2,5 days of informal learning (see paragraph 14). Although informal learning is not recorded in Syslog, the audit survey of middle managers found that respondents spent an average of some 5 days in 2010 participating in informal learning activities [27]. Those respondents who could make an estimate replied that their staff spent on average some 4 days on informal learning.FIGURE 5AVERAGE NUMBER OF DAYS TRAINING PER STAFF MEMBER+++++ TIFF +++++Source: Human Resources and Security DG.STAFF HAVE OPPORTUNITIES TO DEVELOP BY CHANGING JOBS28. Opportunities for career development through moving jobs are available to staff in the Commission. The 2002 Guidelines on Mobility encourage staff to move every 5 years, which would imply an average annual mobility of 20 %. Between 2005 and 2010, the inter-DG mobility rate was around 6 % (in the order of 1400 staff) per year [28]. Statistics on staff who move jobs within the same DG are not available. However, the most recent progress report on mobility in 2007 showed that intra-DG mobility represented some 60 % of all mobility [29]. This would mean an intra-DG mobility rate of 9 %.The resulting overall mobility rate of 15 % represents satisfactory progress towards the guidance of 20 %.THE COMMISSION HAS NOT CREATED A SUFFICIENTLY STRONG LEARNING ENVIRONMENT TO CAPITALISE ON THE LEARNING OFFER29. Although the Commission provides extensive opportunities for development, this does not necessarily mean that staff participate in these activities. Furthermore, mere attendance at training courses is not proof of increased ability and does not necessarily impact on effectiveness in the workplace. The audit therefore examined the extent to which staff actually took part in development activities and whether the Commission supported them in applying new skills at work. The audit examined whether the Commission motivated staff to take part in development activities by monitoring participation and giving sufficient recognition to staff who demonstrated a commitment to self development.THERE ARE VARYING RATES OF PARTICIPATION IN DEVELOPMENT ACTIVITIESSTAFF ATTEND 35 % OF THE COURSES PLANNED IN THEIR TRAINING MAPS30. In 2010, staff attended only 35 % of training courses planned in their training maps. The Commission suggests that part of this non-fulfilment is because staff attend other courses than those listed in the training map which meet the same learning needs. Also, staff training needs change during the year, for example, due to a job move or a change in responsibilities. Nevertheless, there is a significant difference between the learning needs identified in the training maps and the courses attended in practice. The 2010 staff opinion survey found that only 58 % of respondents felt their managers supported them in attending the training agreed in their training map [30] (down from 79 % in 2008).THERE ARE HIGH ABSENCE AND DROPOUT RATES FROM LANGUAGE COURSES31. In 2010, levels of absences [31] were 9 % for general training, 11 % for IT training and 29 % for language training. They have remained at similar levels since 2005. The duration of language courses (2 weeks for intensive courses and 4 months for twice-weekly courses) partly explains their high rates of absence. Language courses were also considered the least useful by respondents to the audit survey of middle managers (see Figure 6). In order to reduce absence rates in language courses, the 2003 evaluation of interinstitutional language training recommended that language training should only be allowed if the language was needed at work [32]. Participation in training which is not useful incurs unnecessary costs in terms of the time spent by the participant as well as the cost of the trainer.32. In 2010, 23 % of participants dropped out of language courses [33]. The dropout rate increases depending on the duration of the course and for twice-weekly courses over 15 weeks amounted to 28 % (see Figure 7) [34]. Dropouts and absences increase the cost per participant in training. At the end of 2011, the Commission launched a pilot e-learning project to offer a more flexible approach to learning for five languages, partly to address the problem of absences and dropouts from language courses.FIGURE 6USEFULNESS AND ABSENCES+++++ TIFF +++++Source: Human Resources and Security DG and audit survey.FIGURE 7MORE DROPOUTS FROM LANGUAGE COURSES OF LONGER DURATION (2010)+++++ TIFF +++++Source: Human Resources and Security DG.OLDER STAFF ON HIGHER GRADES TAKE PART IN FEWER DEVELOPMENT ACTIONS33. Although Commission staff participated in an average of 6,9 days’ training in 2010 (see paragraph 26), 30 % of all staff participated in less than 2 days’ training. Figure 8 shows the range in the number of days training attended by Commission staff in 2010.34. Figures 9 and 10 show that younger staff on lower grades participate in most training. On first starting work in the Commission they are required to attend induction courses. Older and higher-grade staff in the Commission participate in less training. Some older staff also perceive difficulties in finding other posts to move to [35]. The 2008 evaluation of older staff in the Commission highlighted the difficulty of managing and motivating high-grade non-management staff in their fifties. The extended working life in the proposed revision of the Staff Regulations [36] will further increase the importance of motivating older staff to develop their skills.FIGURE 8RANGE IN NUMBER OF DAYS TRAINING ATTENDED IN 2010+++++ TIFF +++++Source: Human Resources and Security DG.FIGURE 9TRAINING DAYS BY AGE IN 2010+++++ TIFF +++++Source: Human Resources and Security DG.FIGURE 10TRAINING DAYS BY GRADE IN 2010 (HIGHER-GRADE STAFF HAVE A HIGHER AD NUMBER)+++++ TIFF +++++Source: Human Resources and Security DG.COMMISSION SYSTEMS PROVIDE INSUFFICIENT INFORMATION ON STAFF PARTICIPATION IN DEVELOPMENT ACTIONS35. Syslog does not provide information on compulsory courses which staff should attend, for example those for new managers or new recruits. Of the five DGs interviewed, only the Regional Policy DG monitored whether staff had registered for and attended the compulsory courses they should have [37]. The audit survey found that all managers in the Regional Policy DG had attended the compulsory training for new managers, compared with an average non-attendance rate across the five DGs of 28 % [38].36. The Human Resources and Security DG monitors staff movements between DGs but not within DGs [39]. Progress reports on mobility were produced until 2007 and included data on intra-DG mobility. The section on mobility in the annual HR report does not cover intra-DG mobility.THE COMMISSION’S OWN STAFF DO NOT DELIVER SUFFICIENT TRAINING AND COACHING.37. The involvement of an organisation’s own staff in the delivery of training reflects the value which it attaches to learning [40]. It can also help to make training more practical and therefore, in the view of the respondents to the audit survey of middle managers, more effective. When staff deliver training they are required to update their knowledge, so that training itself becomes a learning activity. The managers replying to the audit survey delivered an average of 0,8 days training in 2010, equivalent to 1200 days across the Commission. Commission staff delivered a further 1800 days training. Together, Commission staff and managers delivered 16 % of the 19000 trainer days for general and IT training (see Figure 4). The level of involvement varies between DGs. The Regional Policy DG and the Information Society and Media DG, for example, use internal trainers to deliver most of their local training courses. The 2010 staff opinion survey found that only 27 % of staff considered that their manager coached them on a regular basis [41].THE COMMISSION PROVIDES LIMITED SUPPORT TO APPLY NEW SKILLS IN THE WORKPLACE38. The annual appraisal provides an opportunity for staff and managers to discuss the impact of training followed. However, support to apply learning in the workplace and follow-up activities to reinforce learning are limited to a small number of programmes (see examples in Box 3).39. The 2010 staff opinion survey found that only 45 % of staff considered that their manager supported them in implementing learning in the workplace (down from 61 % in 2008). Only 25 % of staff considered there was strong support in their DG to help them implement what they had learnt [42]. The audit survey of middle managers found that only 18 % of respondents had participated in follow-up activities for management training. 53 % said they needed more so that they could apply it more effectively in the workplace.BOX 3EXAMPLES OF FOLLOW-UP ACTIVITIES TO REINFORCE AND APPLY LEARNINGThe external management development programme uses business schools. Participants complete a pre-course questionnaire, an evaluation directly after the training and a further evaluation 3 months later. These all ask participants how they expect to implement, or how they have implemented, the learning in their day-to-day work and how much their performance has improved.Participants in the negotiators’ learning path have an individual feedback session with the trainer after the initial 5-day seminar. Once they have completed the fundamental negotiation skills trainings they become a member of the Negotiators' Club and can access advanced courses. The members of the Negotiators' Club receive information on negotiation learning events and literature on negotiating skills.The Development and Cooperation DG — EuropeAid aims to send follow-up messages over a longer time period in order to reinforce training and keep learning alive. Training courses have a practical focus which emphasises the applicability of the learning content because staff are required, in order for the organisation to function effectively, to use the acquired skills and knowledge in the workplace.Source: ECA on the basis of interviews with the Human Resources and Security DG and the Development and Cooperation DG — EuropeAid.THE COMMISSION DOES NOT SUFFICIENTLY RECOGNISE STAFF WHO LEARN AND APPLY NEW SKILLS40. Staff who learn and apply new skills should improve their performance. The administrative reform launched in 2000 aimed to base promotion more on merit [43]. However, promotions continued up to 2011 to be based on elements not solely related to performance, for the following reasons:(a) The Staff Regulations provide for a certain proportion of staff to be promoted within a certain time [44].(b) The promotion system in use until 2011 was based on the allocation of points on the basis of the performance level achieved. Promotion points were cumulative over the years and promotion was awarded on passing a threshold of points.41. Furthermore, managers consider the measures for dealing with underperformance to be ineffective:(a) The audit survey of middle managers found that only 11 % of respondents considered that the measures for identifying and dealing with underperformance were effective.(b) Only 32 % were confident that they would receive the necessary support in tackling underperformance.(c) Several free-text comments received in response to the audit survey of middle managers highlighted the ineffectiveness of measures to address underperformance.(d) Underperformers are rarely classified as such in the appraisal process (see paragraph 11).(e) Managers cited cases where they allocated the work of underperforming staff to others and facilitated their move somewhere else. If underperforming staff are in the wrong job or have developed unconstructive working relationships, moving jobs may help to resolve the problem, although it might just transfer it somewhere else.(f) Under the appraisal and promotion system in use until 2011, provided they were not formally classified as underperformers in the appraisal system [45], underperforming staff could still be promoted once they had accumulated the necessary points: this could have a demotivating effect on other staff [46].42. By blurring the line between good performance and poor performance, the former promotion system in use until 2011 reduced the incentive for staff to keep their skills up to date [47]. Furthermore, a specific provision in the Staff Regulations which requires training to be taken into account for purposes of promotion has had little impact [48]. Although in theory the promotion system can reward outstanding performance in any field, in practice it does not sufficiently recognise a commitment to learning and development by applying new skills, delivering training or moving jobs. In the audit survey only 17 % of respondents agreed that the commitment of staff to learning was recognised through the marks given to staff in the appraisal reports. Only 6 % of respondents agreed that the commitment of staff to learning was recognised through the speed of their promotions.THE COMMISSION RARELY EVALUATES THE EFFECTIVENESS OF DEVELOPMENT ACTIONS, THOUGH IT DOES MEASURE STAFF SATISFACTION WITH THEM43. Reliable information on the effectiveness of development actions is necessary in order to demonstrate their contribution to organisational objectives and to inform decisions on where to target learning and development resources. The audit therefore examined whether the Commission evaluated effectiveness at the four levels identified by the Kirkpatrick methodology [49] shown in Figure 11.FIGURE 11KIRKPATRICK’S FOUR LEVELS OF EVALUATING LEARNING+++++ TIFF +++++EVALUATIONS MEASURE STAFF SATISFACTION WITH DEVELOPMENT ACTIONS44. The Commission sends feedback questionnaires to course participants immediately following a training course. The response rate is in the order of 80 %. The replies for 2010 show that 78 % of staff are satisfied with general training, 76 % with IT training and 85 % with language training [50]. Course managers and contractors receive a summary of feedback results via Syslog. They may also receive direct feedback from participants or trainers. Course managers can then address the criticisms, for example, by changing the course content or the trainer. For language courses there is a specific procedure for dealing with complaints agreed between the Commission and the contractor.45. The Commission evaluates staff satisfaction with job moves by means of the staff opinion survey carried out every 2 years. The 2010 survey found that only 34 % of staff were satisfied with opportunities for mobility [51]. The evaluation of the involvement and motivation of older Commission staff highlighted the perceived difficulties of some staff over 50 in finding another post [52].46. In addition, there are more general indicators of overall staff satisfaction to which training and job moves contribute, for example, replies to staff satisfaction surveys, the level of staff turnover and the number of days’ sickness absence. However, these indicators of staff satisfaction are influenced by many factors (e.g. job content, working conditions, management style and remuneration) and the contribution of staff development measures cannot be isolated.THE COMMISSION DOES NOT ANALYSE IN DETAIL WHY STAFF HAVE NOT ATTENDED PLANNED TRAINING AND (EXCEPT FOR LANGUAGE AND CERTIFICATION TRAINING) GENERALLY DOES NOT TEST THE ACQUISITION OF NEW SKILLS47. Training maps of individual staff members showed the expected contribution of training towards ensuring that the Commission has the skills it needs to meet its objectives. In order to know whether learning needs have been met, the Commission needs information on:(a) whether staff attended courses planned in training maps;(b) whether staff have successfully acquired new skills.48. Staff did not attend 65 % of training courses planned in their training maps in 2010. This overstates the real level of non-fulfilment of training needs identified in training maps (see paragraph 30). However, the Commission has not analysed in detail the reasons for non-attendance in order to establish the real extent to which the needs identified in training maps have not been fulfilled.49. Tests to measure the acquisition of knowledge and skills are mainly limited to language courses and the certification exercise (by which officials from the Assistant function group can become members of the Administrator function group). The pass rates in 2010 were 94 % for language courses and 65 % for the certification exercise. However, 23 % of participants in language courses dropped out. Consequently, only 72 % of those registered for the training successfully acquired the intended new knowledge (see Figure 7). Other training does not generally test whether participants have acquired knowledge and skills although there have been interinstitutional discussions on evaluation which are considering introducing more tests to check knowledge acquired in IT and general training. Also the Development and Cooperation DG — EuropeAid has piloted pre- and post-testing in some e-learning courses [53].THERE ARE LIMITED ATTEMPTS TO EVALUATE THE UTILITY OF DEVELOPMENT ACTIONS IN THE WORKPLACE50. Perceptions of the impact of training in the workplace differ. DGs perceive that training organised locally is more focused on increasing efficiency and effectiveness at work whilst central courses focus on personal development and have less impact in the workplace. Conversely, the Human Resources and Security DG perceives that some local training, for example group learning activities, has limited impact in the workplace.51. The feedback questionnaires sent to participants immediately after the training course include a number of questions to assess whether participants will be able to apply the learning in the workplace [54]. In addition, there are some attempts to establish whether they have actually been able to apply the learning in practice. For example:(a) the 2010 staff opinion survey asked staff whether they could put the knowledge acquired from training into practice in their everyday work (70 % said they could) [55];(b) for the external management development programme, in addition to the evaluation directly after the course, there is a further evaluation 3 months later asking participants if and how they have applied their learning in the workplace;(c) new staff attending the Welcome training during their first 2 days at work do not receive the general Syslog evaluation form. Instead they are sent a questionnaire some months after the training course which includes questions on whether the different presentations or training themes were useful;(d) the project for improvement of evaluation forms proposes a series of questions to measure the impact of training 3 to 6 months after the course;(e) for local courses, the Environment DG is moving the emphasis of evaluations towards a reflection of knowledge acquired and how it will be used (see Box 4);(f) for certification training the European Administrative School sends questionnaires to certified staff 6 to 9 months after their appointment to an AD post, asking them how frequently they have used the skills learnt in the programme. 67 % say they frequently use the skills acquired and 74 % say that the training helped in their transition from Assistant to Administrator [56].52. The attempts to measure the utility of learning in the workplace are mainly based on the opinions of participants. Although managers have an important role to play in ensuring that training corresponds to organisational needs (see paragraph 17) they are rarely asked for their opinion on the utility of training undertaken by their staff. The audit survey found that only 13 % of respondents were asked for their opinion on the effectiveness of training courses attended by their staff and only 10 % of respondents thought their views were taken into account.BOX 4ENVIRONMENT DG INITIATIVES TO REFLECT ON KNOWLEDGE ACQUIRED AND HOW IT WILL BE USEDThe Environment DG has developed its own evaluation forms which focus on learning gained and how it will be used, rather than on participant satisfaction.Participants in the influencing skills course are asked to discuss the course with their Head of Unit before attending.At the end of the Environment DG’s strategic leadership course, participants are asked to make a presentation on one or more aspects of the course and to develop the ideas and how they would put them into practice. This encourages participants to reflect on how they will put their learning into practice.The Environment DG’s initiative to help internal trainers gain confidence and skills to coach and teach internally similarly finishes with individual presentations by participants and personal feedback from the trainer.To ensure participants are aware that they should learn something and put it into practice, all courses aim to include the following elements:(a) a discussion between participants and their Head of Unit on the goals for the course;(b) a session during the course on how to put the learning into practice;(c) follow-up, which can be in different formats: a follow-up discussion with the Head of Unit; an action plan; a follow-up course; or a presentation, as in the Strategic Leadership course.Source: ECA on the basis of interviews with the Environment DG.53. However, the opinion of participants on the value of training is not a reliable substitute for the views of managers on its utility in the workplace. For example, Figure 12 shows that 85 % of participants considered that language training met their objectives in 2010, whilst 64 % of managers replying to the audit survey considered that language training helped their staff to do their job better.54. Furthermore, evaluations of the effectiveness of learning actions rarely make reference to objective indicators. An example of the use of one such indicator is for certification training where the Commission monitors the number of candidates successfully passing the exams who are then appointed to Administrator (AD) posts. This indicates the utility of the training in the workplace (see Figure 13).55. Other possible indicators could measure the Commission’s use of follow-up actions to support staff in applying what they have learnt. Such follow-up actions could include post-course discussions with their manager, assessments by participants 6 months after the course and use of communities of practice, coaching and mentoring.56. Also, the Commission has little information on the views of staff or managers, or from objective indicators, concerning the impact of job moves in the workplace. In order to measure the effectiveness of job moves in the Communication DG, career guidance officers contact staff after rotation and 9 months later to question them about motivation and integration (see Box 1).57. In other DGs there were insufficient resources to carry out follow-up interviews to assess the effectiveness of job moves. Mobility aims to enable staff to develop skills in order to contribute to meeting organisational objectives. However, it can remove experienced staff and there is a risk that they are not replaced in a way which helps to achieve organisational goals. The audit survey of middle managers found that only 33 % of respondents considered that the procedures for replacing staff were effective.FIGURE 12EFFECTIVENESS PERCEIVED BY MANAGERS AND PARTICIPANTS+++++ TIFF +++++Source: Human Resources and Security DG (staff view) and audit survey (managers’ view).FIGURE 13SUCCESS RATES OF CERTIFICATION CANDIDATES+++++ TIFF +++++Source: Human Resources and Security DG.THE COMMISSION DOES NOT EVALUATE THE IMPACT OF DEVELOPMENT ACTIONS ON ORGANISATIONAL RESULTS58. The Commission does not evaluate the contribution of training and job moves to achieving organisational results. It therefore does not have the information necessary to identify and examine those cases where the investment in training has not resulted in improving performance or meeting objectives and to revisit its learning and development strategy accordingly. The Commission could identify indicators to measure the impact of various development actions, for example:(a) the number of transactions processed without errors or the number of calls to the Helpdesk to measure the contribution of training on financial or IT procedures;(b) the results achieved by an entity and the motivation of its staff to measure the impact of management training;(c) changes in the quality of documents to measure the contribution of courses in administrative drafting;(d) the quantity and quality of work of staff following job moves to measure the contribution of mobility.CONCLUSIONS AND RECOMMENDATIONSTHE COMMISSION SHOULD CLOSELY ALIGN STAFF DEVELOPMENT WITH ORGANISATIONAL NEEDS59. The Commission does not have sufficient consolidated information on staff skills to plan the development of its staff in line with the needs of the organisation. The annual appraisal in use until 2011 was not a reliable record of existing staff skills and did not reliably identify underperformance. Although it has been part of the Commission’s human resource management information system since 2007, the e-CV is of limited use because it still lacks an effective search tool and Commission managers have access to the e-CVs of only 5 % of staff. The systems for identifying the resources needed to meet future challenges focus on staff numbers rather than on skills needed. The new staff appraisal system introduced in 2012 and the further development of the e-CV aim to improve the information available on staff skills (see paragraphs 8 to 13).60. The top-down training strategy does not convincingly demonstrate how it will contribute to the achievement of the Commission’s policy objectives. There is no strategy for developing staff through job moves although there are some initiatives which aim to better align mobility with organisational objectives (see paragraphs 14 to 16).61. At an operational level, individual learning solutions were documented in training maps resulting from the annual dialogue between Heads of Unit and their staff. Generally, staff plan their own job moves in the context of an internal job market with insufficient information on staff skills and job vacancies. Some development actions, for example some language courses and job moves, do not address organisational needs (see paragraphs 17 to 21).RECOMMENDATION 1In order to align development actions more closely with the objectives of the organisation the Commission should:(a) determine the core skills most relevant to the organisation and assess the performance of staff against them on recruitment and regularly throughout their career;(b) identify skills necessary to meet future challenges based on a vision of what the organisation will look like in the medium term;(c) prepare a multiannual staff development strategy which convincingly demonstrates how training and mobility aim to contribute to achieving organisational goals by closing the gap between existing skills and those needed to meet current and future challenges.RECOMMENDATION 2In order to support this process the Commission should:(a) ensure that all its staff complete their e-CV;(b) make the e-CV a part of the annual discussion on learning and development needs between managers and their staff in order to keep it up to date;(c) take the necessary steps to ensure managers have appropriate access to e-CVs;(d) develop an effective search tool for the e-CV system;(e) make full use of the e-CV, combined with publication of vacant posts, in order to improve the identification of suitable candidates;(f) support staff in identifying training needs before matching these to specific training courses;(g) approve training for staff, including language training, when it is aligned with the interests of the service;(h) support staff in preparing longer-term career development plans, taking into account the wider interest of the respective DGs, the Commission and the EU institutions;(i) support staff in moving jobs through a more visible career guidance function which provides advice to staff on opportunities for development and how to make the best use of their skills.THE COMMISSION PROVIDES A WIDE RANGE OF OPPORTUNITIES FOR STAFF TO DEVELOP62. The Commission provides a wide range of opportunities for staff to develop through formal and informal learning and through moving jobs. In 2010 staff participated in an average of 6,9 days formal training. Informal learning is not recorded, but managers replying to the audit survey estimated that staff spent an average 4 days on informal learning. Each year 6 % of staff move to another DG and there are also considerable opportunities for staff to change jobs within the same DG (see paragraphs 22 to 28).THE COMMISSION SHOULD STRENGTHEN ITS LEARNING ENVIRONMENT TO CAPITALISE ON THE LEARNING OFFER63. The Commission has not created a sufficiently strong learning environment to enable it to capitalise on the extensive learning offer. Figure 14 illustrates how the wide range of opportunities for development is offset by varying levels of participation, limited support to apply new skills in the workplace and insufficient recognition of staff who develop their skills.FIGURE 14THE COMMISSION HAS NOT CREATED A SUFFICIENTLY STRONG LEARNING ENVIRONMENT TO CAPITALISE ON THE LEARNING OFFER+++++ TIFF +++++64. In 2010, staff attended only 35 % of the courses identified in their training maps. Although Commission staff participated in an average of 6,9 days’ training in 2010, 30 % of staff participated in less than 2 days’ training. Older and higher-grade staff participate in less training than younger staff on lower grades. Moreover, there are high levels of absences and dropout rates from language courses. The Commission does not closely monitor whether staff participate in planned training courses and it no longer monitors the extent of mobility within DGs. The Commission’s own managers and staff deliver some training, but not sufficient to indicate that staff development is valued enough by the organisation (see paragraphs 29 to 37).65. The Commission generally provides limited support to apply new skills in the workplace (see paragraphs 38 and 39).66. The promotion system in place until 2011 did not sufficiently distinguish between good performers who develop their skills and poor performers who do not. This lack of recognition contributes to the risk that staff are not motivated to participate in the wide range of development opportunities available. The new system introduced in 2012 aims to make a clearer distinction between good and poor performers and not to promote those whose performance is below average (see paragraphs 40 to 42).RECOMMENDATION 3The Commission should develop its systems in order to monitor more effectively:(a) whether staff have attended the compulsory courses they should have;(b) how many staff change jobs whilst remaining within the same DG.RECOMMENDATION 4The Commission should encourage all staff to participate in the wide range of development opportunities available by:(a) giving more recognition through the appraisal and promotion system to the commitment of staff to developing their skills and those of others, for example by delivering training;(b) addressing the issue of underperformance, including by providing as early as possible appropriate central support for line managers in addressing skills gaps of underperforming staff;(c) using the annual discussion of training needs to encourage older staff on higher grades to keep their skills up to date and share their knowledge and experience with others.RECOMMENDATION 5The Commission should test and certify the acquisition of new skills where practicable and support their application in the workplace by providing follow-up activities.THE COMMISSION SHOULD BETTER EVALUATE THE EFFECTIVENESS OF DEVELOPMENT ACTIONS67. The Commission measures staff satisfaction with development actions, notably through feedback questionnaires sent to participants after a training course. In addition, the Commission obtains more general indications of staff satisfaction with training and job moves through the staff satisfaction survey carried out every 2 years.68. However, the Commission does not analyse why staff do not attend courses identified in their training maps and the real extent of unfulfilled needs. Nor does it assess whether participants have acquired new skills (with the main exception of language training). The attempts to evaluate whether new skills are used in the workplace are based mainly on the opinions of participants. Managers are rarely asked for their opinion on the effectiveness of training courses attended by their staff or on the impact of job moves in the workplace. The Commission rarely has objective indicators to demonstrate the utility of development actions and their contribution to organisational results. Consequently the Commission lacks the information necessary to inform decisions on where to target learning and development resources (see paragraphs 43 to 58).RECOMMENDATION 6The Commission should better evaluate the effectiveness of development actions, including:(a) objective indicators of progress towards achieving organisational goals resulting from development actions;(b) an analysis of the extent of unfulfilled training needs;(c) tests of the acquisition of new skills;(d) measures of the use of follow-up actions to support staff in applying in the workplace what they have learnt;(e) the analysis of managers on the effectiveness of development actions;(f) opinions of staff on the utility of development actions in the workplace.This Report was adopted by Chamber IV, headed by Mr Louis GALEA, Member of the Court of Auditors, in Luxembourg at its meeting of 22 May 2012.For the Court of Auditors+++++ TIFF +++++Vítor Manuel da Silva CaldeiraPresident[1] Commission key figures card for 2011. The 32949 Commission staff on 1 June 2011 consisted of 22526 officials, 1402 temporary agents, 5871 contract agents, 2022 local agents, 54 special advisers and 1074 agents under national law. The 22526 officials consisted of 12032 administrators (AD staff), of which 1502 were managers, and 10494 assistants (AST staff).[2] Staff on fixed-term contracts include temporary agents, contract agents, local agents, special advisers and agents under national law.[3] The 2010 training budget amounted to 31,1 million euro of which 4,5 million euro was for other institutions. It was reduced by 5 % in 2011 to 29,7 million euro and by a further 11 % in 2012 to 26,3 million euro.[4] Commission Decision E(2002) 729 of 7 May 2002 on Staff Training.[5] COM(2000) 200 final of 5 April 2000.[6] Guidelines on Mobility, Communication to the Commission SEC(2002) 146 of 12 February 2002.[7] Council Regulation (EC, Euratom) No 723/2004 of 22 March 2004 amending the Staff Regulations of officials of the European Communities and the Conditions of Employment of other servants of the European Communities (OJ L 124, 27.4.2004, p. 1).[8] 2010 staff opinion survey among Commission staff.[9] Annex I, Article 43 of Regulation (EC, Euratom) No 723/2004.[10] The promotion points awarded can be influenced by how close a person is to promotion rather than being an objective assessment of performance. In order to maximise promotions within a DG, individuals may be awarded promotion points which they do not merit. In reply to the audit survey of middle managers, some respondents highlighted the difficulty of providing an objective appraisal because of the influence of the promotion exercise.[11] Underperformance is defined as performance level IV. Commission Decision C(2008) 3028 of 18 June 2008 on general provisions for implementing Article 45 of the Staff Regulation describes the performance levels.:- Performance level IA: consistently exceeded expectations with regard to efficiency, ability and conduct in the service;- Performance level IB: frequently exceeded expectations;- Performance level II: fully met expectations;- Performance level III: partly met expectations;- Performance level IV: did not meet any expectations.[12] The most recent progress report on mobility in 2007 showed that half of staff movements (1350 out of 2700) were reassignments under Article 7 of the Staff Regulation rather than published vacancies under Article 29.[13] European Commission Human Resources Report 2011, p. 73.[14] The target of 7,5 days formal learning consists of 3,5 days general training, 1,5 days IT training and 2,5 days language training.[15] "Implementation of the mobility policy" in the Commission Progress Report for the Year 2006, July 2007, and "Implementation of the mobility policy" in the Commission Progress Report for the Year 2005, July 2006.[16] European Commission Human Resources Report 2011, pp. 74– 75, and IAS Report on Human Resources Management 2006.[17] 2010 staff opinion survey, p. 31.[18] 2010 staff opinion survey, p. 25.[19] 2010 staff opinion survey, p. 164.[20] Paragraph 44 of Special Report No 1/2011 "Has the devolution of the Commission’s management of external assistance from its headquarters to its delegations led to improved aid delivery?" (http://eca.europa.eu).[21] The client satisfaction survey for October 2009 to March 2010 found that 88 % of SCOP customers were satisfied and 72 % thought the advice given in the interview clarified their situation.[22] The 2010 staff opinion survey found that 67 % of staff were not aware of the services offered by the SCOP.[23] The Learning and Development Unit is composed of 65 staff (55 in Brussels and 10 in Luxembourg). Its main task is to provide general and language training, largely contracted out. In 2008, it committed 14,3 million euro for organising 4000 courses with some 133000 participant days (about 25 % of the courses and 50 % of the participant days of all Commission training).[24] The European Administrative School formally came into existence in 2005. Its mission is to provide high-quality training and learning opportunities that meet the needs of all EU institutions and their staff.[25] Language training is provided for all institutions and so in addition to Commission staff there are participants from other institutions not included in Figure 4.[26] The Learning Centres in Brussels and Luxembourg offer a range of different learning aids which can be studied on site or borrowed.[27] For example, participants in the "External Management Development Programme" share the learning by giving a Management Matters Live lunchtime session, writing an article for the monthly Management Matters Live publication or producing a video for the Learning Channel. Participants in the Fellowships Programme present their research by videoconference from their university. On their return they draw up a report of their research.[28] The figures exclude mobility resulting from the reorganisation of services at the beginning of the mandate of the current Commission which affected some 1730 posts (Draft General Budget of the European Commission for the Financial Year 2012: Working Document Part II: Commission Human Resources, p. 67.)[29] Implementation of the mobility policy in the Commission Progress report for the year 2006, July 2007.[30] 2010 staff opinion survey, p. 25.[31] Those registered for courses who do not attend at least partly are recorded as absent. For language courses, absences are recorded for each lesson missed.[32] The evaluation found that a quarter of managers agreed to language training in languages other than the main languages used for work purposes.[33] Participants are automatically excluded from the language course they are attending if they reach a certain number of absences.[34] Twice-weekly courses represented 48 % of all language courses in 2010.[35] "Evaluation of the involvement and motivation of older Commission staff (above 50 years)", November 2008, Executive Summary, p. vii.[36] COM(2011) 890 final of 13 December 2011 —Proposal for a Regulation of the European Parliament and the Council amending the Staff Regulations of Officials and the Conditions of Employment of Other Servants of the European Union.[37] "Internal Control Standard 4: Staff evaluation and development" requires that "management ensure that every staff member attends at least the training courses of a compulsory nature as defined in the strategic frameworks of the Commission and the DG".[38] Of these, 26 % were Heads of Unit for less than a year and so may not yet have completed the training. A further 31 % had over 4 years experience as Head of Unit and so may not be considered "new". The remaining 43 % were Heads of Unit for between 1 and 3 years and so they should have attended the full compulsory management training programme.[39] "Internal Control Standard 3: Staff Allocation and Mobility" requires mobility to be monitored in order to ensure that the right person is in the right job at the right time.[40] The 2000 White Paper on reforming the Commission recommended more training by the Commission’s own staff as a means of promoting a learning culture.[41] 2010 staff opinion survey, p. 25.[42] 2010 staff opinion survey, p. 25.[43] Reforming the Commission: A White Paper — Part II: Action Plan, p. 28.[44] Annex I of Regulation (EC, Euratom) No 723/2004, Annex IB and Commission Decision C(2008) 3028 of 18 June 2008 on general provisions for implementing Article 45 of the Staff Regulations, p. 7.[45] In the appraisal and promotion system in use until 2011, staff allocated to performance level IV were classified as underperformers.[46] The new appraisal system from 2012 aims to ensure that staff performing below average are not promoted. Promotion points are not cumulative in the new system and promotion is no longer awarded on passing a threshold of points.[47] The level of staff dissatisfaction with the appraisal and promotion system is indicated by the number of appeals against the promotion points. In 2010, there were some 3400 appeals representing 16 % of 21700 reports.[48] Article 24a of the Staff Regulations states: "The Communities shall facilitate such further training and instruction for officials as is compatible with the proper functioning of the service and is in accordance with its own interests. Such training and instruction shall be taken into account for purposes of promotion in their careers." Furthermore, the common appraisal standards require a willingness to develop new knowledge and skills through training to be taken into account as an aspect of conduct.[49] Donald Kirkpatrick first proposed his theory of evaluation in 1959. More recently see Donald Kirkpatrick and James Kirkpatrick, "Evaluating training programmes — The four levels", 2006. The four-level model is widely used across training communities. The Court chose the Kirkpatrick model to structure the assessment.[50] The staff opinion survey also measures staff satisfaction with training and found that 74 % of staff are satisfied with central courses and 71 % with local courses (2010 staff opinion survey, p. 34).[51] 2010 staff opinion survey, p. 45.[52] "Evaluation of the involvement and motivation of older Commission staff (above 50 years)", November 2008, Executive Summary, p. vii.[53] The e-learning application Blackboard has an option to oblige the participant to pass a test before being allowed access to the next element of the training module.[54] The first statement which participants are asked to assess is "This course was relevant to my work". There is a free text box asking "What help/support do you need to implement your learning in the workplace/on the job?" The question "How useful was this training action in terms of meeting your learning objectives?" could also reflect the usefulness of the course in the workplace.[55] The 2008 staff opinion survey asked staff to assess the utility in the workplace of different types of training and found 73 % for general training, 78 % for IT training and 76 % for language training.[56] Annual activity report of the European Administrative School 2011.--------------------------------------------------