EN Official Journal of the European Union C 312/3

SPECIAL REPORT No 6/2007

on the effectiveness of technical assistance in the context of capacity development together with the Commission's replies

(pursuant to Article 248(4), second subparagraph, EC)

(2007/C 312/02)

TABLE OF CONTENTS

GLOSSARY

I-VIII EXECUTIVE SUMMARY

1-6 INTRODUCTION

7-8 AUDIT SCOPE AND APPROACH

9-43 OBSERVATIONS

9-12 Does the Commission have a sound strategy for institutional capacity development, including the use of technical assistance?

10-11 Inadequate institutional analysis in Country Strategy Papers

12 No guidance on technical assistance

13-17 Are technical assistance activities within capacity development projects well designed?

14-16 Project choice relevant but design often inadequate

17 Local ownership essential but not systematically addressed by the Commission

18-34 Are technical assistance activities implemented efficiently?

19-21 Long preparation phase often leads to outdated project design

22-26 Problems during procurement and project start-up reduce time available for implementation

27 Experts often replaced after award of contract

28 Inappropriate selection criteria for technical assistance

29-30 Limited choice between procurement procedures

31 Inefficient use of technical assistance

32 Donor coordination varies between countries

33-34 Implementation arrangements not yet favorable to local ownership

35-39 Are technical assistance activities and performance adequately monitored and evaluated?

40-43 Are technical assistance activities effective in terms of capacity development?

40-42 Technical assistance contribution satisfactory but success of projects not guaranteed

43 Sustainability uncertain

44-48 CONCLUSIONS AND RECOMMENDATIONS

Annex I — List of the projects and programmes examined

Annex II — Effectiveness and prospects of sustainability of projects and programmes examined

The Commission's replies

GLOSSARYACP African, Caribbean and Pacific Countries.Financing agreement Document signed between the Commission and the partner country, describing a particular project or programme to be funded. It represents the formal commitment of the European Union and the partner country to finance the measures described.Financing decision Decision taken by the Commission, on the basis of a financing proposal, to commit funds to a particular project or programme.Financing proposal Draft document prepared by the Commission services, submitted to the relevant Financing Committee of Member States' representatives for an opinion, and having received a favourable opinion, submitted to the Commission for decision. The document describes the background, nature, scope and objectives, implementation measures and the amount of funding foreseen for a particular project or programme.MDGs Millennium Development Goals consisting of eight goals to be achieved by 2015 — which range from halving extreme poverty to halting the spread of HIV/AIDS and providing universal primary education — were agreed upon by all countries and the world's leading development institutions at the UN Millennium Summit of September 2000.OECD/DAC Development Assistance Committee of the Organisation for Economic Cooperation and Development.Paris Declaration on Aid Effectiveness Statement adopted at a High Level Forum in Paris, 28 February — 2 March 2005 by a large number of countries and organisations, containing a number of commitments concerning ownership, harmonisation, alignment, results and mutual accountability, all with the aim of increasing aid effectiveness.

EXECUTIVE SUMMARY

I. Technical assistance is a resource used in many development projects and programmes of different types and can be defined as ‘experts contracted for the transfer of know-how and skills and the creation and strengthening of institutions’. In recent years, it has been increasingly used to support capacity development of central and local government services in beneficiary countries, in areas such as public finance management, health and education, among others. This change of focus towards the functioning of government services has been the result of a growing awareness among donors, including the European Union, that a lack of institutional capacity is a fundamental obstacle to sustainable development (see paragraphs 1 to 6).

II. The purpose of the Court's audit was to determine the extent to which the Commission ensures that technical assistance contributes to institutional capacity development in beneficiary countries. This was done by assessing the Commission's approach towards capacity development and the use of technical assistance and its management of the design and implementation of technical assistance activities within capacity development projects (see paragraphs 7 to 8).

III. Although institutional capacity building has been recognised in the European Community's development policy as a key element in the fight against poverty, the Commission has not included in its Country Strategy Papers a comprehensive assessment of institutional capacity weaknesses nor identified priority needs for capacity development within beneficiary governments' administrations and institutions. Neither does the Commission have guidelines on the role technical assistance should play in this context (see paragraphs 9 to 12).

IV. Project choice is mostly relevant but design is often inadequate due to over-ambitious objectives, tight timeframes and insufficient assessment of the existing weak institutional environment and level of local ownership of the projects (see paragraphs 13 to 17).

V. Project implementation is often hindered by a number of problems, including a long preparation and start-up phase, which makes the project design outdated by the time implementation starts and significantly reduces the time available for implementation. Recruitment of technical assistance often adds to the delays as the experts selected are frequently not available once contracts have been awarded. Current procurement procedures for contracting technical assistance are not conducive to the participation of public institutions, while in certain areas better qualified experts may be found within such organisations. Donors still provide technical assistance support mainly through separate, uncoordinated projects, while implementation arrangements do not encourage local ownership (see paragraphs 18 to 34).

VI. Project reporting, monitoring and evaluation are in most cases adequate. However, there is no management information system in place to assess and record the performance of technical assistance companies or experts, and provide information when decisions on the procurement of technical assistance are taken (see paragraphs 35 to 39).

VII. Project results vary considerably but the Court concludes that, in the light of the often difficult environment in which they take place, project results and the contribution made by the technical assistance used can be considered as reasonable. The quality of the technical assistance provided was generally found to be satisfactory, but other important factors have reduced the effectiveness of the support provided. Although the Commission does not have control over some of these factors, it should nevertheless better anticipate them (see paragraphs 40 to 43).

VIII. On the basis of these observations, the Court makes detailed recommendations which could help the Commission to establish a more focussed and structured approach to the use of technical assistance, in order to provide more effective support for capacity development (see paragraphs 44 to 48).

INTRODUCTION

1. Development cooperation is sub-divided into two types of cooperation or assistance: financial assistance and technical assistance, or in other words ‘money’ and ‘know-how’ or ‘ideas’. This report covers the Court's audit of technical assistance, which the Commission defines as ‘Specialists, consultants, trainers, advisers etc. contracted for the transfer of know-how and skills and the creation and strengthening of institutions’ (1).

2. Technical assistance is a resource used in many development projects and programmes of different types, and its importance or effectiveness is judged through its contribution to the outputs and results of the related projects and programmes. Specific areas or tasks in the context of development cooperation, in which technical assistance has played or plays a role include: (a) the design and supervision of works, such as road construction; (b) project management, whereby experts carry direct responsibility for implementation of activities; (c) advisory services, in the past often for areas such as agricultural or rural development. More recently increasing use has been made of them to improve central government services, including public finance management, health, education, and the judiciary and also to help other public institutions, local authorities or non-state actors.

3. The change of focus towards the functioning of central and local government services has been the result of a growing awareness among donors, including the European Union, that a lack of institutional capacity is a fundamental obstacle to sustainable development. Capacity development has therefore become one of the focal areas in the European Community's Development Policy of 2000 as well as in the new policy formulated in 2005 (2).

4. The importance of capacity development has also been stressed in a recent OECD/DAC document (3), which states: ‘Adequate country capacity is one of the critical missing factors in current efforts to meet the Millennium Development Goals. Development efforts in many of the poorest countries will fail, even if they are supported with substantially increased funding, if the development of sustainable capacity is not given greater and more careful attention’ (4).

5. Further evidence of the importance of capacity development can be found in the Paris Declaration on Aid Effectiveness of March 2005 (5), which states that partner countries' capacity to plan, manage, implement and account for results of policies and programmes is critical for achieving development objectives. It emphasises the need for donors to base their support on partner countries' national development strategies, institutions and procedures as much as possible and to provide support to address existing weaknesses in partner countries' institutional capacities. Although capacity development can be supported through a variety of other measures — such as training, scholarships, computer equipment and software, documentation and even physical infrastructure — technical assistance is the main instrument used in this context.

6. The Commission's accounting and management information systems did not allow the Court to obtain precise and reliable information on the amounts spent on technical assistance, either by the European Development Funds or through the General Budget. The Court has estimated the total value of commitments for technical assistance contracts, which were signed during the period 2000-2005 and were still open at the beginning of 2006, at about 2 200 million euro (6). The share of technical assistance absorbed by capacity development activities could not be estimated.

AUDIT SCOPE AND APPROACH

7. The purpose of the Court's audit was to determine the extent to which the Commission ensures that Commission-financed (7) technical assistance contributes to institutional capacity development in beneficiary countries. More specifically, the audit set out to answer the following questions: (a) Does the Commission have a sound strategy for institutional capacity development, including the use of technical assistance? (b) Are technical assistance activities within capacity development projects well designed? (c) Are technical assistance activities implemented efficiently? (d) Are technical assistance activities and performance adequately monitored and evaluated? (e) Are technical assistance activities effective in terms of capacity development?

8. The Court audited the Commission's approach and activities, at policy and project level, in relation to 12 beneficiary countries, selected from four different geographical areas receiving development assistance: Africa, Asia, Latin America and the Mediterranean region (8). Five of these countries were visited on the spot, while the other seven countries were covered through a documentary review and questionnaires sent to the Delegations and the beneficiary authorities of the respective countries. From an inventory of capacity development projects, 32 projects were selected which involved 33 technical assistance contracts being implemented during the period 2000-2006. The total amount committed to the projects audited was 400 million euro, with some 144 million euro devoted to technical assistance spending. The complete list of the 32 projects and programmes examined is included in Annex I.

OBSERVATIONS

Does the Commission have a sound strategy for institutional capacity development, including the use of technical assistance?

9. The Commission translates the principles of the overall European Community's development policy into specific country strategies, usually covering five-year periods, by preparing Country Strategy Papers and more detailed National Indicative Programmes for all individual beneficiary countries. For the regions covered in the audit, the Court examined, on the basis of these documents (9), whether the Commission makes a systematic assessment of recipient countries' institutional capacity strengths and weaknesses, and whether these countries' overall capacity development needs are identified and prioritised, and translated into technical assistance needs.

Inadequate institutional analysis in Country Strategy Papers

10. The examination of the Commission's relevant policy documents, for the countries covered in the audit, as well as the individual countries' own national development plans and/or so-called Poverty Reduction Strategy Papers, revealed a growing recognition of the importance of capacity development. However, the Commission did not present a comprehensive assessment at country level of institutional weaknesses, nor identify priority needs for capacity development within beneficiary governments' administrations and institutions. As a result, the selection of focal sectors for EC support was made without such information being available, with the risk that potentially important sectors or programmes in need of donor support may not have been identified.

11. In 2005, the Commission prepared an internal Concept Paper on Institutional Assessment and Capacity Development, describing the meaning and importance of capacity development. The paper provides guidance on how to assess institutional capacity and support capacity development processes. However, these guidelines are not applicable to the Country Strategy Papers. At the lower level of sector plans greater attention is paid to capacity development issues — albeit in very general terms. These plans are prepared after the focal sectors to be supported in a specific beneficiary country have been decided.

No guidance on technical assistance

12. Although technical assistance is a long-established and frequently used instrument in Commission-financed capacity development projects and programmes, the Commission has not defined the role of technical assistance and its contribution to capacity development. Neither are there specific guidelines for use by the Delegations when preparing projects, describing when and how to use the instrument in relation to capacity development. In addition, the ‘Twinning’ instrument, which has been successfully used to provide technical assistance to pre-accession countries (and more recently also to ‘Neighbourhood Policy’ countries), has so far not been available to the ACP-, Asian, or Latin-American countries. While the concept paper mentioned in paragraph 11 contains some considerations on technical assistance issues, it provides no guidance on the use of the various kinds of technical assistance in different situations, sectors or countries.

Are technical assistance activities within capacity development projects well designed?

13. For the 32 projects audited, the Court assessed to what extent the choice of these projects was consistent with the Commission's, and the partner countries', national strategies, whether the project designs set clear objectives and related performance indicators and adequately justified the proposed type and amount of technical assistance. The Court further examined whether the beneficiary authorities had actively participated in project preparation, and provided clear indication of ownership of the project.

Project choice relevant but design often inadequate

14. All the projects audited were consistent with the Commission's own Country Strategy Papers and with the sector priorities established therein, and they were generally in line with the overall development priorities of the beneficiary countries concerned.

15. About three-quarters of the projects examined showed one or more of the following design weaknesses: — project designs were overly complex, involving multiple components and stakeholders; — they set over-ambitious objectives and relatively short implementation periods; — project objectives were described in general terms, and when performance indicators were established, these were mostly of a very general, qualitative nature, compromising their ability to adequately assess subsequent performance; — the existing institutional weaknesses in the project environment were insufficiently assessed. These factors, in turn, impede the proper formulation and implementation of the technical assistance tasks.

16. Whereas all projects examined included technical assistance as a major input, in about half of the cases the project documents proposed certain types (international or local) and amounts (long-term or short-term assignments) of technical assistance, without giving clear reasons for the proposed combination of technical assistance resources. Furthermore, there was also a lack of clarity regarding the expected role of the experts because the advisory tasks relating to the project's capacity development objectives usually had to be combined with a ‘gap-filling’ role to support the institution concerned in its regular tasks. Box 1Example of project design weaknessesSupport Programme to the Health Sector — AngolaThe project objectives included strengthening institutional capacities at the central and provincial levels, improving basic health services, improving the availability of essential medicines and establishing a system for the prevention of HIV. Although the evaluation of a previous project had mentioned the difficulty found in ensuring an active role by the Ministry of Health, the feasibility study did not carry out an assessment of the local capacities to ensure the effective implementation of this ambitious project. Only when the project had started, was it noticed that some of the Angolan partners did not have adequate capacity to implement the project. The inception report prepared by the experts concluded that the project was too ambitious, but no changes were made until 18 months before the end of the project, when the objectives and activities were significantly reduced.

Local ownership essential but not systematically addressed by the Commission

17. Ownership of the planned institutional development, on the part of the beneficiary institution and the people working for it is a key factor, determining the environment in which the technical assistance can operate and thereby contributing to the successful implementation of the capacity development plans. Being conscious of this, the Commission generally involved local beneficiary institutions in the design phase of its projects. However, it did not sufficiently assess the risk that a beneficiary government might not fulfil its commitments. The fact that a beneficiary government signs a financing agreement with a donor does not automatically imply that the assumptions, formulated in most project agreements about the government's commitment to reform, will be fulfilled. Nor does it mean that the project will be supported by all the people within the institution, or affected in some way by the project, as demonstrated by a lack of interest displayed in several cases by staff of the institutions receiving aid. Box 2Example of inadequate local ownershipHealth project — BeninThe project objective was to improve the quality and accessibility of health care services. However, the experts had to deal with many problems, such as the malfunctioning of the human resources management in the health sector, which was characterised by frequent transfers and arbitrary appointments, negatively influencing the motivation of staff. The weak involvement of the Ministry in the project caused many delays, e.g. it refused to second staff to act as local counterparts to the project management unit. Other staff supposed to act as contact points between the different directorates of the Ministry and the project lacked motivation because they considered that they should be paid for this activity which they regarded as ‘extra’ work. The Commission rightly opposes such payments.

Are technical assistance activities implemented efficiently?

18. In order to assess whether the projects were implemented efficiently, the Court examined whether the project started on time and was implemented as planned, whether the procurement and mobilisation of technical assistance proceeded without undue delays, whether the level of coordination with other related projects or activities was adequate and whether the project implementation arrangements encourage ownership by the beneficiary institutions.

Long preparation phase often leads to outdated project design

19. The preparation phase of the projects examined generally took between one and two years. It involved a number of required steps, including one or more preparatory consultants' missions (for identification and formulation), two rounds of advice and comments by the Quality Support Group (10) in the Commission's Headquarters, and submission of the financing proposal to the relevant Member States' advisory committee.

20. After the proposal has been approved by the Commission, a procurement procedure follows (see paragraphs 22 to 26), in order to contract the required experts. By the time they arrive on the spot to start the project — which is in most cases at least two to three years after the design phase started — local conditions have often changed: there have been new political or other developments (elections, new appointments, new national policies), or other donors have started projects. As a result, the experts often find that the original project assumptions are no longer valid or some of the activities planned are no longer relevant. This means that in such cases the experts first have to review the beneficiary's current capacity development needs and revise the project design accordingly before they can start with the regular task of preparing the project's detailed work plan.

21. However, such revisions rarely occur at the start of a project, because of the administrative work and time involved in getting approval to change a financing agreement. The Commission usually starts the project as designed and waits for the Mid-Term Review (normally carried out after some two years of implementation), before a decision is made about changes. By then, the time remaining to implement any necessary changes is very limited and overall effectiveness is compromised. Box 3Example of a long preparation period causing design to be outdatedEuropean Technical Assistance Programme for Vietnam (ETV2)The project was prepared with the help of two separate missions, in October 2000 and July 2002. The financing agreement was signed in April 2003, but for various reasons the technical assistance contract was not signed until August 2005. The project started in January 2006, mainly as a result of the original team leader no longer being available and having to be replaced. By that time 3,5 years had passed since the project was formulated and, given the rapid pace of change in Vietnam, many activities which had been defined were no longer relevant. A major task for the experts, therefore, during the inception phase was to redesign the project, with the result that less than half of the planned time was available for the implementation of the project activities.

Problems during procurement and project start-up reduce time available for implementation

22. Technical assistance required for the implementation of projects is normally procured through an international restricted tender procedure (11). This procedure requires some six to eight months to complete. As the procurement procedure generally starts only after the financing agreement has been signed, it takes at least ten months before the experts are in place (eight months for procurement and two months for mobilisation of the experts). This period is taken into account when the formal implementation period of the project is determined in the financing agreement.

23. However, the Court's audit of the 32 projects has shown that many problems have arisen during tender procedures, or were linked to the start of projects, which delayed the launching of project activities and reduced the available implementation period. Typical problems include: (a) tenders which fail for lack of sufficient numbers of valid offers and therefore have to be relaunched; (b) delays in setting up the tender evaluation committee; (c) late arrival of experts; (d) late appointment of required local counterpart staff; (e) difficulties with setting up project support structures, such as office facilities, project bank accounts, project manuals and procedures; (f) the preparation of the overall work plan proving more difficult than expected; (g) the need to revise the project design during the inception period. The overall result was that the actual implementation period for practically all the projects examined had become, or was likely to become, too short to have all the planned activities implemented and the expected results achieved.

24. Apart from the overall time limit set for project implementation, there is also a contracting deadline to respect (the so-called D + 3 rule), introduced in the 2003 Financial Regulations (12). Therefore, all the delays arising after the date of the Financing Decision, e.g. during the tender for the main technical assistance contract or during the start-up phase, effectively limit the period during which further implementation-related contracts for works, supplies or other services can be concluded. Several projects designed since this rule came into force have already been, or are likely to be, affected. As from 1 May 2007 a modification to the Financial Regulation applicable to the general budget (13) has come into effect, which eases the constraint of the D + 3 rule to some extent, with the 3 year period starting from the date of the financing agreement. However, any delay occurring after this date will still have the same effect of reducing the implementation period as well as the period available for contracting.

25. It is technically possible to start the procurement process earlier — i.e. before the financing agreement is signed, or even before the financing decision is taken — in which case the procurement notice must be published with a ‘suspension clause’ (14). This would help to reduce the time period which normally passes between signing the financing agreement and the start of project activities. The use of such a clause is actually provided for, at least for the ACP countries, in one of the Annexes to the ACP-EU Partnership Agreement (the Cotonou-Agreement of 23 June 2000), ‘in order to ensure early project start-up’ (15).

26. According to the Commission's internal guidelines on procurement this possibility should only be used in exceptional cases (without indicating what these could be). The Commission considers that there is a risk that tenders may have to be cancelled and that, if this occurs too frequently, the interest of potential service providers in future Commission-financed projects may be reduced. The suspension clause is therefore rarely used. However, the risk that a project is completely cancelled after the preparation has been completed and the financing decision has been taken is, in practice, limited. The Commission should clarify in its guidelines under what circumstances and conditions the suspension clause may be used. Box 4Example of reduced implementation period due to delaysDevelopment of Human Resources for Health — UgandaThe technical assistance contract was signed in September 2003, some 18 months after the financing agreement was signed, and the first experts arrived 2 months after that. A combination of factors had caused the tender procedure to have taken so long, including the transfer of the health training schools from the Ministry of Health to the Ministry of Education, and internal changes at the time in the Delegation (staff leaving, devolution, transfer of portfolios). Although the financing agreement had an implementation period of five years, and the technical assistance contract a performance period of four years, the 20 months' delay at the start had left only three years and three months for implementation.

Experts often replaced after award of contract

27. In one-third of the contracts examined, almost immediately after they had won the contract for supplying technical assistants, the consultancy companies informed the Commission that key experts were no longer available. In some cases, this was due to the late completion of the tender evaluation and award of the technical assistance contract, which meant that experts were no longer committed as to their availability (16). In other cases, the nonavailability occurred within the validity period. The most common reason given was health problems. There is a risk that companies deliberately propose experts with good CVs in order to win the contract, while knowing that the expert will not be available to take up the assignment. In all cases, finding replacements delays the start up of the project. Box 5Example of replacement of expert after award of contractFollowing an efficient tender procedure, a consortium of several organisations won the technical assistance contract, principally on the strength of its proposed team leader. However, this team leader never took up his duties as he was withdrawn shortly after the contract was signed because it was discovered that he was susceptible to a disease which occurs in the country in question. This is something which should have been known earlier. His replacement arrived at the project almost three months later.

Inappropriate selection criteria for technical assistance

28. The regular withdrawal of key experts raises the question as to whether the criteria used for evaluation of proposals for technical assistance are actually appropriate (17). The most important single criterion used in the technical evaluation of tenders for service contracts is the evaluation of the curriculum vitae of the long term experts proposed, in particular that of the team leader. Only in some cases are interviews with experts organised. Other potential indicators for technical quality are not taken into account, such as the overall capacity of the company, and/or the Commission's actual experience of the company or the proposed experts in previous Commission-financed contracts.

Limited choice between procurement procedures

29. When it comes to the recruitment of experts for the implementation of capacity development projects, there is only a limited choice between different sources from which to recruit experts. There are basically two ways to recruit experts, i.e. through the Framework Contract system (18), or through a restricted international tender for services (see paragraph 22).

30. In both cases the contract under which experts are to be hired is signed with a consulting firm. In certain areas of expertise such as public finance management or public administration, better qualified experts may be found within other types of organisation, such as the national administrations of Member States, e.g. in the case of public finance management, within national Treasuries, Ministries of Finance or Central Banks, similar to what happens in ‘Twinning’ projects in pre-accession countries. In cases where very specific local expertise is required, neither of the two procedures is in practice conducive to tapping into the local or regional market for technical expertise.

Inefficient use of technical assistance

31. International experts were not always used efficiently, because of the different roles assigned to them. Although they are principally selected on the basis of their technical expertise, the Commission also expects experts to play a role in project management and financial control. Under the current Financial Regulation the international technical assistance team leader is no longer the Co-Director for the project with joint signatory powers, as was the practice under the previous Financial Regulation. However, in practice, the team leader not only verifies but also prepares the project work plans and budgets, the reports, tender dossiers, draft contracts and payment proposals from the project account. In some cases these administrative duties were taking up a disproportionate amount of time, thus reducing the time the expert(s) can spend on technical capacity development activities. It also leads to a certain ambiguity concerning the experts' accountability, i.e. to the beneficiary institution or the Delegation, which complicates their tasks and can compromise the climate of confidence between the expert(s) and the local counterparts.

Donor coordination varies between countries

32. The level of coordination of capacity development programmes varies considerably between the countries concerned. This ranges from limited exchanges of information to more elaborate systems of sectoral and/or thematic working groups which bring together all active partners and display a high level of coordination, with the Commission usually being one of the most active donors in this respect. Where a programme of general budget aid or a sector policy support programme had been established, coordination was better organised and more intensive. However, even then the technical assistance was often still provided through separate projects, whereby each donor applied its own procedures.

Implementation arrangements not yet favorable to local ownership

33. Project Implementation Units (or Project Support Units) are still set up in most countries as temporary structures, for the duration of a project and in parallel to the existing national administration, although in several cases they are already more integrated within the regular institutional structure of the beneficiary administration. In none of the countries concerned was the procurement of, and payment for, Commission-financed technical assistance being managed through national systems and procedures, as the Commission considered these to be unreliable and lacking transparency. However, several of these countries (i.e. Benin, Chad, Morocco, Sierra Leone, Uganda and Vietnam) are recipients of Commission-financed budget aid, whereby expenditure of the funds is, by definition, subject to the national financial procedures. This illustrates an incoherence in the Commission's approach regarding the use of the beneficiary countries' public financial management and procurement systems under its different aid instruments.

34. In the context of the Paris Declaration (see paragraph 5) donors have undertaken a number of commitments to align their support with partner countries' development strategies, institutions and procedures. Several of these commitments are directly related to the issue of capacity development. In this context the European Union formulated in three areas even stricter targets for itself, i.e. to provide all capacity-building assistance through coordinated programmes with an increasing use of multi-donor arrangements; to channel 50 % of government to government assistance through country systems; and to avoid the establishment of any new Project Implementation Units. Given the design of most current capacity development projects, the Commission's task to meet its commitments under the Paris Declaration is a challenging one.

Are technical assistance activities and performance adequately monitored and evaluated?

35. The quality and timeliness of reporting by the experts was generally satisfactory and projects are in most cases monitored actively by the Delegations' staff. In addition, there are two main systems in place for the external monitoring and evaluation of projects: (a) the Results-Oriented Monitoring system (ROM), carried out by consultants under the authority of the Commission; (b) Mid-Term Reviews of projects, which are provided for in the project's financing agreement and carried out by consultants selected and/or approved by the Commission and the partner country.

36. Both systems are used regularly. In some cases projects had been visited by the ROM consultants a short time after the start of activities, so that only the project design could be assessed and not the efficiency or effectiveness of implementation or the results.

37. The ROM system represents good practice, particularly as it involves regular visits to projects during their implementation. However, the Court found in a number of cases that the scores given to projects were unrealistic or inappropriate or that the scores given by consecutive ROM visits to the same project were not consistent. This may be influenced by the fact that these visits are necessarily very brief (in view of the large number of projects) and because of regular changes of monitoring staff.

38. Mid-Term Reviews are carried out for most projects and are generally seen as a very useful instrument providing a balanced, outside view on a project. Their recommendations are often used by the Delegations when deciding on changing, refocusing or extending projects. However, with the current trend of reducing implementation periods, it becomes more difficult to schedule a Mid-Term Review in such a way that a project has had sufficient time to progress, yet also leave sufficient time to implement and benefit from the recommendations.

39. Systematic assessment of technical assistance performance, by companies and experts, does not take place. Any assessment carried out, either by a Delegation or by a local beneficiary or the local contracting authority (in the case of the EDF) is done on an ad-hoc and informal basis only. The Commission does not have any system to allow even this limited information to be shared. The lack of an adequate management information system in relation to technical assistance means that valuable experience from the past in working with certain firms or experts cannot be exploited when new decisions on the procurement of technical assistance are taken.

Are technical assistance activities effective in terms of capacity development?

Technical assistance contribution satisfactory but success of projects not guaranteed

40. Overall, the actual or likely results for the 32 projects audited vary considerably (see Annex II). Ten projects were still at too early a stage of implementation to allow any reasonable anticipation of their likely results. As far as the remainder of the projects is concerned: — one-third of the projects have been, or are likely to be, successful in reaching their objectives (see Box 6); — one-third of the projects have experienced some problems and have achieved, or are likely to achieve, only part of their objectives (see Box 7); — one-third of the projects have experienced significant problems and have achieved, or are likely to achieve, very few of the planned objectives (see Box 8). Box 6Example of a successful projectSupport to decentralisation in BeninThe project's objective was to establish at municipal level the administrative capacity to provide an improved level of public services to the population.An innovative approach was followed whereby a single international expert, responsible for leading and coordinating activities at central level, was combined with a large team of local experts, who had the specific local knowledge and provided support at the local level. This required an initial training of the local teams by the international expert in order to ensure a common approach and develop standardised tools. The project design included clear objectives and performance indicators, although they were not initially quantified. Mobilisation of the expert was within six months of signing the financing agreement. Upon the start of the project the expert refined the logical framework and established a database for monitoring and evaluation. Despite a difficult environment, due to the lack of clear political support for the decentralisation process, the project has largely achieved its objectives. The Commission, taking a long-term view towards development in this area, has prepared a second phase for the project, to support the implementation of the municipal development plans and to increase the capacity of the supervisory institutions.

41. As technical assistance is the main input used in capacity development projects, its quality obviously has a significant influence on the project results. Performance by experts was generally considered satisfactory by the beneficiary institutions and the Delegations. Although in about half of the projects audited there have been cases of replacement of experts, this was usually due to personality differences but sometimes to underperformance. There have been only a few cases where a whole team of experts did not perform well and had to be replaced. However, good quality technical assistance does not necessarily ensure the success of a project, as other factors have a decisive influence on the outcome.

42. For the large majority of projects, an insufficient implementation period has been a major contributory factor to projects not being able to produce their results within the time allocated by the financing agreement. The duration of many of these projects has been, or will be, extended, which should lead to further results being obtained. Extensions of the time period alone are not sufficient to guarantee a successful completion of projects. Other important factors influencing the effectiveness of the projects are the institutional environment in the beneficiary countries and the degree of ownership displayed by the beneficiary institutions during the preparation and implementation of the projects. For the countries visited by the Court, these factors have been generally positive in Vietnam and Uganda, and to a lesser extent in Algeria and Benin, but this was not the case in Angola. Such variations in local institutional capacity and ownership go a long way towards explaining the differences in the results obtained or expected from the projects audited by the Court in the respective countries. Some of these factors are outside the direct control of the Commission, which nevertheless should better anticipate them. Box 7Example of a project with partial results and uncertain sustainabilityTraining of primary school teachers — AngolaPoor project preparation and delays led to half of the planned activities being dropped. As a result, the project did not achieve its full objectives, as only 4 training centres are operational, against a target of 16, and 11 000 teachers were trained compared with an original target of 14 250. The government of Angola has not respected its commitments to finance these provincial training centres. Without classrooms, water or electricity, their infrastructure was totally inadequate and courses were taking place in the primary schools, obliging the children to have their courses in the playground. Teachers were generally happy with the training provided, but they had not yet received the formal certification by the Ministry of Education and therefore did not yet have the related professional upgrading or salary increases. A statistical data system was installed but not working. Other equipment provided by the project was not working due to lack of maintenance and spare parts. Box 8Example of a project with unsatisfactory resultsHealth Sector Support Programme — Sierra LeoneProject design took place under difficult circumstances in the late nineties, when the security situation was still very unstable. The project objectives were overly ambitious, in view of this difficult situation and the complexity of the decentralisation process. Delays in recruitment and inadequate performance of the experts, as well as the lack of ownership by the Ministry of Health, which did not provide the required local counterpart staff, have seriously affected the activities of the project. As a result, the project has fallen far short of its objectives. At the end of 2006, the Commission's services were preparing a rider to the financing agreement to decommit 13 million euro (from the original contribution of 28 million euro) and extend the project by two and a half years with a view to try to complete parts of the two more successful components, i.e. the primary health care at decentralised level and the pharmaceutical component.

Sustainability uncertain

43. The prospects for the project results to be sustainable, i.e. that any improvement in capacity will be permanent, are in many cases uncertain (see Annex II). The inadequate implementation periods of many projects has led to a great deal of pressure to get project activities completed as quickly as possible, while not creating the proper conditions for a gradual withdrawal from the project of the external assistance. Sustainability will depend mostly on the commitment and the capacity of the beneficiary governments and institutions to continue to support development and implementation of reforms in the functioning of central government services, and more generally of public administration, after the Commission's support has come to an end. Apart from the political will required for such continued support, there are usually also financial implications, in terms of increased costs to the national or local government budget. In many countries audited, this creates significant doubts about the continuation of many reform and capacity development processes without on-going technical and/or financial assistance.

CONCLUSIONS AND RECOMMENDATIONS

Does the Commission have a sound strategy for institutional capacity development, including the possible use of technical assistance?

44. Although institutional capacity building has been recognised in the European Community's development policy as a key element in the fight against poverty, the Commission has not included in its Country Strategy Papers an overall assessment and prioritisation of existing institutional weaknesses in individual countries, nor is such information used for the selection of focal sectors for EC support. Neither does the Commission have guidelines on the role technical assistance should play in this context (paragraphs 9 to 12). Recommendation 1In its Country Strategy Papers, the Commission should make a comprehensive and structured analysis of existing institutional capacity weaknesses and of capacity development needs.Recommendation 2The Commission should develop guidelines on technical assistance defining its role in the area of capacity development and providing a sound approach and tools to consider when and how to use it.

Are technical assistance activities within capacity development projects well designed?

45. At the project level, technical assistance-related capacity development projects selected for support are mostly consistent with the, usually very widely formulated, overall national development strategies of the countries concerned, but they often lack a detailed justification of the specific technical assistance needs. Project objectives are frequently over-ambitious, given the tight timeframes set, and do not take sufficiently into account the weak institutional environment in which the projects take place, the level of local ownership of the project and the commitment to the proposed institutional changes, thereby impeding the proper formulation and implementation of the technical assistance tasks (paragraphs 13 to 17). Recommendation 3Design of capacity development projects should be improved, by facilitating effective ownership and leadership of the national part of the process, by better defining specific capacity development objectives and related technical assistance requirements, by avoiding overly complex implementation structures, by being more realistic in terms of objectives to be achieved and by planning longer implementation periods.

Are technical assistance activities within capacity development projects implemented efficiently?

46. For a number of reasons, implementation periods are in most cases too short to implement all project activities and achieve the project objectives. This, in many cases, leads to a necessary extension of the project period. The main problems affecting project implementation, thereby reducing the technical assistance effectiveness and efficiency, are the following: — the long preparation phase often makes the project design outdated by the time implementation starts (paragraphs 19 to 21); — the lengthy process of procuring the technical assistance and starting up the project significantly reduces the implementation time (paragraphs 22 to 26); — experts selected are frequently not available once contracts have been awarded (paragraphs 27 to 28); — the current procurement procedures for contracting technical assistance experts are not conducive to the participation of national public institutions (paragraphs 29 to 30); — donor coordination varies but in many cases technical assistance support is still provided through separate uncoordinated projects, while implementation arrangements do not encourage local ownership (paragraphs 32 to 34). Recommendation 4The procedures governing the project preparation and start-up phase, including the procurement of technical assistance, should be reviewed, in order to create more time for implementation, and more flexibility should be allowed during the inception phase to adjust the project design and/or the Terms of Reference for the technical assistance to changes in circumstances.Recommendation 5The evaluation criteria in technical assistance tenders should be reviewed, in order to better reflect the quality and previous experience of the experts and the consultancy company.Recommendation 6More options should be considered regarding procurement possibilities to allow the best possible choice of technical expertise, including expertise from public institutions and expertise available in the beneficiary country or the region.Recommendation 7In line with the Paris Declaration commitments, the Commission should increase its use of technical assistance through coordinated programmes and apply, where possible, implementation arrangements which encourage local ownership.

Are technical assistance activities and performance effectively monitored and evaluated?

47. The Court found that the reporting, monitoring and evaluation of technical assistance activities were in most cases adequate. Project reporting, monitoring and evaluation are carried out regularly and used to identify and address problems and to adjust and refocus projects where necessary. However, there is no system in place to assess and record the performance of technical assistance companies or experts, which could be used for future reference when experts or technical assistance companies are being recruited for Commission-financed projects (paragraphs 35 to 39). Recommendation 8Technical assistance performance by companies and experts should be assessed systematically and a management information system for recording, reporting and consulting this performance should be developed.

Do technical assistance activities produce satisfactory results in terms of capacity development?

48. In the light of the difficult political and institutional environment in which they take place, overall, the results of Commission-financed capacity development projects can be considered as reasonable. While the quality of the technical assistance provided was generally found to be satisfactory, other important factors have reduced the effectiveness of the support provided. Capacity development is a difficult task, the more so because, as many of the projects audited have shown, existing institutional capacity is often weak and local commitment to institutional changes not always strongly present.

This report was adopted by the Court of Auditors in Luxembourg at its meeting of 17 October 2007. For the Court of Auditors Hubert WEBER President

(1) In this report the term ‘expert’ will be used to denote the different providers.

(2) Communication from the Commission to the Council and the European Parliament: The European Community's Development Policy, 26.4.2000 COM(2000) 212 final.

Joint Statement by the Council and the representatives of the Governments of the Member States meeting within the Council, the European Parliament and the Commission: ‘The European Consensus on Development’, General Affairs and External Relations Council on 22 November 2005 (14820/05).

(3) DAC Network on Governance: ‘The challenge of capacity development: working towards good practice’, February 2006. DAC is the Development Assistance Committee of the Organisation for Economic Cooperation and Development (OECD).

(4) Millennium Development Goals or MDGs: see www.un.org/millenniumgoals and glossary.

(5) See www.aidharmonization.org and glossary.

(6) Divided almost equally between the EDFs (covering all ACP countries) and the general budget (covering the three main geographical programmes for the Middle East and southern Mediterranean, Latin America and Asia).

(7) Either through the General Budget of the European Union or through the European Development Funds.

(8) The 12 countries are: Algeria, Angola, Benin, Uganda, Vietnam (all visited), and Chad, Indonesia, Mexico, Morocco, Nigeria, Peru, Sierra Leone (documentary review).

(9) At the time of the audit the current Country Strategy Papers (CSPs) covered the period 2002-2007 (for the EDF-countries) or 2002-2006 (for the non-EDF countries). Where new draft CSPs for the period 2007/8-2013 were already available, these were also examined.

(10) Quality Support Groups are Commission inter-service groups which perform peer reviews of external assistance measures managed by EuropeAid during their identification and formulation.

(11) Practical Guide to contract procedures for EC external actions: for services contracts with a value of 200 000 euro or more an international restricted tender procedure is compulsory.

(12) Council Regulation (EC, Euratom) No 1605/2002 of 25 June 2002 on the Financial Regulation applicable to the general budget of the European Communities (OJ L 248, 16.9.2002, p. 1), Article 166; and the Financial Regulation of 27 March 2003 applicable to the 9th EDF (OJ L 83, 1.4.2003, p. 1), Articles 53 and 54. This rule requires contracts for works, supplies or services to be concluded not later than 3 years after the date when the Commission takes the Financing Decision concerning a project.

(13) Council Regulation (EC, Euratom) No 1995/2006 of 13 December 2006 amending Regulation (EC, Euratom) No 1605/2002 on the Financial Regulation applicable to the general budget of the European Communities (OJ L 390, 30.12.2006, p. 1), Article 100. The Financial Regulation applicable to the 9th EDF has not yet been modified in this respect.

(14) This means that the award of the contract will be subject to the conclusion of a financing agreement and the provision of funding.

(15) ACP-EU Partnership Agreement, Annex IV — Implementation and Management Procedures, Article 19.

(16) Experts are required to sign a statement that they will be available during the validity period of the offer, which is usually 90 days.

(17) According to the Commission's guidelines, the technical aspects (i.e. the ‘quality’) of the services are given a ‘weight’ of 80 % in the overall assessment. The financial aspect is not only determined by the remaining 20 % weight in the combined assessment, but also by the fact that all offers should stay within the limit of the available budget.

(18) Because of frequent use of short-term service contracts throughout the project cycle, the Commission has established, through an international tender, so-called Framework Contracts (FWC). These are global contracts, which establish the essential terms (e.g. duration, subjects, prices, and conditions of performance) for the award of specific, individual contracts required during a given period.

ANNEX I LIST OF THE PROJECTS AND PROGRAMMES EXAMINED (euro) Project Amount TA Amount COUNTRIES VISITED Algeria Support Programme for the reform of telecommunications and postal and development services of the Information Society (PTT) 17 000 000 13 750 742 Programme of modernisation and assistance in respect of administrative reforms (MARA) 25 000 000 10 899 900 Support Programme for education reform (Education) 17 000 000 8 997 400 Support for the reform of justice in Algeria (Justice) 15 000 000 6 175 960 Angola Training of Primary School Teachers 6 000 000 950 000 Support Programme for the Health Sector 25 000 000 7 150 000 Support for the National Institute of Statistics 1 900 000 475 000 Strengthening the Planning and Budget Process 13 500 000 6 500 000 Benin Support for the Health Sector 10 200 000 2 160 000 Support Programme for the starting-up of communes — PRODECOM 8 850 000 4 681 900 Support for the Integrated Programme for strengthening legal and judiciary systems 12 150 000 1 394 630 Uganda Support for National Authorising Officer's (NAO) Office 1 725 000 1 725 000 Poverty alleviation budgetary support IV (TA component only) 1 190 000 921 540 Development of Human Resources for Health 17 000 000 2 874 844 Technical Assistance to the Road Agency Formation Unit 1 999 500 1 746 788 Vietnam European Technical Assistance Programme for Vietnam (ETV2) 11 034 800 5 850 560 Support for the Renovation of Education Management (SREM) 12 000 000 2 979 620 Institutional Support Project (ISP) 8 000 000 2 549 100 Poverty Reduction Support Credit 3 (PRSC3) (TA component only) 4 500 000 4 100 000 COUNTRIES NOT VISITED Indonesia Trade Support Programme 8 500 000 5 488 682 Mexico Strengthening and modernisation of justice administration in Mexico 4 600 000 2 071 825 Facilitation of the EU-Mexico free trade agreement (PROTLCUEM) (EU contribution) 8 000 000 4 249 580 Morocco Modernisation of jurisdictions (EU contribution) 27 671 000 4 634 720 Support for implementing the Association Agreement (PAAA) — Stage I 5 000 000 5 000 000 Nigeria Economic Management Capacity Project (EMCAP) 12 000 000 939 300 Support for reforming institutions programme (SRIP) 27 600 000 10 780 409 Peru Support for the vocational education and training system of Peru (APROLAB) 6 000 000 1 200 000 Programme in support of the reform of the Peruvian system of justice (JUSPER) 12 000 000 1 400 000 Sierra Leone Institutional Support for the Ministry of Finance 4 500 000 1 262 016 Health sector support programme (HSSP) 28 000 000 7 533 000 Chad Support for health policy 42 084 965 9 757 541 Budgetary support programme for the reduction of poverty and growth (2003-2006) (TA component only) 4 000 000 4 000 000 Total 399 005 265 144 200 057 NB: For the budget support programmes, only the TA component is taken into account in the project amount.

ANNEX II EFFECTIVENESS AND PROSPECTS FOR SUSTAINABILITY OF PROJECTS AND PROGRAMMES EXAMINED Effectiveness Prospects of sustainability Successful Partially successful Unsuccessful Too early Good Uncertain Doubtful Too early COUNTRIES VISITED Algeria Support Programme for the reform of telecommunications and postal and development services of the Information Society (PTT) × × Programme of modernisation and assistance in respect of administrative reforms (MARA) × × Support Programme for education reform (Education) × × Support for the reform of justice in Algeria (Justice) × × Angola Training of Primary School Teachers × × Support Programme for the Health Sector × × Support for the National Institute of Statistics × × Strengthening the Planning and Budget Process × × Benin Support for the Health Sector × × Support Programme for the starting-up of communes — PRODECOM × × Support for the Integrated Programme for strengthening legal and judiciary systems × × Uganda Support for National Authorising Officer's (NAO) Office × × Poverty alleviation budgetary support IV (TA component only) × × Development of Human Resources for Health × × Technical Assistance to the Road Agency Formation Unit × × Vietnam European Technical Assistance Programme for Vietnam (ETV2) × × Support for the Renovation of Education Management (SREM) × × Institutional Support Project (ISP) × × Poverty Reduction Support Credit 3 (PRSC3) (TA component only) × × COUNTRIES NOT VISITED Indonesia Trade Support Programme × × Mexico Strengthening and modernisation of justice administration in Mexico × × Facilitation of the EU-Mexico free trade agreement (PROTLCUEM) (EU contribution) × × Morocco Modernisation of jurisdictions (EU contribution) × × Support for implementing the Association Agreement (PAAA) — Stage I × × Nigeria Economic Management Capacity Project (EMCAP) × × Support for reforming institutions programme (SRIP) × × Peru Support for the vocational education and training system of Peru (APROLAB) × × Programme in support of the reform of the Peruvian system of justice (JUSPER) × × Sierra Leone Institutional Support for the Ministry of Finance × × Health sector support programme (HSSP) × × Chad Support for health policy × × Budgetary support programme for the reduction of poverty and growth (2003-2006) (TA component only) × × Total 7 8 7 10 4 15 3 10