Evaluation of the European Commission-European Youth Forum operating grant agreements 2007-2011/12 : annexes to the final report.
Date: 2015-07-24
Institutions: Directorate-General for Education, Youth, Sport and Culture
Subjects: EU financing, EU programme, European citizenship, education policy, young person, youth policy

Evaluation of the European 
Commission-European Youth 
Forum Operating Grant Agreements 
2007-2011/12 
Annexes to the final report 

Client: DG EAC 

Rotterdam, 6 November 2013 

 
 
 
 
 
 
 
 
 
Evaluation of the European 
Commission-European Youth 
Forum Operating Grant Agreements 
2007-2011/12 

Annexes to the final report 

Client: DG EAC 

Sacha Koppert 

Susan Warmerdam 

Reinout van Brakel 

Rotterdam, 6 November 2013 

Cat. N°. NC-02-13-493-EN-N 

ISBN 978-92-79-33660-7 

doi: 10.2766/63989 

Disclaimer: 

The conclusions, recommendations and opinions in this report are those of the authors 
and they do not necessarily represent the views of the European Commission. 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Table of contents 

Annex 1 

Terms of Reference 

Annex 2 

Methodology 

Annex 3 

Literature 

Annex 4 

Research tools: Questionnaire 

Annex 5 

Research tool: Cost effectiveness analysis 

Annex 6 

Survey respondents and results 

Annex 7 

List of interviewees 

Annex 8 

Financial information-available upon request 

page 

5 

27 

29 

33 

45 

47 

49 

53 

4 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ANNEX 1: TERMS OF REFERENCE 

Evaluations of 

The European Commission - European Youth Forum operating 
grant agreements 

& 

The European Commission - Council of Europe youth partnership 
agreements 

for the years 2007-2011/12 

Contracting Authority: European Commission 

1.  Context ......................................................................................................................... 6 
1.1  Background ........................................................................................................... 6 
1.2  Objectives of the evaluation subjects .................................................................... 7 
1.3  Description of the evaluation subjects .................................................................. 9 
1.3.1  Actions .............................................................................................................  
1.3.2  Budget and duration of the evaluation subjects ...............................................  
1.3.3  Countries involved ...........................................................................................  
1.4 
Implementation .......................................................................................................  
1.5  Evaluation and Monitoring Provisions ................................................................ 13 
1.5.1  Monitoring Provisions ................................................................................. 13 
Previous evaluations .................................................................................... 13 
1.5.2 
2.  Task specification for the assignment ........................................................................ 14 
2.1  Evaluation questions ........................................................................................... 14 
2.1.1  Relevance and coherence ............................................................................. 14 
Effectiveness ................................................................................................ 15 
2.1.2 
Efficiency ..................................................................................................... 18 
2.1.3 
Sustainability ............................................................................................... 18 
2.1.4 
2.2  Other tasks under the assignment ........................................................................ 19 
3.  Reporting and deliverables ........................................................................................ 19 
4.  Organisation, timetable and budget ........................................................................... 21 
4.1  Organisation ........................................................................................................ 21 
4.2  Meetings .............................................................................................................. 22 
4.3  Timetable ............................................................................................................. 22 
4.4  Budget ................................................................................................................. 22 
5.  References .................................................................................................................. 23 
5.1  Basic documents and Web references ................................................................. 23 
5.2  Documents and information to be provided after contract signature (not 
exhaustive) ..................................................................................................................... 24 
6.  Requirements ............................................................................................................. 24 
6.1  Methodology ....................................................................................................... 24 
6.2  Resources ............................................................................................................ 24 
7.  Award criteria ............................................................................................................ 25 
7.1  Quality criteria .................................................................................................... 25 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

5 

 
 
 
 
 
 
 
7.2 
Financial criteria .................................................................................................. 25 
7.3  Contract award .................................................................................................... 25 

The contract covers the evaluations of  

1)  The  European  Commission's  operating  grant  agreements  with  the 
European Youth Forum  

and 

2) The EU-CoE youth partnership (between the Commission and the Council 
of Europe)  

The contractor should carry out two evaluations that are bundled together in this 
contract because of shared overall objectives, a common political and legal basis 
and  overlaps  in  stakeholders  involved.  For  the  specific  evaluations  of  the  two 
subjects, the contractor should provide two separate reports. 

–  CONTEXT 

a. 

Background 

Youth Policy 

Since 1993, the Treaty establishing the European Community includes a mandate 
to  "encourage  the  development  of  youth  exchanges  and  of  exchanges  of  socio-
educational  instructors".  The  current  legal  base  is  article  165 of  the  Treaty  of 
Lisbon,  which  envisages  EU  action  to  aim  at  "encouraging  the  development  of 
youth  exchanges  and  of  exchanges  of  socio-educational  instructors,  and 
encouraging the participation of young people in democratic life in Europe". 

To  step  up  participation  of  young  people  in  democratic  life  in  Europe,  the 
Commission and Member States are working together under the Open Method of 
Coordination (OMC).  The OMC is a non-binding, intergovernmental framework 
for  cooperation  and  policy  exchange.  The  OMC  in  the  youth  field  is  led  by  the 
Council of Youth Ministers, which meets twice a year, and further developed and 
implemented by civil servants in Member States and the Commission, also based 
on inputs from experts and other stakeholders. 

The  current  EU  Youth  Strategy1  is  based  on  a  Council  Resolution  from  27 
November  2009  renewing  the  overall  policy  framework  for  cooperation  in  the 
youth field from 2010-2018. It pursues two overall objectives: 

  To provide more and equal opportunities for young people in education and 

in the labour market; 

  To encourage young people to be active citizens and participate in society.  

It advocates a cross-sectorial approach to youth issues and proposes activities in 
eight fields of action: Education and Training; Employment and Entrepreneurship; 
Health  and  Well-being;  Participation;  Voluntary  activities;  Social  inclusion; 

1  

Endorsed by the Council on 27 November 2009 further to a Commission Communication in April 2009. Texts and further 
information, see http://ec.europa.eu/youth/youth-policies/doc1687_en.htm. 

6 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
                                                           
Youth  and  the  world  and  Creativity  and  Culture.  Youth  work  development  is 
supported as a cross-sectorial activity.  

The Strategy is implemented through the following tools: knowledge building and 
evidence-based  youth  policy;  mutual  learning;  progress  reporting;  dissemination 
of  results;  monitoring;  consultations  and  structured  dialogue  with  young  people 
and youth organisations.  

Youth in Action Programme 

Programmes like "Youth" (2000-2006) and "Youth in Action" (2007-2013) have 
supported  European  Youth  Policy  for  more  than  a  decade.  These  programmes 
build  on  the  experiences  of  the  former  programmes  "Youth  for  Europe"  and 
"European Voluntary Service". 

The  current  Youth  in  Action  Programme  2007-20132  has  the  following  general 
objectives:  

a)  To  promote  young  people's  active citizenship  in  general  and  their  European 

citizenship in particular; 

b)  To  develop  solidarity  and  promote  tolerance  among  young  people,  in 

particular in order to reinforce social cohesion in the EU; 

c)  To foster mutual understanding between young people in different countries; 

d)  To contribute to developing the quality of support systems for youth activities 

and the capabilities of civil society organisations in the youth field; 

e)  To promote European cooperation in the youth field. 

b. 

Objectives of the evaluation subjects 

European Youth Forum 

The Youth in Action programme 2007-2013 (point 4.2 'Support for the European 
Youth  Forum'  of  the  Annex)  lists  the  objectives  (referred  to  as  'activities')  of 
the European Youth Forum: 

  Representing youth organisations vis-à-vis the EU; 
  Coordinating the positions of its members vis-à-vis the EU; 
  Relaying information on youth vis-à-vis the European Institutions; 
  Relaying information form the EU to the national youth councils and non-

governmental organisations; 

  Promoting and preparing the participation of young people in democratic 

life; 

  Contributing  to  the  new  cooperation  framework  in  the  youth  field 

established at the level of the EU; 

  Contributing  to  the  development  of  youth  policies,  youth  work  and 
educational  opportunities,  and  to  relaying  information  concerning  young 

2  

Decision No 1719/2006/EC of the European Parliament and of the Council of 15 November 2006 establishing the ‘Youth in 
Action’ programme for the period 2007 to 2013. 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

7 

 
 
 
 
 
 
 
                                                           
people  and  developing  representative  structures  for  young  people 
throughout Europe; 

  Engaging  in  discussion  and  reflection  on  youth  in  Europe  and  in  other 
parts of the world and on the Community's action for young people.  

The Youth in Action programme foresees (Annex, point 4.2), that grants may be 
awarded to support the on-going activities of the European Youth Forum with due 
respect for the following principles: 

 
 

  The  Forum's  independence  in  the  selection  of  its  members,  ensuring  the 
broadest possible representation of different kinds of youth organisations; 
It's autonomy in the detailed specification of its activities; 
the broadest possible involvement in the Forum's activities of non-member 
youth organisations and young people who do not belong to organisations; 
  The active contribution by the Forum to the political processes relevant to 
youth  at  European  level,  in  particular  by  responding  to  the  European 
institutions  when  they  consult  civil  society  and  explaining  the  positions 
adopted by these institutions to its members. 

EU-CoE youth partnership 

The EU Youth Strategy calls for, as part of its tool 'Mutual learning', cooperation 
with  international  organisations  such  as  the  Council  of  Europe,  in  order  to 
provide a source of reference and inspiration. Such cooperation is in line with 
the Youth in Action Programme general objective of promoting cooperation in the 
youth field. Article 7 of this programme stipulates that it is open to the Council of 
Europe.  Under  point  5.3  'Cooperation  with  international  organisations'  of  the 
annex,  it  indicates  that  under  this  action  line  cooperation  with  the  Council  of 
Europe can be supported. 

The  "EU-CoE  youth  partnership"  is  the  framework  for  cooperation  with  the 
Council  of  Europe.  It  is  regulated  by  periodical  Framework  Partnership 
Agreements  (FPA)  between  the  Commission  and  the  Council  of  Europe 
(hereinafter referred to as the 'partner institutions').  

General objectives 

The current FPA (2010-2013) stipulates that the EU-CoE youth partnership aims 
to  provide  a  framework  for  the  joint  development  of  cooperation  and  a 
coherent strategy in the field of youth in Europe with the direct involvement 
of young people, youth workers youth researchers and youth policy makers. 
It  should  contribute to  complementarity  and  coherence  between  the  activities 
of  the  partner  institutions,  also  on  current  and  future  cooperation  between  the 
institutions outside the Partnership framework. 

Specific objectives  

Activities should support the fields of actions and key areas identified by the two 
partners  in  their  political  strategies.  They  aim  at  creating  better  opportunities 
and  improving  access  for  young  people,  promoting  their  active  citizenship 

8 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
and 

fostering 

solidarity 

in 

the 

following 

fields  of  cooperation: 

 
 
 
 
 

social inclusion; 
democracy;  
human rights;  
citizenship and participation;  
intercultural dialogue.  

The EU-CoE youth partnership shall foster cooperation in the following fields of 
action: 

  Promoting evidence-based youth policy; 

  Supporting youth work and capacity building; 

  Activities  with  a  geographic  scope  (South  East  Europe;  Mediterranean; 

Eastern Partnership; Russia; and Africa); 

  Information and dissemination. 

c. 

i. 

Description of the evaluation subjects 

Actions 

European Youth Forum 

The European Youth Forum is the platform representing National Youth Councils 
and  international  non-governmental  youth  organisations  from  across  Europe.  It 
represents young people and advocates their needs and interests and those of their 
organisations.  The  Youth 
legal  basis  (Decision 
N°1719/2006/EC)  stipulates  that  the  objectives  and  main  activities  of  the 
European Youth Forum are as indicated in point 1.2. 

in  Action  Programme's 

The European Youth Forum's statutes define the Forum's aims as: 

  Being  a  consultative  body  for  international  institutions  (European  Union, 
Council  of  Europe,  United  Nations  Organisation,  etc.)  on  all  issues 
relevant to young people and to youth organisations; 

  Promoting youth policy through government and institutional policy; 

 

 

Influencing the policy of international institutions on youth related issues; 

Increasing  the  participation  of  young  people  and  youth  organisations  in 
society as well as in the decision making process; 

  Promoting  the  exchange  of  ideas  and  experiences,  mutual  understanding, 

and equal rights and opportunities among young people in Europe. 

The  European  Youth  Forum's  Work  Plan  2011-2012  lists  the  activities  of  the 
Forum in the following main areas: 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

9 

 
 
 
 
 
 
 
 
 
  Young Volunteers: Builders of European Society:   

The  European  Youth  Forum  advocates  for  a  Europe  where  the  role  of 
youth  volunteering  is  actively  promoted,  recognised  and  supported.  It 
particularly supports participatory youth organisations where the volunteer 
has access to all decision-making levels within the organisation and a high 
level of influence.  

  Strong Youth Organisation for stronger Civil Society: 

In  order  to  build  the  capacity  of  youth  organisations,  a  stronger  Civil 
Society  and  their  outreach,  the  European  Youth  Forum  focuses  on  all 
aspects  that  youth  organisations  bring  to  societies  and  builds  sustainable 
support  mechanisms  to  enable  more  and  more  diverse  young  people  to 
fulfil their potential in youth organisations and contribute to the creation of 
a more connected society. 

  A Rights-Based Approach for an Inclusive and Participatory Europe; 

Furthermore,  the  European  Youth  Forum  works  towards  the  aim  of  full 
realisation  of  the  rights  of  young  people  as  a  pre-requisite  to  further 
develop and build a democratic Europe with a sound social and economic 
model at all levels. 

EU-CoE youth partnership  

The EU-CoE youth partnership focuses on a limited number of activities. In order 
to promote evidence-based policy, the EU-CoE youth partnership coordinates:  

  The  European  Knowledge  Centre  on  Youth  Policy  (EKCYP)  which  consists 
of a web-portal and a network of national correspondents. It gathers data and 
research on youth issues subsequently made available in the EKCYP database; 

  The  Pool  of  European  Youth  Researchers  (PEYR)  which  consists  of  25 
experienced  researchers  from  across  Europe  with  different  specialisations 
who,  through  meetings  and  contributions  facilitate  a  closer  link  between 
European youth policy and youth research; 

  Thematic fora, seminars and events to enhance knowledge on youth. 
In  order  to  support  youth  work  and  build  its  capacity,  the  EU-CoE  youth 
partnership  promotes  recognition  of  youth  work  and  non-formal  learning  and 
innovation in youth work. It provides trainings and peer reviews to exchange good 
practices and support policy. 

The EU-CoE youth partnership pays special attention to the geographic scope of 
its  initiatives  by  prioritising  activities  involving  countries  that  are  candidates  to 
EU accession, part of EU neighbourhood policy and other countries 'on demand', 
such as e.g. EU-Africa cooperation. 

The  EU-CoE  youth  partnership  disseminates  youth-related  information  and 
publications, online through its web-portal3, which includes information linked to 
EKCYP and PEYR (see bullet points above) including on youth policy topics and 
country specific information. It releases publications and training materials for the 
youth sector and a magazine for youth workers called 'Coyote'. 

3  

http://youth-partnership-eu.coe.int/youth-partnership/. 

10 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
                                                           
ii. 

Budget and duration of the evaluation subjects 

The Youth in Action programme 2007-2013 stipulates that the European Youth 
Forum receives an annual operating grant of not less than EUR 2 million from the 
European Commission.  At least 20%  of the Forum's budget  must  be covered by 
non-Community sources. 

The  EU-CoE  youth  partnership  is  financed  with  equal  contributions  by  the 
Council  of  Europe  and  the  European  Commission  in  the  same  proportion.  The 
Commission  gives  annual  grants  to  the  Council  of  Europe,  under  which  it 
reimburses 50% of the partnership's costs, which are managed by the Council of 
Europe.  Annual  grant  agreements  include  specific  work  plans  and  are  supported 
by the Commission for usually of about EUR 600.000 per year.  

iii. 

Countries involved 

Activities  are  in  principle  open  to  young  people,  youth  workers,  researchers, 
policy-makers  and  resource  persons.  They  are  mainly  addressed  to  the  needs  of 
young  people  in  the  Member  States  of  both  partner  institutions  and  programme 
countries of the Community Youth in Action programme. 

Young people and resource persons from other countries are invited to participate 
when considered relevant. 

d. 

Implementation 

European Youth Forum 

The European Commission has financially supported the European Youth Forum 
and its preceding bodies for more than 20 years. The Youth in Action Programme 
2007-2013  stipulates  that  the  Programme  may  award  grants  to  the  European 
Youth Forum, an organisation pursuing a goal of general European interest, while 
foreseeing that the Forum respects a number of principles (see point 1.2 above). 

The European Youth Forum receives an annual operating grant, which is awarded 
on  the  basis  of  a  work  programme  submitted  by  the  European  Youth  Forum, 
followed by approval by the Commission services. 

The  grants  cover  activities  described  in  the  European  Youth  Forum's  work 
programme  and  administrative  costs  to  run  the  Forum.  The  Forum's  Secretariat 
involves usually about 20 young people, placed in Brussels. 

The Forum's highest decision making body is the "General Assembly". It agrees 
e.g.  on  the  Forum's  policy  guidelines,  the  work  plan  and  new  members.  The 
"Council  of  Members"  defines  the  Forum's  policies,  vision  and  direction  on  the 
basis of these guidelines. Guidelines and policies are transposed into strategy by 
the  "Board",  which  is  also  responsible  for  financial  and  budgetary  matters. 
"Secretary-General" and "Secretariat" carry out the Forum's day-to-day work. 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

11 

 
 
 
 
 
 
 
 
The European Youth Forum's statutory structures and their tasks can be found on 
the  European  Youth  Forum's  website:  http://www.youthforum.org/  and  in  the 
European Youth Forum's Statutes. 

EU-CoE youth partnership 

The cooperation with the Council of Europe in the youth field goes back to 1998 
and  has  in  later  years  been  organised  through  multi-annual  Framework 
Partnership Agreements  between  the  two  institutions.  Specific  annual EU  grants 
are  allocated  by  the  Commission  to  the  Council  of  Europe  under  the  Youth  in 
Action Programme. These grants are awarded on the basis of a work programme 
submitted  by  the  Council  of  Europe  and  then  approved  by  the  Commission 
services. 

The  grants  cover  activities  described  in  a  work  programme  and  administrative 
costs to run a team of 9 staff (6 in Strasbourg, 2 in Brussels and 1 in Budapest). 
This team has no separate legal entity. Staff is recruited by the Council of Europe 
but the manager reports to both institutions. Staff is placed in the location of one 
of  the  partners;  currently  staff  is  placed  in  the  Council  of  Europe  premises,  two 
staff  members  having  been  moved  from  Strasbourg  to  the  Council  of  Europe's 
Brussels  office  with  the  objective  of  strengthening  cooperation  with  the 
Commission.  

The  EU-CoE  youth  partnership  team  is  in  charge  of  the  following  (non-
exhaustive) list of tasks: 

  Managing the implementation of the EU-CoE youth partnership work plan, as 

approved and amended by both partner institutions; 

  Ensuring  quality  of  actions  taken  in  terms  of  contents,  impact,  visibility, 

dissemination and administration; 

  Acting  as  a  liaison  group  and  point  to  information  vis-à-vis  the  two  partner 

institutions; 

  Reporting  adequately  and  timely  to  both  partner  institutions  on  all  its 

activities; 

  Ensuring  equal  visibility  of  both  partner  institutions  in  information  and 

communication. 

Insofar  as  appropriate,  the  institutions  take  part  in  EU-CoE  youth  partnership 
activities and invite the EU-CoE youth partnership to theirs. There is also mutual 
information  exchange  and  support  to  inform  the  general  public  about  relevant 
activities.  The  EU-CoE  youth  partnership  team  cooperates  with  and  actively 
involves other organisations in the youth field. 

Decisions  on  the  EU-CoE  youth  partnership,  including  Framework  Partnership 
Agreements  and  annual  specific  grant  agreements,  are  taken  by  the  Partnership 
Management Board (PMB), consisting of Directors of Youth of the Commission 
and the Council of Europe. In meetings of the PMB, delegations are composed of 
Commission staff and, on the side of the Council of Europe, by staff members of 
the secretariat and representatives of the statutory bodies. The partner institutions 
also  invite  the  staff  of  the  EU-CoE  youth  partnership  and  the  European  Youth 

12 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
Forum as observers. There is at least one Partnership Management Board meeting 
a year; the venues alternate between Brussels and Strasbourg.  

An  Advisory  Group4  consisting  of  stakeholders  in  the  European  youth  sector 
meets once a year to provide input to the following year's work plan of the EU-
CoE youth partnership.  

EU-CoE youth partnership activities should take place in countries covered by the 
institutions.  A  balance  approaching  equal  distribution  should  be  pursued  for 
locating  activities  and  meetings  to  one  of  the  cities  hosting  either  of  the  partner 
institutions. 

e. 

i. 

Evaluation and Monitoring Provisions 

Monitoring Provisions 

European Youth Forum 

  Frequent  and  direct  contacts  between  the  European  Youth  Forum  and  the 
European  Commission  at  policy  and  project,  management  and  desk  officer 
levels; 

  Every  year,  the  European  Youth  Forum  reports  to  the  Commission  on  the 
implementation  of the work programme and the  budgetary  execution  both at 
interim and final report level. 

EU-CoE youth partnership 

  Frequent and direct contacts between the Commission and the EU-CoE youth 

partnership on day-to-day work; 

  Meetings of the Partnership Management Board and Advisory Board; 

  After  the  closing  of  the  year,  the  Council  of  Europe  sends  a  report  on  the 
implementation of the action, both describing activities carried out and budget 
used. 

Interim  and  Final  reports  of  the  European  Youth  Forum's  and  EU-CoE  youth 
partnership will be shared with the contractors once the contract is signed. 

ii. 

Previous evaluations 

European Youth Forum 

  Final external evaluation of the Community action programme to promote 

bodies active at European level in the field of youth (2004-2006) 
http://ec.europa.eu/dgs/education_culture/evalreports/index_en.htm 

4  

The group involves representatives from the European Youth Forum, the National Agencies and SALTO resource centres 
of the Commission's Youth in Action programme, the Eurodesk network, the statutory bodies of the Council of Europe, as 
well as policy-makers, researchers and other interested parties. 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

13 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                                           
EU-CoE youth partnership 

  Ex-ante  Evaluation  study  of  the  research  and  policy  strand  of  the 
Partnership  on  Youth  between  the  Council  of  Europe  and  the  European 
Commission by Maria-Carmen Pantea (2008); 

  Evaluation of the partnership covenant on training in youth work between 
the European Commission and the Council of Europe by Deloitte & 
Touche (2004) 
http://ec.europa.eu/dgs/education_culture/evalreports/index_en.htm 

–  TASK SPECIFICATION FOR THE ASSIGNMENT 

The  purpose  of  this  contract  is  to  obtain  two  external,  independent  evaluations 
concerning  the  agreements  made  between  the  Commission  and  the  European 
Youth  Forum  and  the  Commission  and  the  Council  of  Europe,  which  include 
financial support in line with the Youth in Action Programme 2007-2013 and the 
EU Youth Strategy. The evaluations should cover the time period 2007-2011 and 
try to include the first six month of the year 2012 as far as possible.  

The evaluation of the support to the European Youth Forum shall assess activities, 
outputs  and  results  against  set  objectives,  as  indicated  in  the  Youth  in  Action 
programme  2007-2013  and  the  annual  operating  grant  agreements  (including 
work plans), focusing on relevance, efficiency, effectiveness and sustainability. It 
should  compare  results  with  the  previous  evaluation  done  in  this  respect  (Final 
external evaluation of the Community action programme to promote bodies active 
at European level in the field of youth (2004-2006)) and evaluate the wider impact 
and  effects  of  the  European's  Youth  Forum's  activities,  outputs  and  results  on 
policy, research, youth work / support practice and youth in Europe. 

The  evaluation  of  the  EU-CoE  partnership  shall  assess  the  EU-CoE  youth 
partnership's activities, outputs and results against set objectives, as set out in the 
Framework  Partnership  Agreement  and  the  annual  specific  grant  agreements 
(including  work  plans),  focusing  on  relevance,  efficiency,  effectiveness  and 
sustainability.  The  evaluation  should  assess  the  wider  impact  and  effects  of  the 
partnership  agreement's  activities,  outputs  and  results  on  policy,  research,  youth 
work / support practice and youth in Europe. It should also include an assessment 
of whether and how Commission interests are fulfilled through this cooperation. 

a. 

Evaluation questions 

The evaluations shall reply to the following questions: 

i. 

Relevance and coherence 

European Youth Forum  

- To what extent are the objectives and activities of the European Youth Forum 
relevant to: 

14 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
  The objectives of the Youth in Action Programme? 
  The  needs  of  the  specific  target  groups  (such  as  young  people,  National 
Youth  Councils,  non-governmental  youth  organisations,  youth  policy 
makers)? 

  The objectives of the future Erasmus for All Programme? 
  The objectives of the EU Youth Strategy. 

-  To  what  extent  do  the  objectives  and  activities  of  the  European  Youth  Forum 
complement or contradict those of any other EU actions, particularly those under 
the Youth in Action Programme and the future Erasmus for All Programme? 

EU-CoE youth partnership 

- To what extent are the objectives and activities of the EU-CoE youth partnership 
relevant to: 

  The objectives of the Youth in Action Programme? 
  The current needs of its target groups (such as youth workers, 
youth researchers and youth policy makers, the Commission)?  

  The objectives of the future Erasmus for All Programme? 

  The objectives of the EU Youth Strategy? 

- To what extent do the objectives and activities of the EU-CoE youth partnership 
complement or contradict those of any other EU actions, particularly those under 
the Youth in Action Programme and the future Erasmus for All Programme ? 

ii. 

Effectiveness 

European Youth Forum 

- To what extent have the objectives foreseen for the European Youth Forum, as 
referred to in the Youth in Action Programme 2007-2013 (annex point 4.2 and 
listed below) been met? Where expectations have not been met, what factors have 
hindered the achievement of the objectives? 

  Representing youth organisations vis-à-vis the EU; 
  Coordinating the positions of its members vis-à-vis the EU; 
  Relaying information on youth vis-à-vis the European Institutions; 
  Relaying information from the EU to the national youth councils and non-

governmental organisations; 

  Promoting and preparing the participation of young people in democratic 

life; 

  Contributing  to  the  new  cooperation  framework  in  the  youth  field 

established at the level of the EU; 

  Contributing  to  the  development  of  youth  policies,  youth  work  and 
educational  opportunities,  and  to  relaying  information  concerning  young 
people  and  developing  representative  structures  for  young  people 
throughout Europe; 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

15 

 
 
 
 
 
 
 
 
 
 
 
 
 
  Engaging  in  discussion  and  reflection  on  youth  in  Europe  and  in  other 
parts of the world and on the Community's action for young people.  

- To what extent have the principles stipulated in the Youth in Action programme 
2007-2013 (point 4.2 of the Annex and listed below) been materialised? Where 
principles have not been put into practise, what factors have hindered their 
implementation? 

  The Forum's independence in the selection of its members, ensuring the 

broadest possible representation of different kinds of youth organisations; 
Its autonomy in the detailed specification of its activities; 

 
  The broadest possible involvement in the Forum's activities of non-
member youth organisation and young people who do not belong to 
organisations; 

  The active contribution by the Forum to the political processes relevant to 
youth at European level, in particular by responding to the European 
institutions when they consult civil society and explaining the positions 
adopted by these institutions to its members.  

- To what extent has the European Youth Forum contributed to the achievement 
of the objectives of the Youth in Action programme 2007-2013 (article 4.4 and 
listed below): 

  To organise structured dialogue between the various actors in the field of 
youth, in particular young people themselves, those active in youth work 
and youth organisations and policymakers; 

  To  support  of  youth  seminars  on  social,  cultural  and  political  issues  in 

which young people are interested; 

  To contribute to the development of policy cooperation in the youth field; 
  To  facilitate  the  development  of  the  networks  necessary  to  a  better 

understanding of youth. 

-  To  what  extent  has  the  support  to  the  European  Youth  Forum  contributed  to 
achieving  the  Youth  in  Action  programme  2007-2013's  general  objectives  (as 
stipulated in Article 2 of the programme and listed below)? 

  To  promote  young  people's  active  citizenship  in  general  and  their 

European citizenship in particular; 

  To  develop  solidarity  and  promote  tolerance  among  young  people,  in 

particular in order to reinforce social cohesion in the EU;   

  To  foster  mutual  understanding  between  young  people  in  different 

countries; 

  To  contribute  to  developing  the  quality  of  support  systems  for  youth 
activities  and  the  capabilities  of  civil  society  organisations  in  the  youth 
field; 

  To promote European cooperation in the youth field. 

- To what extent has the support to the European Youth Forum produced effects 
that would not have been obtained by alternative options?  

16 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EU-CoE youth partnership  

- To what extent has the EU-CoE youth partnership achieved its general 
objectives to 

  Provide a framework for the joint development of cooperation and coherent 
strategy in the field of youth in Europe with the direct involvement of young 
people, youth workers, youth researchers and youth policy makers?  
  Foster cooperation, complementarity and synergies in the field of youth 

policy, youth research and youth work? 

- To what extent has the EU-CoE youth partnership achieved its specific 
objectives to: 

  Create better opportunities and improve access for young people, promoting 
their active citizenship and fostering solidarity in the following fields of 
cooperation: 
o  Develop social inclusion; 
o  Develop democracy and human rights, democratic citizenship and youth 

participation; 

o  Develop intercultural dialogue and diversity? 

  Cooperate in the following fields of action: 
o  Promote knowledge-based youth policy; 
o  Support youth work; 
o  Build capacity of youth workers; 
o  Pay special attention to the designated geographical scope; 
o  Gather, disseminate and provide information and publications on youth? 

Where expectations have not been met, what factors have hindered the 
achievement of the objectives? 

- To what extent has the EU-CoE youth partnership contributed to achieving the 

EU Youth Strategy's and Youth in Action programme's general objectives: 

o  to provide more and equal opportunities for young people in education 

and in the labour market; 

o  to encourage young people to be active citizens and participate in society 

/ to promote young people's active citizenship in general and their 
European citizenship in particular; 

o  to develop solidarity and promote tolerance among young people, in 

particular in order to reinforce social cohesion in the EU; 

o  to foster mutual understanding between young people in different 

countries. 

- To what extent has the EU-CoE youth partnership produced effects that would 
not have been obtained by alternative options?  

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

17 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
iii. 

Efficiency 

European Youth Forum 

- Is the use of an annual operating grant agreement to support the European Youth 
Forum  the  most  effective  and  efficient  solution  or  would  other  instruments  or 
mechanisms provide better cost-effectiveness? 

-  How  efficiently  has  the  European  Youth  Forum  used  financial  and  human 
resources  to  achieve  the  agreed  outputs  and  results  from  an  economic  point  of 
view? 

- Is the size of the annual operating grant (in terms of volume and percentage of 
the European Youth Forum's overall annual budget) appropriate and proportional 
to  the  effects  that  the  cooperation  with  the  European  Youth  Forum  is  meant  to 
achieve? Could the same effects have been achieved with less funding? 

- How effective are the monitoring mechanisms, including the indicators? 

EU-CoE youth partnership 

- Is the use of multi-annual Framework Partnership Agreements and annual grants 
to support a Partnership Team the most effective and efficient solution or would 
other instruments or mechanisms have provided better cost-effectiveness? 

-  How  efficiently  is  the  EU-CoE  youth  partnership  managed  by  the  partner 
institutions,  in  terms  of  overall  management,  decision-making  procedures, 
supervision,  organisational  setup;  financial,  operational  and  administrative 
procedures; distribution of human resources; and other elements of its operation? 

- Is the size of the annual grant appropriate and proportional to the effects that the 
EU-CoE youth partnership is meant to achieve? Could the same effects have been 
achieved with less funding?  

-  How  efficient  is  the  EU-CoE  youth  partnership  in  its  cooperation  and  joint 
activities with third parties and advisory structures? 

- How effective are the monitoring mechanisms, including the indicators? 

iv. 

Sustainability 

European Youth Forum 

-  Which  of  the  current  activities  or  elements  of  the  EU-European  Youth  Forum 
cooperation would be likely to continue and in which form if the financial support 
from  the  European  Commission  was  withdrawn  or  substantially  decreased  and 
how would such activities be likely to be organised? 

-  To  what  extent  have  the  results  of  the  actions  been  properly  disseminated  to 
stakeholders  and  the  public?  What  is  their  exploitable  potential,  and  to  what 
extent can one say that this potential has been fully exploited? 

18 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EU-CoE youth partnership 

-  Which  of  the  current  activities  or  elements  of  the  EU-CoE  youth  partnership 
would  be  likely  to  continue  and  in  which  form  if  the  financial  support  from  the 
European Commission was withdrawn or substantially decreased and how would 
such activities be likely to be organised? 

-  To  what  extent  have  the  results  of  the  actions  been  properly  disseminated  to 
stakeholders  and  the  public?  What  is  their  exploitable  potential,  and  to  what 
extent can one say that this potential has been fully exploited? 

b. 

Other tasks under the assignment 

Evaluation follow-up 

The Contractor should:  

  Provide  a  one-page  statement  for  each  of  the  organisations  under 
evaluation about the validity of the evaluation results, i.e. to what extent it 
has been possible to provide reliable statements on all essential aspects of 
the  Community  intervention  examined.  Issues  to  be  referred  to  may 
include scoping of the evaluation exercise, availability of data, unexpected 
problems  encountered  in  the  evaluation  process,  proportionality  between 
budget and objectives of the assignment, etc. 

  Make  two  separate  proposals  for  the  dissemination  of  the  evaluation 
results  for  each  organisation  evaluated,  on  the  basis  of  the  draft 
Dissemination Plan annexed to these Terms of Reference. 

–  REPORTING AND DELIVERABLES 

General reporting requirements  

The  Evaluator  should  prepare  at  all  stages  two  separate  reports,  one  for  each 
evaluation subject. 

Each  report  (except  the  final  version  of  the  Final  Report)  should  have  an 
introductory page providing an overview and orientation of the report. It should 
describe  what  parts  of  the  document,  on  the  one  hand,  have  been  carried  over 
from  previous  reports  or  been  recycled  from  other  documents,  and  on  the  other 
hand, represent progress of the evaluation work with reference to the work plan. 

All  reports  must  be  drafted  in  English  and  submitted  according  to  the  timetable 
below to the responsible body. Electronic files must be provided in Microsoft ® 
Word for Windows format. Additionally, besides Word, the Final Report must be 
delivered in Adobe ® Acrobat pdf format and in 5 hard copies. 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

19 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
The Commission will comment on all reports within maximum 30 calendar days. 
In  the  absence  of  observations  from  the  Commission  within  the  deadline  the 
report will be considered as being approved. 

Within  maximum  14  calendar  days  of  receiving  the  Commission’s  observations 
the  Contractor  will  submit  the  report  in  definitive  form,  taking  full  account  of 
these  observations,  either  by  following  them  precisely  or  by  explaining  clearly 
why  they  could  not  be  followed.  Should  the  Commission  still  not  consider  the 
report  acceptable,  the  Contractor  will  be  invited  to  amend  the  report  insofar  as 
such  amendments  do  not  interfere  with  the  independence  of  the  evaluator  in 
respect of their findings, conclusions or recommendations. 

Inception Report 

The inception report must describe in detail how the methodology proposed by the 
Contractor is going to be implemented in the light of an examination of the quality 
and appropriateness of existing data. It should include the Contractor's 
understanding of the intervention logic, as well as the quantitative and qualitative 
indicators that he/she will use in addressing each of the evaluation questions. A 
detailed work plan including the allocation of experts per task per number of 
working-days should also be provided.  

It shall not exceed 30 pages, annexes excluded. 

Interim Report 

The report is to be produced after the desk and field research has been completed, 
and  should,  to  the  extent  possible,  include  some  preliminary  conclusions.  The 
report must as a minimum provide: 

  An overview of the status of the evaluation project; 

  A description of problems encountered and solutions found; 

  A summary of initial findings and results of the data gathering; 

  An assessment of the data, whether it meets expectations and will provide 

a sound basis for responding to the evaluation questions; 

  A  conclusion  whether  any  changes  are  required  to  the  work  plan,  or  any 
other solutions should be sought in order to ensure that the required results 
of the evaluation are achieved. If any such issues are to be identified, they 
must be discussed in the meeting with the Steering Group dedicated to this 
report; 

  A proposal for the final structure of the Final Report, as well as a structure 

of the Executive Summary. 

It shall not exceed 30 pages, annexes excluded. 

Draft Final Report 

This  document  should  deliver  the  results  of  all  tasks  covered  by  these  Terms  of 
Reference, and must be clear enough for any potential reader to understand. 

20 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
It should contain: 

  Main report: The main report must be limited to 75 pages and present, in full, 
the results of the analyses, conclusions and recommendations arising from the 
evaluation.  It  must  also  contain  a  description  of  the  subject  evaluated,  the 
context of the evaluation, and the methodology used (including an analysis of 
its strengths and weaknesses). 

  Annexes: These must collate the technical details of the evaluation, and must 
include  the  Terms  of  Reference,  questionnaire  templates,  interview  guides, 
any additional tables or graphics, and references and sources. 

Final Report 

The Final Report follows in principle the same format as the draft Final Report. In 
addition  to  the  contents  described  above  (main  report  and  annexes),  it  should 
contain: 

  Executive summary: It sets out, in no more than 7 pages, a summary of the 
evaluation’s  main  conclusions,  the  main  evidence  supporting  them  and  the 
recommendations  arising  from  them.  Furthermore,  the  Executive  Summary 
should  be  translated  into  French  and  German  by  a  professional  translation 
agency, once it has been approved by the responsible body. 

  Summary statement: A ½ page summary of the main evaluation conclusions 

and recommendations. 

The Final Report must  take into account the results of the quality assessment of 
the  draft  Final  Report,  and  the  relevant  discussions  with  the  Steering  Group 
insofar as these do not interfere with the independence of the Contractor in respect 
of the conclusions they have reached and the recommendations made. 

The  contracting  authority  will  publish  the  Final  Report,  the  Executive Summary 
and the annexes on the World-Wide Web5. 

–  ORGANISATION, TIMETABLE AND BUDGET 

a. 

Organisation 

The  contract  will  be  managed  by  Unit  EAC/E2  of  the  European  Commission 
Directorate-General for Education and Culture. 

A  Steering  Group  will  be  involved  in  the  management  of  the  evaluation.  The 
responsibilities of the Steering Group will include: 

  providing the external evaluator with access to information; 

  supporting and monitoring the work of the external evaluator; 

5 

On the site http://ec.europa.eu/comm/dgs/education_culture/evalreports/index_en.htm. 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

21 

 
 
 
 
 
 
                                                           
  assessing  the  quality  of  the  reports  submitted  by  the  external  evaluator, 
while ensuring that the Contractor's independence is not compromised; 

b. 

Meetings 

It is expected that the contractor participate in four meetings in Brussels with the 
evaluation Steering Group. The evaluation team leader and other relevant experts 
must  participate  in  these  meetings.  For  these  meetings,  minutes  should  be 
drafted  by  the  contractor  within  5  working  days,  to  be  agreed  among  the 
participants and approved and signed by the chair person, who will be appointed 
from Unit EAC/R2. 

c. 

Timetable 

The indicative starting date is 6 November 2012. The contract will start after both 
parties have signed it. The period of execution of the contract is 9 months. 
The following outline work plan and indicative timetable are envisaged:   

 Deadline (from starting date)  Task 

Kick-off meeting 
End October 2012 / 
Beginning of November 2012 

First  meeting  between  Contractor  and  Steering 
Group in Brussels to clarify the work to be done 
and  allow  contractor  to  better  prepare  the 
inception report. 

Inception Report 
Mid-December 2012 

Interim Report 
Mid-March 2013 

Draft Final Report 
Mid-June 2013 

Final Report 
End July 2013 

d. 

Budget 

Contractor  prepares 
presents to Steering Group in Brussels. 

inception  report  and 

Desk  and  field  research  completed.  Contractor 
presents  interim  report  to  Steering  Group  in 
Brussels. 

Contractor  presents  a  draft  final  report, 
including  an  executive  summary,  to  Steering 
Group in Brussels. 

Taking account of the Commission’s comments 
contractor sends final report and summary to 
Steering Group in Brussels. 

The maximum budget for the evaluation of the two evaluation subjects, covering 
all the results to be achieved by the contractor as listed in sections 2 and 3 above, 
is EUR 175 000. 

22 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
 
 
–  REFERENCES 

a. 

Basic documents and Web references 

Decision  N°  1719/2006  of  the  European  Parliament  and  of  the  Council  of  15 
November 2006 establishing the 'Youth in Action' programme for the period 2007 
to 2013 

http://eur-
lex.europa.eu/LexUriServ/site/en/oj/2006/l_327/l_32720061124en00300044.pdf. 

Youth in Action interim evaluation (2011) by ECORYS Research and Consulting: 
http://ec.europa.eu/dgs/education_culture/evalreports/youth/2011/interimreport_e
n.pdf. 

The Commission Youth website (with comprehensive information on both youth 
policy and the Youth in Action programme) 

http://ec.europa.eu/youth/index_en.htm. 

The Commission Youth website, page on evidence-based youth policy. 

The EU Youth Strategy  
http://ec.europa.eu/youth/policy/eu-youth-strategy_en.htm 

http://ec.europa.eu/youth/youth-policies/doc1687_en.htm. 

Documents specific for the European Youth Forum 

European  Youth  Forum  website, 
http://www.youthforum.org/. 

also 

for  EYF's 

annual 

reports: 

Final external evaluation of the Community action programme to promote bodies 
active at European level in the field of youth (2004-2006) 
http://ec.europa.eu/dgs/education_culture/evalreports/index_en.htm. 

Documents specific for the EU-CoE youth partnership 

EU-CoE youth partnership web-portal. 

Evaluation  of  the  partnership  covenant  on  training  in  youth  work  between  the 
European Commission and the Council of Europe by Deloitte & Touche (2004) 
http://ec.europa.eu/dgs/education_culture/evalreports/index_en.htm. 

Ex-ante evaluation  study  of  the  research and  policy strand  of  the  Partnership  on 
Youth between the Council  of Europe  and the European Commission  by Maria-
Carmen Pantea (2008) (available upon request). 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

23 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b. 

Documents  and  information  to  be  provided  after  contract  signature 
(not exhaustive) 

Documents specific for the European Youth Forum 

Operating  grant  agreements,  interim  and  final  reports  for  the  years  being  dealt 
with. 

Documents specific for the Council of Europe 

Framework Partnership Agreements, Specific Grant Agreements, interim and final 
reports for the years being dealt with. 

–  REQUIREMENTS 

a. 

Methodology 

The  contractor  will  have  a  free  choice  as  to  the  methods  used  to  gather  and 
analyse information and for making the assessment, but must take account of the 
following: 

–  The evaluation must be based on recognised evaluation techniques; 

–  The choice and a detailed description of the methodology must form part 
of the offer submitted. There should be a clear link between the evaluation 
questions  addressed  and  the  corresponding  methodology  proposed.  The 
evaluation  questions  can  be  further  elaborated,  e.g.  by  providing 
operational sub-questions under each question;  

–  The Evaluator is invited to interview the organisations under evaluation as 

it deems necessary; 

–  Considerable  emphasis  should  be  placed  on  the  analysis  phase  of  the 
evaluation.  In addressing the evaluation questions,  quantitative indicators 
should be sought and used as far as possible. The contractor must support 
findings and recommendations by explaining the degree to which these are 
based  on  opinion,  analysis  and  objectively  verifiable  evidence.  Where 
opinion is the main source, the degree of consensus and the steps taken to 
test the opinion should be given; 

– 

It is not expected that all individual projects financed by the programme be 
assessed,  but  the  sample  of  projects  examined  should  be  drawn  up  in  a 
manner  suitable  for  each  evaluation  question  addressed,  and  should  be 
such  as  to  enable  the  evaluators  to  draw  general  conclusions  on  the 
actions. 

a. 

Resources 

The Contractor  shall ensure that experts are adequately supported and equipped. 
In  particular,  sufficient  administrative,  secretarial  and  interpreting  resources,  as 
well as junior experts, must be available to enable senior experts to concentrate on 
their core evaluation tasks. 

24 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
–  AWARD CRITERIA 

1.1  Quality criteria 

N° 

1. 

2. 

3. 
Total points 

Qualitative Award criteria 

Knowledge and understanding of the subject and its 
context, and of the purpose and requirements of the 
tasks to be performed. 
Relevance, feasibility and credibility of the approach 
proposed for the management of the work, including 
work plan and timetable, resource allocation, 
composition of the team of experts and quality 
assurance. 

Quality and relevance of the methodologies and tools 
proposed. 

Weighting 
(max. points) 
25 

25 

50 

100 

Offers  for  which  the  technical  quality  assessment  score  of  one  (or  more)  of  the 
quality criteria is less than 50% of the maximum points assigned to that specific 
criterion will not be considered for the price assessment and for the award of the 
contract. 

Offers  for  which  the  overall  technical  quality  assessment  score  is  less  than  65 
points  will  not  be  considered  for  the  price  assessment  and  for  the  award  of  the 
contract. 

If notified in the specific service request, the project leader and an additional staff 
member  presented  in  the  technical  offers  passing  the  thresholds  specified  above 
may  be  invited  for  an  interview.  The  interview  will  serve  to  complete  the 
appreciation  made  by  the  evaluation  committee.  This  interview  should  last  a 
maximum of two hours and will take place at the Commission premises. The first 
assessment will be weighted with 0,8 and the interview would then be rated with 
0,2. 

1.2 

Financial criteria 

Each  offer  will  be  assessed  in  terms  of  the  total  price  on  the  basis  of  the  unit 
prices set in the Framework Contract, broken down by categories of experts, and 
the reimbursable items. 

The lowest bid with technical sufficient score (i.e. at least 65 points) receives 100 
points. The others are awarded points by means of the following formula: 

Points = (price of the lowest bid / price of the bid in question) X 100. 

1.3 

Contract award 

The contract will be awarded to the bid offering the best value for money, taking 
into account the quality of services by weighing technical quality against price on 
a 70/30 basis. This is done by multiplying: 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

25 

 
 
 
 
 
 
 
 
 
 
 
 
 
- 

- 

the scores awarded for the technical quality by 0.70; 

the scores awarded for the financial bid by 0.30. 

The  technical  and  financial  scores  multiplied  by  the  weighting  factors  are  then 
added  together,  and  the  contract  is  awarded  to  the  tender  achieving  the  highest 
score. 

26 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
Annex 2  Methodology 

The figure below provides an overview of the methodology. 

Figure A2.1 Methodology overview 

The key tasks, outcomes and timescales for each of the work package are presented in table A2.1 
below. In the sub-sections that follow we describe the work packages (WPs) in more detail. 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

27 

Inception PhaseReporting PhaseWP1: Kick Off MeetingWP2: Quantitative Data CollectionWP3: Desk ResearchWP10: Focus GroupWP11: Final Analysis and ReportingWP4: Key Stakeholder InterviewsMain Research PhaseWP7: Online SurveyWP8: Interviews with Other Stakeholders in Youth SectorTiming:9 Novemberr –  Mid December 20121-1.5 monthsTiming:Mid March – End July 2013Within 9 monthsTiming:Mid December 2012 – Mid March 2013 Within 4.5 monthsWP9: Interim reportWP6: Inception ReportWP5: Reconstruction evaluation framework 
 
 
 
 
 
 
 
 
Table A2.1 Work packages, tasks, timetable 

Work Package 

Key tasks 

Inception phase 

WP 1: Kick-off 

  Understanding of clients expectations; 

meeting 

  Feedback on the approach, method and research tools; 

  Feedback on data availability; 

  Participation the meeting and drafting the minutes. 

WP2: Quantitative 

  Collection and analysis of monitoring data; 

data collection 

  Collection and analysis of web statistics. 

WP 3: Desk 

research 

  Review of European-level documents (policy documents, previous evaluations); 

  Review of documents relevant to each organisation (annual reports, operating 

grant agreements, work programmes, publications and information on the 

WP 4: Key 

stakeholder 

interviews 

WP 5: 

websites); 

  Wider literature on youth sector in Europe. 

 

 

Interviews with organisations subject to evaluation; 

Interviews with representatives of the DG EAC and Council of Europe. 

  Reconstruction intervention logic; 

Reconstruction of 

  Development of evaluation framework;  

the evaluation 

framework 

  Develop the methodology and prepare a more detailed work plan. 

WP 6: Inception 

 

Inception Report; 

report 

  Steering Group meeting; 

  Revised Inception report. 

Main Research Phase 

WP 7: Online survey  Online survey of youth organisations (member and partner organisations, national/ 

international youth organisations, European/ international youth networks, youth 

organisations not involved in activities of YFJ and/or EU-CoE youth partnership). 

WP 8: Interviews 

  60 interviews with wider stakeholders. 

with other 

stakeholders in 

youth sector 

WP 9: Interim report 

 

Interim Report; 

  Steering Group meeting; 

  Revised Interim report. 

Analysis and Reporting 

WP 10: Focus group  Half day focus group with EU-level stakeholders in Brussels to validate findings and 

WP 11: Final 

analysis and 

reporting 

suggest initial recommendations. 

  Additional interviews with DG EAC if needed; 

  Analysis of qualitative and quantitative data; 

  Draft final report; 

  Steering Group meeting; 

  Final report. 

28 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
Annex 3 

Literature 

"The impact of the Lisbon Treaty on the Council of Europe"- Parliamentary Assembly 

Recommendation 1982(2011) -Reply adopted by the Committee of Ministers on 28 march 2012 at 

the 1138th meeting of the Ministers' Deputies 

http://www.coe.int/t/der/docs/EU/CMASRec1982_EN.pdf. 

European Commission, An EU Strategy for Youth – Investing and Empowering A renewed open 

method of coordination to address youth challenges and opportunities, COM(2009) 200 Final, 

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2009:0200:FIN:EN:PDF. 

Council Resolution of 27 November 2009 on a renewed framework for European cooperation in the 

youth field (2010-2018) http://youth-partnership-eu.coe.int/youth-

partnership/documents/EKCYP/Youth_Policy/docs/YP_strategies/Policy/doc1648_en.pdf. 

DECISION No 1719/2006/EC OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 15 

November 2006 establishing the ‘Youth in Action’ programme for the period 2007 to 2013. 

Deloite&Touche Evaluation of the partnership covenant on training in youth work between the 

European Commission and the Council of Europe. 

Europe 2020 - A strategy for smart, sustainable and inclusive growth, COM (2010) 2020 Final 

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=COM:2010:2020:FIN:EN:PDF. 

European Commission, White Paper - A new Impetus for European Youth, COM(2001) 681 Final 

http://ec.europa.eu/youth/archive/whitepaper/download/whitepaper_en.pdf. 

European Commission, Europe 2020, A European strategy for smart, sustainable and inclusive 

growth, 2010. 

European Commission, Communication from the Commission to the European Parliament, the 

Council, the European Economic and Social Committee and the Committee of the Regions, 

ERASMUS FOR ALL: The EU Programme for Education, Training, Youth and Sport. 

European Commission, Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND 

OF THE COUNCIL establishing "ERASMUS FOR ALL" The Union Programme for Education, 

Training, Youth and Sport, 2011. 

European Youth Forum Operating Grant Agreement 2007 – 

European Youth Forum Operating Grant Agreement 2007 amendment- 

European Youth Forum Operating Grant Agreement 2008 – 

European Youth Forum Operating Grant Agreement 2008 - amendment- 

European Youth Forum Operating Grant Agreement 2009 – 

European Youth Forum Operating Grant Agreement 2009 amendment. 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

29 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
European Youth Forum Operating Grant Agreement 2010 – 

European Youth Forum Operating Grant Agreement 2010 amendment -  

European Youth Forum Operating Grant Agreement 2011 - 

European Youth Forum Operating Grant Agreement 2011 no1-- 

European Youth Forum Operating Grant Agreement 2012. 

European Youth Forum (2010), Statutes. 

European Youth Forum, Work Plan of the European Youth Forum 2007 – 2008. 

European Youth Forum, Work Plan of the European Youth Forum 2009 – 2010. 

European Youth Forum, Work Plan of the European Youth Forum 2011 – 2012. 

European Youth Forum, Work Plan 2013-2014, November 2012. 

European Youth Forum, Youth Forum Jeunesse, Internal financial guidelines, version 7.11 

November 2007. 

European Youth Forum/Youth Forum Jeunesse (2006), Strategic priorities of the European Youth 

Forum 2007-2012. 

European Youth Forum (2012) Strategic priorities 2013-2018, April 2012. 

European Youth Forum, Consultation with Member Organisations: Strategic Priorities of the 

European Youth Forum 2013 – 2018. 

European Youth Forum, Position Paper on structured dialogue, adopted by the Council of Members 

Brussels, 19-22 April 2012. 

European Youth Forum, Shadow Report on the Evaluation of the current Framework of European 

Cooperation in the Youth field, 2009. 

European Parliament, Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF 

THE COUNCIL establishing "ERASMUS FOR ALL" The Union Programme for Education, Training, 

Youth and Sport. 

Ministry of National Resources Hungary, Compendium of the structured dialogue with young people 

and youth organisations on youth employment during the Spanish, Belgian and Hungarian EU 

Presidencies, August 2011. 

The evaluation partnership, Final external evaluation of the Community action programme to 

promote bodies active at European level in the field of youth 2004-2006). 

Websites 

Joint Programmes between the Council of Europe and the European Union 

http://www.jp.coe.int/default.asp. 

30 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EU-CoE Youth Partnership 

http://youth-partnership-eu.coe.int/youth-partnership/. 

European Commission 

http://ec.europa.eu/index_en.htm. 

YFJ 

http://www.youthforum.org/#.  

http://issuu.com/yomag/docs/annual-report-2011-en.  

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

31 

 
 
 
 
 
 
 
 
 
 
 
Annex 4  Research tools: Questionnaire 

The questionnaire presented below was launched in English, French and German. 

General background 

1.  I am affiliated to: 

  a national youth council; 
  an international non-governmental youth organisation/institution; 
  a European non-governmental youth organisation/institution; 
  a national non-governmental youth organisation/institution; 
  a ministry or other governmental authority responsible for youth; 
  a research institution; 
  an organisation/institution linked to youth work; 
  a SALTO Resource Centre; 
  a National Agency of the Youth in Action programme; 
  an informal youth group; 
  other type of organisation/institution. 

2.  Which country are you based in?  

  Albania; 
  Andorra; 
  Armenia; 
  Austria; 
  Azerbaijan; 
  Belgium; 
  Bosnia and Herzegovina; 
  Bulgaria; 
  Croatia; 
  Cyprus; 
  Czech Republic; 
  Denmark; 
  Estonia; 
  Finland; 
  France; 
  Georgia; 
  Germany; 
  Greece; 
  Hungary; 
 
 
 
  Latvia; 
  Liechtenstein; 
  Lithuania; 
  Luxembourg; 
  Malta; 
  Republic of Moldova; 
  Monaco; 
  Montenegro; 
  Netherlands; 
  Norway; 
  Poland; 

Iceland; 
Ireland; 
Italy; 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

33 

 
 
 
 
 
 
 
  Portugal; 
  Romania; 
  Russian Federation; 
  San Marino; 
  Serbia; 
  Slovak Republic; 
  Slovenia; 
  Spain; 
  Sweden; 
  Switzerland; 
  The former Yugoslav Republic of Macedonia; 
  Turkey; 
  Ukraine; 
  United Kingdom; 
  Other (Please specify). 

3.  What is the profile of the young people your organisation supports? 

(Please tick all that are relevant) 
  Students (university or college); 
  School children; 
  Young people involved in non-formal learning activities; 
  Disadvantaged young people; 
  Disabled young people; 
  Young people from ethnic minority backgrounds; 
  Young people living in rural areas; 
  Young people living in cities; 
  Young people in general. 

4.  What type of activities or support could the European Youth Forum or the EU-CoE youth 

partnership offer, that would be useful to your youth organisation?  
(open question: please answer in English, if not possible, please answer only in French or 
German) 

5.  Do you belong to an organisation which is currently a member of the European Youth Forum? 

  Yes, a full member; 
  Yes, a candidate member; 
  Yes, an observer member; 
  No, not a member <routing>; 
  No, a member in the past. 

6.  Are you familiar with the activities of the European Youth Forum? 
  Yes <routing to all questions on European Youth Forum>; 
  No <if no: go to Q12 and than leave questionnaire>. 

7.  Do you participate in any of the activities of the European Youth Forum? 

  Yes; 
  No. 

<IF NO> 

8.  If not, why not? (more than one answer possible) 

  Never been invited; 
  Not interested; 
  Not relevant; 
  Not known to me/organisation; 

34 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
 
  No time available; 
  No budget available; 
  Don’t know; 
  Other, …. (please answer in English, if not possible, please answer only in French or 

German). 

9.  Are you familiar with the activities, tools and products of the EU-CoE youth partnership? 

  Yes, the Pool of European Youth Researchers; 
  Yes, the European Knowledge Centre for Youth Policy; 
  Yes, the Coyote magazine; 
  Yes, conferences, symposia or workshops on youth policy; 
  Yes, other activities of the EU-CoE youth partnership activities (please answer in English, 

if not possible, please answer only in French or German); 

  No, I am not familiar with any of the activities of the EU-CoE youth partnership < go to Q12 

and than leave questionnaire>. 

<routing: if answered yes, to all questions on EU-CoE youth partnership>. 

10. Have you participated or are you participating in any activities or any used tools and products 

organised by the EU-CoE youth partnership? 
  Yes; 
  No; 

<IF NO> 

11. If not, why not? (more than one answer possible) 

  Never been invited; 
  Not interested; 
  Not known to me/organisation; 
  Don’t know; 
  Other, …. (please answer in English, if not possible, please answer only in French or 

German). 

12. Are you involved in any other youth organisations/institutions at a European level? 

  Yes (please indicate which one(s); 
  No. 

(ROUTING: QUESTIONS FOR THE EUROPEAN YOUTH FORUM) 

13. Do you feel the European Youth Forum is advocating for the interest of youth organisations vis-

à-vis European institutions? 
  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know/not applicable. 

14. In your opinion, does the European Youth Forum advocate successfully the ideas and interests 

of young people? 
  Yes, to a great extent; 
  Yes, to some extent; 
  No, not at all; 
  Don’t know/not applicable. 

<ONLY FOR MEMBER ORGANISATIONS: routing Q3> 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

35 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
15. In general, to what extent do the activities of the European Youth Forum fit in with the needs of 

your organisation/institution? 
  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know/not applicable. 

16. Do the activities of the European Youth Forum fit in with the needs of young people in your 

country? 
  Yes, to a great extent; 
  Yes, to some extent; 
  No, not at all; 
  Don’t know/not applicable. 

17. What are the five most relevant activities [listed below] of the European Youth Forum for your 

organisation/institution?  

List of type of activities: 

-  Seminars; 

-  Training; 

-  Network meetings; 

-  Working groups; 

-  Debate/ discussion on youth policy; 

-  Development of sustainable consultation structures; 

-  Dissemination of information; 

-  Presentations; 

-  Production of policy papers; 

-  Development of youth policy tools; 

-  Lobbying; 

-  Advocating; 

-  Exchange of good practices; 

-  Other. 

18. How relevant are the following topics to your organisation/institution? 

-  Education; 

-  Youth work development; 

-  Participation and youth policy mainstreaming; 

-  Human right and social cohesion; 

-  Employment and social affairs: 

  Very relevant; 
  Relevant; 
  Slightly relevant; 
  Not at all relevant; 
  Don’t know/not applicable. 

19. How relevant are the following values to your organisation/institution? 

-  Participation; 

-  Empowerment of young people; 

-  Equality, gender equality and equal opportunities; 

- 

- 

Inclusiveness; 

Intercultural understanding; 

-  Democracy; 

-  Respect for human rights; 

36 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
-  Solidarity with all young people; 

-  Global responsibility; 

-  Diversity and mutual respect: 

  Very relevant; 
  Relevant; 
  Slightly relevant; 
  Not at all relevant; 
  Don’t know/not applicable. 

20. Why/Is your organisation involved in activities of the European Youth Forum? (more than one 

option possible) 
  Because it is beneficial to our organisation/institution; 
  Because it is beneficial to our staff; 
  Because it is beneficial to the young people we represent; 
  To be represented in the European youth policy discussions; 
  To take part in representative structures for young people; 
  To stay informed on youth policy; 
  To stay informed on opportunities in the national youth field; 
  To stay informed on opportunities in EU youth policy field; 
  To stay informed on youth work in general; 
  To stay informed on educational opportunities; 
  Other…; 
  My organisation is not involved. 

21. Are there any areas where your organisation/institution needs more support from the European 

Youth Forum? 
  Yes, namely (please answer in English, if not possible, please answer only in French or 

German); 

  No. 

22. In general, to what extent are you satisfied with the activities that are undertaken by the 

European Youth Forum? 
  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know/not applicable. 

23. Are the following activities effective for your organisation/institution? <matrix question> 

  Very effective; 
  Effective; 
  Slightly effective; 
  Not effective; 
  Don’t know/not applicable. 

[insert list of activities listed above] 

24. Are the following activities effective for the young people of your country?<matrix question>  

  Very effective; 
  Effective; 
  Slightly effective; 
  Not effective; 
  Don’t know/not applicable. 

[insert list of activities listed above] 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

37 

 
 
 
 
 
 
 
 
 
 
 
25. Do the activities undertaken by the European Youth Forum contribute to cooperation among: 

-  youth organisations in the same field; 

-  youth organisations in different fields; 

-  national youth organisations; 

- 

international youth organisations; 

-  national youth councils: 

  Yes, to a great extent; 
  Yes, to some extent; 
  No, not at all. 

26. To what extent do the activities of the European Youth Forum have an effect: 

-  on working practices in your organisation/institution? 

-  on working practices in your country? 

-  on decision makers in your country? 

-  on young people in your country? 

  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know/not applicable. 

27. Would any of these effects happen without the European Youth Forum? 

  Yes, to a great extent; 
  Yes, to some extent; 
  No, not at all. 

28. To your knowledge, has the European Youth Forum undertaken activities concerning: 

-  promotional actions for young people participating in democratic life; 

-  contribution to a new cooperation framework in the youth field established at the level of 

the EU; 

-  contribution to the development of youth policies, youth work and educational 

opportunities; 

- 

the achievement of discussions and reflections on youth in Europe and in other parts of 

the world and on the Community's action for young people: 
  Yes, enough activities were undertaken; 
  No, not enough activities were undertaken; 
  No activities were undertaken; 
  Don’t know/not applicable. 

29. Are you currently participating or have you participated in activities organised by the European 

Youth Forum in the past? 
  Yes; 
  No; 
  Don’t know/not applicable. 

<only for non members> 

30. Do you feel the activities of the European Youth Forum are open to non-member organisations: 

  Yes, the activities are open to non-member organisations; 
  No; 
  Don’t know. 

38 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
 
 
31. In your opinion, do the activities of the European Youth Forum contribute to: (obligatory to 

answer) 

-  Representing youth organisations vis-à-vis the EU; 

-  Coordinating the positions of its members vis-à-vis the EU; 

-  Relaying information on youth vis-à-vis the European Institutions; 

-  Relaying information form the EU to the national youth councils and non-governmental 

organisations; 

- 

Promoting and preparing the participation of young people in democratic life; 

-  Contributing to the new cooperation framework in the youth field established at the level of 

the EU; 

-  Contributing to the development of youth policies, youth work and educational 

opportunities, and to relaying information concerning young people and developing 

representative structures for young people throughout Europe; 

- 

Engaging in discussion and reflection on youth in Europe and in other parts of the world 

and on the Community's action for young people: 
  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know/not applicable. 

<only for member organisations, routing q5> 

32. Is the process to became a member: 

- 

- 

transparent; 

independent: 
  Yes, very; 
  Yes, to some extent; 
  No, not at all; 
  Don’t know. 

33. Do you receive information from the European Youth Forum concerning young people: 

  Yes, it is what we need; 
  Yes, but it is not what we need; 
  No; 
  Don’t know/not applicable; 

34. Does your organisation receive information from the European Youth Forum concerning 

developing representative structures for young people throughout Europe: 
  Yes, it is what we need; 
  Yes, but it is not what we need; 
  No we do not; 
  Don’t know/not applicable. 

In general… 

35. Do you have any general suggestions or recommendations? 

  No; 
  Yes, namely (please answer in English, if not possible, please answer only in French or 

German). 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

39 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
<ROUTING QUESTIONS FOR THE EU-COE YOUTH PARTNERSHIP> 

36. To what extent do you feel the EU-CoE youth partnership is present in the European youth 

policy discussion? 
  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know/not applicable. 

37. To what extent do the activities, tools and products of the EU-CoE youth partnership fit in with 

the needs of your organisation/institution? 
  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know/not applicable. 

38. To what extent do the activities, tools and products of the EU-CoE youth partnership fit in with 

the needs of young people in your country? 
  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know/not applicable. 

39. Are the following activities, tools and products of the EU-CoE youth partnership relevant to 

young people in your country?  
  Very relevant; 
  Relevant; 
  Slightly relevant; 
  Not at all relevant; 
  Don’t know/not applicable. 

List  

-  European Knowledge Centre for Youth Policy (EKCYP); 

-  Seminars, conferences, workshops, symposia; 

-  Pool of European Youth Researchers; 

-  Publications (including the Coyote magazine and books). 

40. Are the following activities, tools and products of the EU-CoE youth partnership relevant to your 

organisation/institution? 
  Very relevant; 
  Relevant; 
  Slightly relevant; 
  Not at all relevant; 
  Don’t know/not applicable. 

List 

-  European Knowledge Centre for Youth Policy (EKCYP); 

-  Seminars, conferences, workshops, symposia; 

-  Pool of European Youth Researchers; 

-  Publications (including the Coyote magazine and books). 

40 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41. Are the following topics relevant for your organisation/institution? 

-  Strengthening social inclusion of young people; 

- 

Increasing young people’s aware of democracy and human rights; 

-  Fostering Intercultural dialogue and youth work in a multicultural environment: 

  Very relevant; 
  Relevant; 
  Slightly relevant; 
  Not at all relevant; 
  Don’t know/not applicable. 

42. To your knowledge, has the EU-CoE youth partnership undertaken activities, and produced 

tools and products concerning: 

-  Promoting knowledge-based youth policy by developing and providing efficient tools; 

-  Supporting youth work and building capacity of youth workers at European level; 

-  Paying special attention to the regional cooperation (South East Europe, Southern 

Mediterranean, Central-Eastern Europe and Caucasus); 

-  Gathering, disseminating and providing information and publications on youth: 

  Yes, many were undertaken; 
  Yes, but not enough were undertaken; 
  Nothing was undertaken; 
  Don’t know/not applicable. 

43. Are the following priorities of the EU-CoE youth partnership relevant for your 

organisation/institution? 

-  Promoting knowledge-based youth policy by developing and providing efficient tools; 

-  Supporting youth work and building capacity of youth workers at European level; 

-  Paying special attention to regional cooperation (South East Europe, Southern 

Mediterranean, Central-Eastern Europe and Caucasus); 

-  Gathering, disseminating and providing information and publications on youth: 

  Very relevant; 
  Relevant; 
  Slightly relevant; 
  Not at all relevant; 
  Don’t know/not applicable. 

44. Why are you involved or do you use the activities, tools and products of the EU-CoE youth 

partnership? 
  Because it is beneficial to our organisation/institution; 
  Because it is beneficial to our staff; 
  Because it is beneficial to the young people we represent; 
  To be represented in the European youth policy discussions; 
  To take part in representative structures for young people; 
  To stay informed about youth policy; 
  To stay informed about the opportunities in the national youth field; 
  To stay informed about opportunities in EU youth policy field; 
  To stay informed about youth work in general; 
  To stay informed about educational opportunities. 

45. In which areas does your organisation/institution need more support from the EU-CoE youth 

partnership? (please answer in English, if not possible, please answer only in French or 

German) 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

41 

 
 
 
 
 
 
 
 
 
 
46. In general, to what extent are you satisfied with the activities undertaken and tools and products 

that are produced by the EU-CoE youth partnership? 
  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know; 
  Not applicable. 

47. How effective are the following activities, tools and products of the EU-CoE youth partnership 

for your organisation/institution?  
  Very effective; 
  Effective; 
  Slightly effective; 
  Not effective; 
  Don’t know/not applicable; 

<list> 

48. How effective are the following activities, tools and products for the young people of your 

country: 
  Very effective; 
  Effective; 
  Slightly effective; 
  Not effective; 
  Don’t know/not applicable. 

<list> 

49. Do the activities, tools and products of the EU-CoE youth partnership contribute to cooperation 

among: 

-  youth organisations in the same field; 

-  youth organisations in different fields; 

-  national youth organisations; 

- 

international youth organisations; 

-  youth researchers; 

-  youth workers; 

-  government representatives/civil servants: 

  Yes, to a great extent; 
  Yes, to some extent; 
  No, not at all. 

50. To what extent do the activities, tools and products of the EU-CoE youth partnership have an 

effect: 

-  on working practices in your organisation/institution? 

-  on working practices in your country? 

-  on decision makers in your country? 

-  on young people in your country? 

-  on youth workers in your country? 

-  on youth researchers in your country? 

  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know/not applicable. 

42 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
 
51. Would any of these effects happen without the EU-CoE youth partnership? 

  Yes, to a great extent; 
  Yes, to some extent; 
  No, not at all. 

52. In your opinion, do the activities of the EU-CoE youth partnership contribute to: 

-  Promoting evidence-based youth policy; 

-  Supporting youth work and capacity building; 

-  Activities with a regional cooperation (South East Europe, Southern Mediterranean, 

Central-Eastern Europe and Caucasus); 

- 

Information and dissemination: 
  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know/not applicable. 

53. In your opinion, do the activities of the EU-CoE youth partnership contribute to:  

-  Creating better opportunities and improve access for young people; 

-  Promoting their active citizenship; 

-  Fostering solidarity; 

-  Provide more and equal opportunities for young people in education and in the labour 

market;  

-  To foster mutual understanding between young people in different countries. 

…in the fields of social inclusion, democracy, human rights, citizenship and intercultural 

dialogue: 

  To a great extent; 
  To some extent; 
  Not at all; 
  Don’t know/not applicable. 

54. How visible is the international work (work with non-member countries of Council of Europe) of 

the EU-CoE youth partnership? 
  Very visible; 
  Somewhat visible; 
  Not at all visible; 
  Don’t know/not applicable. 

55. Were you aware that the EU-CoE youth partnership organises events promoting evidence-

based youth policies in the following areas? 

-  South-East Europe; 

-  Eastern Europe and the Caucasus; 

-  Southern Mediterranean; 

-  Africa Europe Youth Cooperation: 

  Yes; 
  No; 

56. Are you aware that the EU-CoE youth partnership is a project of the Council of Europe and the 

European Commission? 
  Yes, I knew both were involved; 
  No, I didn’t know either were involved; 
 
 

I only knew the Council of Europe was involved; 
I only knew the European Commission was involved. 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

43 

 
 
 
 
 
 
 
 
 
 
 
57. Do you feel that the EU-CoE youth partnership represents: 

  Mainly the interests of the Council of Europe; 
  Mainly the interests of the European Commission; 
  Both the interests of the European Commission and the Council of Europe 
  No opinion. 

58. Do you feel the EU-CoE youth partnership is organised in such a way that it enables the youth 

partnership to produce the desired results, which is to: 

provide a framework for joint development of cooperation and a coherent strategy in the field of 

youth in Europe, with involvement of young people, youth workers, youth researchers and youth 

policy makers: 
  Yes; 
  No, because …. (please answer in English, if not possible, please answer only in French 

or German). 

59. Do you receive information from the EU-CoE youth partnership concerning young people: 

  Yes it is what we need; 
  Yes, but it is not what we need; 
  No we do not; 
  Don’t know/not applicable. 

60. How effective have the dissemination activities of the EU-CoE youth partnership been? 

  Very effective; 
  Effective; 
  Slightly effective; 
  Not effective; 
  Don’t know/not applicable. 

In general… 

61. Do you have any general suggestions or recommendations? 

  No; 
  Yes, namely (please answer in English, if not possible, please answer only in French or 

German). 

44 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
Annex 5  Research tool: Cost effectiveness 
analysis 

The Council Financial Regulation defines efficiency as: “…the best relationship between resources 
employed and results achieved“6 whilst the European Commission defines efficiency as: “…the 
extent [to which] the desired effects are achieved at reasonable cost”.7 For most EAC programmes, 
the question is whether a measure has produced (or will produce) the highest level of effects for the 

(fixed) budget available. This concept can be termed “yield maximisation” and implies the use of 

ratios that relate effects to costs. This however requires a choice to be made whether to analyse 

the relationship between inputs and outputs or between inputs and higher-level outcomes (results 

and/or impacts). This will determine the data to be gathered (and what is available) as well as the 

type of conclusions that will be drawn.  

According to the Ecorys CEA Report8, there are three basic levels of efficiency analysis. For the 
evaluation of the YFJ, Level 0 is most appropriate since no data is available to the evaluators on 

costs in relation to individual activities or outcomes.  

The same CEA report developed four possible approaches to the analysis of efficiency of EAC 

programmes:  

Types of intervention for which each approach is applicable 

Approach 0 involves describing and providing an opinion on some efficiency-related aspects of an intervention. 

It is suitable for interventions that are not spending programmes, such as flagship, emblematic and symbolic 

actions (e.g. events, prizes) or information, dissemination and programme support actions. 

Approach 1 involves identifying and, where possible, benchmarking unit costs, which can then be compared 

with other interventions. It is suitable for interventions that produce tangible end products, where a unit cost can 

easily be identified and thus compared with other projects/approaches, e.g. literary translations (based on 

number or readers), distribution of films (based on number of viewers), theatre productions (based on audience 

numbers); likely to include sub-programmes that offer support for production, distribution and promotion of 

products or services. 

The main feature of Approach 2 is to use beneficiary feedback to produce evidence on how projects have 

impacted on capabilities. It is suitable for interventions that develop capabilities of individuals or organisations, 

e.g. mobility placements, training courses, capacity building measures, etc. Comparisons made on strength of 

improvement in capabilities relative to alternative approaches and taking relative cost into account. 

Approach 3 involves analysing clusters of case study projects that produce tangible outcomes which can be 

quantified or at least assigned rated against a scale, which might then allow comparison against other projects 

or alternative approaches. It is suitable for interventions that produce multiple and diverse effects, but which are 

nonetheless tangible (perhaps even quantifiable) and that relate well to the lower-level objectives of 

programmes (i.e. specific or operational); likely to include sub-programmes that support multilateral cooperation 

projects and partnerships, organisations and networks. 

6  

7  

8  

Council Regulation (EC, Euratom) No 1605/2002 of 25 June 2002 on the Financial Regulation applicable to the general 
budget of the European Communities. 
Evaluating EU Activities: A practical guide for the Commission services; November 2003, European Commission DG 
Budget. 
Study on cost-effectiveness of education and culture spending programmes: Final Report to the Directorate-General for 
Education and Culture, January 2013. 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

45 

 
 
 
 
 
 
 
 
                                                           
Types of intervention for which each approach is applicable 

Approach 4 involves analysing clusters of case study projects and attempting to draw some general conclusions 

about the efficiency of processes and the value added compared to: i) the previous situation; ii) discontinuation. 

It is suitable for interventions that produce multiple and diverse effects that are very intangible and that relate 

mostly only to the higher-level objectives of programmes (i.e. general) if at all and where comparisons with other 

approaches are difficult; likely to include sub-programmes that support multilateral cooperation projects and 

partnerships, organisations and networks. 

In the case of the YFJ, and with reference to the logic model, the application of any of those 

methods is difficult:  
 

In terms of Approach 1, data is not available to the evaluator on costs and in terms of a number 

of specific activities; 

  Given the emphasis of the YFJ on lobbying activities and policy contributions and as there is no 
data available on costs on individual events to the evaluator, Approach 2 can not be used to 

examine effects on participants and beneficiaries in the light of the costs involved and 

alternatives for delivering similar outputs; 

  Approach 3 can be used to explore particular activities corresponding to specific or operational 

objectives e.g. events, databases, newsletters/website and again look at costs and potential for 

improvements. However, again no data is available on costs and in most cases there is no data 

on quantified outputs; 

  Approach 4 can assist in capturing higher-level effects associated with the YFJ activities (e.g. 
raising issues such as intercultural dialogue and diversity, democracy and human rights and 

social inclusion). In the case of the YFJ it can be stated that the effects are mostly intangible. 

However, seen the status of the organisation it has access to important decision makers, and 

has influence on them, which is difficult to achieve through other channels.  

Seen the lack of cost data on individual activities to the evaluators, costs efficiency is assessed on 

the basis of impression, internal rules for the coverage of costs, the use of volunteers by the YFJ 

and the assessment by stakeholders.  

46 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
 
 
Annex 6  Survey respondents and results 

The survey targeted a broad group of stakeholders: national governments; national youth councils; 

national agencies of the Youth in Action programme; research institutions/ organisations linked to 

youth work; SALTO- youth resource centres; informal youth groups; and national, European and 

international non-governmental youth organisations. Among the contacted non-governmental youth 

organisations and national youth councils are the members of the YFJ, as well as non-members 

(243 respondents). 

The geographical coverage appears satisfactory since the respondents were based in almost all 

EU-27 countries as well as several countries from neighbouring regions of the EU like Russia, 

Albania and Croatia.  

The table below shows the organisations to which the respondents are affiliated. The table shows 

that more than 37% of the respondents were affiliated to national (13%), European (11%) or 

international (13%) non-governmental youth organisations. Another 14% were affiliated to an 

organisation or institution linked to youth work. Only 2% were working for a SALTO-YOUTH 

Resource Centre.  

Organisations to which respondents are affiliated 

I am affiliated to 

A national youth council 

An international non-governmental youth organisation/institution 

A European non-governmental youth organisation/institution 

A national non-governmental youth organisation/institution 

A ministry or other governmental authority responsible for youth 

A research institution 

An organisation/institution linked to youth work 

A SALTO Resource Centre 

A National Agency of the Youth in Action programme 

An informal youth group 

Other, type of organisation/institution 

Total 

Source: Web survey of youth organisations (2013). Base size 342 organisations. 

Count 

13 

32 

27 

32 

37 

21 

27 

5 

25 

5 

19 

% 

5% 

13% 

11% 

13% 

15% 

9% 

11% 

2% 

10% 

2% 

8% 

243 

100,0% 

The category ‘other’ covers many different types of organisations including non-profit organisations, 

a prime minister chancellery, associations (with several different target groups), the African Union 

Commission, local governmental organisations, universities, formal networks and Eurodesk national 

contact points, as well as freelancers in the youth work field and professional service providers to 

youth supporting institutions. 

The profile of young people supported by the responding organisations is presented in the table, 

overleaf. Almost 71% of the respondents indicated that their organisation supports young people in 

general. More than 64% of the respondents support young people involved in non-formal learning 

activities.  

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

47 

 
 
 
 
 
 
 
 
 
 
 
Profile of the young people your organisation supports (N=342)?* 

Young people in general 

Young people involved in non-formal learning activities 

Students (university or college) 

Disadvantaged young people 

Young people from ethnic minority backgrounds 

Young people living in cities 

Young people living in rural areas 

School children 

Disabled young people 

Count 

172 

156 

124 

116 

99 

93 

88 

69 

62 

% 

71% 

64% 

51% 

48% 

41% 

38% 

36% 

28% 

26% 

Source: Web survey of youth organisations (2013). Base size 342 organisations. 
**The total does not add up to 100% as this is a multiple response question.  

Other key features of the survey sample include: 
  Over 70% of the respondents are not current or previous members of the YFJ. Almost 21% of 

respondents are full -members. Some 3% of the respondents are candidate members, and an 

equal percentage are observer members. More than 3% were members of the YFJ in the past; 
  Almost 46% of respondents are either participating or have previously participated in activities 

of the YFJ; 

  Most respondents (43%) who had never participated in activities of the YFJ, indicated that they 
have never been invited to do so. Another frequently mentioned reason for respondents not to 

participate in YFJ activities is the absence of funding (28%). Only 3% of respondents stated that 

they are not interested in participating. Some 11% of the respondents did not find it relevant for 

their organisation to participate. In addition, some organisations (17%) provided a wide range of 

other reasons for not participating, e.g. the organisation is not a target group for the activities of 

YFJ but they do cooperate with YFJ and benefit from the information YFJ provides. Youth 

organisations, SALTOs, National Agencies, research institutes and other organisations all 

ranked ‘never been invited’ as the most important reason they have not participated. Ministries 

on the other hand most frequently selected the answer category ‘not relevant’ (33%); 

  Next to being involved with the YFJ and/ or the Youth Partnership, some 58% of respondents 

were involved in other youth organisations/institutions at European level. Organisations with 

whom they are involved most often were:: The Council of Europe, the European Commission, 

the Youth in Action programme, SALTOs, Eurodesks, ERYICA (European youth information 

and counselling agency), several types of fora e.g. Forum of European Roma Young People, 

and a number of European networks and European youth organisations.  

Only the respondents that were familiar with the activities completed the remainder of the survey for 

the YFJ. 

48 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
  
 
 
 
 
Annex 7  List of interviewees 

Interviews key stakeholders 

European Commission 

Astrid Brey 

Artur Payer 

Head of Sector EAC/D.1.001 EC 

Programme manager – EU policies- Jeunesse en Action (Actions 3 et 4 et Autres) 

EAC/D.1 EC 

Floor van Houdt 

Deputy Head of Unit – Youth Policy EAC/D.1 EC 

Finn Denstad 

Formes Policy officer – Youth Policy EAC/D.1: overall cooperation and evidence-

based youth policy (one of the three main strands of the youth partnership EC 

Claire Herman 

Policy officer – Cooperation and International programmes EAC/C.4 

Fabiene Metayer 

Policy officer – Education and Creativity and Culture EAC/D.1: 

Sergej Koperdak 

Former Head of Unit for Youth Policy in DG EAC (Since 2013 an advisor on new 

Pascal Lejeune  

Head of Unit – Youth Policy EAC/D.1EC 

technologies and social media, DG EAC) 

Jorgen Hansen 

Policy officer – Dialogue with Youth and interinstitutional relations EAC/D.1 

YFJ 

Guiseppe Porcaro 

Secretary General YFJ, interviews in November and May 

David Ferreira 

Administrative and financial director YFJ, interviews in November and May 

Alix Masson 

Head of Policy and Advocacy department YFJ 

Miroslaw Krzanik  

Board Member YFJ 

Other stakeholders 

Howard Williamson 

Professor of European Youth Policy, University of South Wales 

Klavidja Cernilogar 

Former Head of Policy and Advocacy department YFJ 

Interviews wider group of stakeholders 

Viviana Galli 

Thomas Maes 

Gerard Tosserams 

Catharina Covolo 

Ben Althoff 

Jerry den Haan 

Rebekka Kemler Muller 

Jochen Rummenhöller 

Ricardo Ibarra Roca 

Juliana Bir 

Mirte van Loenen 

Obessu 

ECOSY 

YMCA 

EYCE  

European Youth 4 Media 

Nationale Jeugd Raad 

ENGSO (sports and youth) 

Deutscher Bundesjugendring/ German national youth council 

CJE 

ETUC 

Nationale Jeugd Raad 

Luis Alvarado Martinez 

AEGEE. European Students Forum 

Matilda Flemming 

Mathilde Bauwin 

Jens Nilsen 

Vito La Fata 

Miroslow Krzanik 

Angelos Mavridis 

United Network of Young Peacebuilders 

Djamel Benin-Mathilde Bauwin 

Danish Youth Councils 

CESIE - Centro Studi ed Iniziative Europeo 

AEGEE (formerly delegate to YFJ), now Board member YFJ 

Hellenic National Youth Council  

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

49 

 
 
 
 
 
 
 
 
 
 
Thomas Leys 

Barbara Bello 

Manfred Zendtner 

Ivett Karvalitis 

Paola Bortini 

Dr Howard Williamson 

Andreia Henriques  

Milos Ondirasik 

Anna Dobrovolskay 

Simona Mursec 

Loreta Senkutė  

Jukka Tahvanainen 

Vladimir Kharchenko 

James Edleston 

Madara Apsalone 

Fanny Charmey 

Marko Boko 

Jacob Fuchs 

International Federation of Liberal Youth 

Trainer and Researcher  

Jugendkultur – Research Institute 

Eurodesk Hungary 

Pame Ambro (Freelance trainer in the field of youth work and non 

formal education) 

Professor of European Youth Policy, University of South Wales 

Programme Manager - Youth Co-operation  

North-South Centre of the Council of Europe 

IUVENTA (Slovakia) 

Advisory Council for Youth in Russia  

National Youth Council of Slovenia 

Lithuanian Youth Council (LiJOT) 

Finnish Youth-Cooperation Allianssi 

National Youth Council Russia 

British Youth Council 

ENL Estonian National Youth Council  

Switzerland National Youth Council 

Croatia National Youth Council 

Danish Agency for International Education  

Sandra (Aleksandra) Kamilova  

Estonia National Youth Council 

Lyubomir Todorov 

Adina Calalfateanu 

Ilaria Esposito 

Lea Renoldner 

Mihai-Paul Floran 

Jeremy Barnett 

Tony Geudens 

Sonja Mitter Škulj  

Kristina Pernits 

Lihong Huang 

Corinne Grassi 

Isabelle Camilleri 

Anastasia Filini 

Claire Conlon  

Des Burke 

Olle Svahn 

Anya Ruhland 

Seija Astala 

Council of Europe – Member of the Advisory Council for Youth 

Council of Europe - Member of the Advisory Council for Youth 

Council of Europe - Member of the Advisory Council for Youth 

Council of Europe - Member of the Advisory Council for Youth 

Council of Europe - Member of the Advisory Council for Youth 

SALTO Cultural Diversity. NOTE  

SALTO Inclusion Resource Centre 

SALTO South East Europe Resource Centre 

SALTO Training and Cooperation Resource Centre. 

Research institute (Norwegian) 

Anna Lindt Foundation  

National Youth Council, Malta 

Neagenia – General Secretariat for Youth  

Eurodesk France 

Leargans 

National Council of Swedish Youth Organisations, LSU  

Eurodesk coordinator Brussels  

Chair person, Council of Europe Steering Committee for Youth 

CDEJ 

50 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

 
 
 
 
 
 
 
 
Participants focus group 

Astrid Brey 

Guiseppe Porcaro 

Viviana Galli 

Marta Medlinska 

Jens Thuesen 

Artur Payer 

Karin Lopatta-Loibl 

Jorgen Hansen 

DG EAC/D.1  

Secretary General YFJ, interviews in November and May 

Obessu 

EU-CoE youth partnership 

Ministry of Education Denmark 

DG EAC/D.1 

DG EAC/D.1 

DG EAC/D.1 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

51 

 
 
 
 
 
 
 
 
 
 
 
 
Annex 8  Financial information-available 
upon request 

Please contact EAC-UNITE-D1@ec.europa.eu for further information requests. 

Evaluation of the European Commission-European Youth Forum Operating Grant Agreements 2007-2011/12 

53 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
P.O. Box 4175 

3006 AD Rotterdam 

The Netherlands 

Watermanweg 44 

3067 GG Rotterdam 

The Netherlands 

T +31 (0)10 453 88 00 

F +31 (0)10 453 07 68 

E netherlands@ecorys.com 

W www.ecorys.nl 

Sound analysis, inspiring ideas 

BELGIUM – BULGARIA – CROATIA - HUNGARY – INDIA – THE NETHERLANDS – POLAND – RUSSIAN FEDERATION – SPAIN – TURKEY – UNITED KINGDOM