Support to SME initiative evaluation in the Member States : final report.
Date: 2021-04-16
Institutions: Directorate-General for Regional and Urban Policy
Subjects: EU Member State, EU financing, European Structural and Investment Funds, aid to undertakings, fund (EU), loan, report, small and medium-sized enterprises

Support to SME initiative 
evaluation in the Member 
States 
Contract no. 2019CE16BAT136 

Final Report 

Written by t33 srl 
March – 2021 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EUROPEAN COMMISSION 

Directorate-General for Regional and Urban Policy 
REGIO B.2 - Evaluation and European Semester 

Contact: Carlo Amati 

E-mail: REGIO-EVAL@ec.europa.eu 

European Commission 
B-1049 Brussels 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
EUROPEAN COMMISSION 

Support to SME initiative 
evaluation in the Member 
States 

Contract no. 2019CE16BAT136 

Final Report 

Directorate-General for Regional and Urban Policy 

2021 

EN 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Europe Direct is a service to help you find answers  
to your questions about the European Union. 

Freephone number (*): 

00 800 6 7 8 9 10 11 

(*) 

The information given is free, as are most calls (though some operators, phone 

boxes or hotels may charge you). 

Manuscript completed in March 2021 

The European Commission is not liable for any consequence stemming from the reuse of this publication.  

Luxembourg: Publications Office of the European Union, 2021 

ISBN 978-92-76-36159-6 
doi: 10.2776/975444 

© European Union, 2021 
Reuse is authorised provided the source is acknowledged.  
The reuse policy of European Commission documents is regulated by Decision 2011/833/EU (OJ L 330, 
14.12.2011, p. 39). 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

Foreword 

Initially  presented  in  June  2013  in  a  Commission's  and  European  Investment  Bank's 
joint report to the  European Council1, the SME  Initiative (SMEi) has been established 
to  complement  and  utilise  synergies  between  existing  SME  support  programmes  at 
national and EU level. SMEi blends EU funds from COSME and Horizon 2020 together 
with  European  Structural  and  Investment  Funds  (ESIF)  in  cooperation  with  the 
European  Investment Bank Group to enable additional lending to small and medium-
sized enterprises (SMEs). 

Spain  was  the  first  EU  country  to  put  SMEI  into  practice  on  26  January  2015  with 
Malta  launching  later  that  year,  then  Bulgaria,  Finland,  Romania  and  Italy  in  2016. 
Only Italy decided to implement SMEi securitisation instrument, while the other SMEi 
participating Member States are implementing uncapped portfolio guarantees. 

Following the Omnibus Regulation2 entering into force, the provisions on evaluation of 
the  Common  Provision  Regulation  are  also  applicable  to  SMEi  Operational 
Programmes.  Thus,  managing  authorities  must  ensure  that  at  least  an  evaluation  is 
carried  out  to  assess  effectiveness,  efficiency  and  impact  of  support  from  the  ESI 
Funds.  A  survey  conducted  among  SMEi  Member  States  by  Malta,  in  agreement  with 
the Commission, showed that the majority of participating Member States were not in 
favour  of  a  joint  evaluation  of  SMEi.  However,  there  was  general  support  for  a 
common methodological approach allowing for country-specific adjustments. 

Against  this  background,  this  service  contract  provides  a  common  methodological 
approach  to  support  SMEi  evaluation  in  the  Member  States3.  A  draft  approach  was 
developed based  on literature review and interviews with SMEi managing authorities. 
This was reviewed based on feedback from a technical seminar organised on 3 March 
2020  in  Brussels,  with  representatives  of  SMEi  managing  authorities,  European 
Commission officials and two external discussants. 

This  is  the  ‘Final  Report’  prepared  under  the  service  contract.  It  provides  the 
common methodological approach, including evaluation purpose and objectives, 
tasks and methods. The approach: 

  Ensures robust and comparable SMEi evaluations, in line with the Common 
Provision  Regulation  and  according  to  international  evaluation  best  practice. 
Where  the  method  requires  for  more  sophisticated  methods  to  be  adopted,  it 
does  so  as  these  are  deemed  necessary  to  help  Member  States  assess  the 

1  European  Commission  and  European  Investment  Bank  (2013),  Increasing  lending  to  the 
economy:  implementing  the  EIB  capital  increase  and  joint  Commission-EIB  initiatives,  Joint 
Commission-EIB report to the European Council. 

2 Regulation (EU, Euratom) 2018/1046 of the European Parliament and of the Council of 18 July 
2018 on the financial rules applicable to the general budget of the Union, amending Regulations 
(EU)  No  1296/2013,  (EU)  No  1301/2013,  (EU)  No  1303/2013,  (EU)  No  1304/2013,  (EU)  No 
1309/2013,  (EU)  No  1316/2013,  (EU)  No  223/2014,  (EU)  No  283/2014,  and  Decision  No 
541/2014/EU and repealing Regulation (EU, Euratom) No 966/2012. 

3  The  service  contract  covers  mainly  ERDF.  Other  sources/programmes  are  considered  only  to 
the  extent  that  they  contribute  to  the  impact  of  the  operations,  be  it  only  from  the  financial 
point of view or with a more complex mechanism. 

 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

opportunity  costs  of  supporting  SMEi  rather  than  other  instruments  or 
objectives4.  

  Considers  the  different  SMEi  sizes  and  options  across  the  Member  States 
and the possible constraints on the resources available for the evaluation. 
On this basis, it offers a modular design of tasks and methods. A core layer of 
the  evaluation  has  been  identified,  based  on  higher  priority  success  criteria 
and  related  evaluation  questions.  In  addition,  given  the  evaluation  questions, 
use  of  methods  is  also  modular  in  that  the  conditions  under  which  methods 
can complement each other or be alternative options are clarified. 

While this document is formulated so as to provide useful input to the evaluations of 
SMEi in the Member States, it only covers some of the necessary sections of a possible 
terms of reference5. 

4  On  the  need  for  more  sophisticated  methods  to  assess  the  wider  economic  impact  of  public 
programmes and control for unconnected factors that impact on any outcomes, see Green, F. J. 
(2009), Assessing the impact of policy interventions: the influence of evaluation methodology. 

5  For  example,  it  does  not  cover  timing  and  deliverables,  the  required  competencies  or 
management arrangements. 

 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

TABLE OF CONTENTS 

ACRONYMS ........................................................................................... 8 

1 GENERAL BACKGROUND ...................................................................... 9 

2 PURPOSE AND OBJECTIVES ............................................................... 14 

3 TASKS ............................................................................................. 19 

Task 1 Mapping and enterprise data survey ...................................... 19 
Task 2 Assessment of effectiveness ................................................. 21 
Task 3 Assessment of impact .......................................................... 26 
Task 4 Assessment of efficiency ...................................................... 29 

4 METHODS ........................................................................................ 30 

4.1 Desk analysis .......................................................................... 36 
4.2 SME survey ............................................................................. 47 
4.3 EIF interview ........................................................................... 53 
4.4 Financial intermediary interviews ............................................... 55 
4.5 Counterfactual analysis............................................................. 57 

7 

 
 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

ACRONYMS 

AECM 
AIR 
CATI 
CAWI 
COSME 

DID 
EIB 
EIF 
ESIF 
EU 
MFI 
NACE 

NUTS 
OP 
PSM 
RDD 
SAFE 
SME 
SMEi 

Association Européenne du Cautionnement 
Annual Implementation Report 
Computer-assisted telephone interviewing 
Computer-assisted web interviewing 
The EU programme for the Competitiveness of Small and Medium-Sized 
Enterprises  
Difference-in-difference  
European Investment Bank 
European Investment Fund 
European Structural and Investment Funds 
European Union 
Monetary Financial Institutions 
Nomenclature statistique des Activités économiques dans la Communauté 
Européenne  
Nomenclature d'Unités Territoriales Statistiques 
Operational Programme 
Propensity Score Matching  
Regression discontinuity design  
Survey on the Access to Finance for Enterprises in the EU area 
Small and medium-sized enterprises 
SME initiative 

8 

 
 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

1 GENERAL BACKGROUND 

Reconstructing  the  intervention  logic  of  SMEi  can  be  very  helpful  in  identifying  the 
evaluation  questions  as  it  can  point  to  relationships  or  assumptions  that  the  evaluation 
might  wish  to  investigate  further6.  Therefore,  as  a  first  step  in  designing  a  common 
methodology that can help the Member States  to prepare their own  Terms of Reference 
to  conduct  their  evaluations,  a  draft  simplified  intervention  logic  of  SMEi  has  been 
developed.  This  can  be  adapted  and  further  developed  by  the  Member  States  to  fit  the 
specific context and Operational Programme (OP) as needed. 

As illustrated in Figure 1, SMEi draft simplified intervention logic considers7: 

  Actual  inputs  as  costs  associated  with  SMEi.  These  include  financial  allocations  to 
SMEi  programmes  net  of  any  repaid  resources.  Thus,  these  cover  management 
costs and fees, any calls for guarantees net of recoveries and risk cover fees8.  

 

In  addition  to  programme  resources,  SMEi  uses  resources  from  EU  centrally 
managed  programmes  and  the  European  Investment  Bank  (EIB)  group,  which 
increases capacity to meet final recipient needs. EIB group’s resources benefit from 
International Financial Institution status, including an AAA credit rating and 0% risk 
weight. The risk management activity calculates risk, pricing, capital allocation and 
tranche structuring. Programme and  EIB  resources cover the most  and least  risky 
tranches respectively. 

  The European Investment Fund (EIF) implements SMEi support, including selecting 
financial  intermediaries.  It  is  remunerated  through  management  costs  and  fees, 
comprising  an 
interest.  Financial 
intermediaries  benefit  from  risk-sharing  and  reduced  regulatory  capital  charges9 
based  on  SMEi  guarantee.  A  higher  (lower)  guarantee  fee  can  be  expected  to  be 
paid by the financial intermediaries the higher (lower) the risk cover fees. In turn, 
this reduces (increases) the benefit that can be passed to final recipients. 

fee  to  promote  alignment  of 

incentive 

  Under operational agreements signed with the EIF, financial intermediaries commit 
to build new credit portfolios during a pre-defined inclusion period. Key features of 
the portfolio and how this differs from standard credit policy are submitted with the 
financial intermediary expression of interest. Financial intermediaries keep a share 
of  risk  of  each  transaction  in  their  own  books,  based  on  the  guarantee  rate. 
Moreover, penalties due in case agreed targets are not achieved promote alignment 
of interest. 

6 See European Commission (2017), Better regulation Toolbox. 

7  The  consultants  reviewed  evaluations  and  research  covering  similar  interventions,  as  well  as 
legislation  and  Commission’s  communications  addressing  SMEi.  SMEi  Operational  Programme 
documents as well as other SMEi official documents in the Member States, especially EIF calls for 
expression  of  interest,  were  also  reviewed.  On  this  basis,  a  draft  simplified  intervention  logic  for 
SMEi was developed.  

8  Additional  costs  may  also  be  considered,  which  are  less  important  according  to  the  literature. 
These include the opportunity cost of capital employed in the scheme or the cost of administering 
the  scheme  borne  by  the  managing  authorities.  See  for  instance  London  Economics  (2017), 
Economic impact evaluation of the Enterprise Finance Guarantee scheme. 

9  Risk  weighting  method  implies  that  initial  capital  absorption  for  SME  exposures  is  affected  by 
whether  banks  use  the  standardized  or  the  internal  rating-based  approach  and  by  whether  loans 
are classified as corporate or retail. 

9 

 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

Figure 1 SMEi draft simplified intervention logic 

Source: t33 

  Financial  intermediaries  are  required  to:  (a)  report  as  part  of  their  expression  of 
interest on financing origination and performance data with respect to SMEs and a 
proxy  of  the  guaranteed  portfolio;  (b)  manage  and  service  the  portfolio  in 
accordance with their standard credit policies applicable to their SME loan book; (c) 
report on key features of selected final recipients e.g. probability of default, internal 
scoring/rating,  and  transactions,  e.g.  interest  rate  margin,  loss  given  default, 
collateralisation  rate.  This  is  intended  to  manage  the  risk  that  financial 
intermediaries  include  in  their  guaranteed  portfolios  enterprises  that  are  not 
financially viable, as risk sharing with public resources may lead to phenomena of 
moral hazard10; or enterprises that would have been financed even in the absence 
public  support  i.e.,  the  intermediaries  do  not  change  their  financing  decision,  so 
called  deadweight.  Moreover,  whereas  eligibility  and  risk  assessment  is  delegated 
to  financial  intermediaries,  based  on  agreed  criteria,  the  fund  manager  should 
replace any irregular transactions with eligible ones before closure of the OP. 

  Guarantee  agreements  include  a  methodology  to  pass  the  full  financial  advantage 
on to supported SMEs, based on reduced interest rate. The EIF relies on the good 
faith  of  financial  intermediaries  when  reporting  back  on  the  financial  advantage 

10  As noted above, SMEi  guarantee agreements imply that credit risk of a fraction of the value  of 
each  individual  loan  remains  with  the  lender.  This  encourages  the  lender  to  carefully  screen  and 
monitor the loans covered by the guarantee scheme avoiding excessive risk-shifting. 

10 

LeverageContribution (impact)INPUTSACTIVITIESOUTPUTSRESULTSEC, EIB, EIF and national authorities agree on risk sharing and product terms and conditionsEIF implements SMEi support, including selection of financial intermediariesFinancial intermediaries provide finance to SMEs backed by SMEi guaranteesImproved access to financeIncreased turnoverOther factorsLeverageContribution (impact)ESIF and national resourcesManagement costs and feesLossesGuarantee feeLosses from defaultsLossesRisk cover feeManagement costs and feesRisk cover fees+--Increased added valueIncreased employment 
 
 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

passed  on  to  SMEs11.  In  addition,  the  total  cost  of  finance  may  also  include  other 
fees e.g., arrangement fee.  

  Based on SMEi resources, credit might be provided to viable enterprises that would 
otherwise  not  be  financed  (see  Box  1  for  the  causes  that  generate  a  failure  in 
obtaining  financing,  based  on  SMEi  ex-ante  assessment  assumptions)12.  Financing 
conditions  for  supported  enterprises  can  also  be  improved,  including  increased 
access  to  finance  (in  case  of  partial  rationing),  longer-term  bank  debt,  lower 
financing costs13. 

  SMEs use new finance14 for investments in tangible and intangible assets as well as 
working capital. Based on intended change of each SMEi OP, this leads to increased 
employment, turnover or added value. However, other factors also drive the actual 
result towards or away from the intended change. Economic benefits also consider 
any  changes  in  the  survival  probability  of  enterprises  and  the  extent  to  which 
improvement in supported SME performance was at the expense of competitors. 

Box 1 SME failures in obtaining financing: SMEi ex-ante approach 

SMEi  aims  to  address  market  failures  by  facilitating  the  provision  of  credit  to  financially 
viable enterprises that could not access that credit otherwise. These are enterprises that: 
(a) have been refused a bank loan; (b) have turned down a bank loan, due to the credit 
conditions; (c) have been discouraged from applying for a bank loan. 

The chart below illustrates discouraged enterprises as a share of total financially viable15 
enterprises  in  each  of  SMEi  participating  Member  States  in  201616,  except  Malta.  It 
should  be  noted  that  the  figures  refer  to  enterprise  reported  behaviour  in  the  past  six 
months.  For  Italy  and  Spain  only17,  the  share  also  shows  the  share  of  enterprises  that 
applied  for  a  loan  but  either  were  rejected  or  refused  because  of  conditions.  While  the 
overall  proportion  of  financially  viable  credit  rationed  enterprises  in  the  two  Member 

11  Insofar  as  the  EIF  does  not  make  an  own  assessment  of  the  probability  of  default  of  final 
recipients.   

12 This is the key rationale of financial additionality as identified in SMEi ex-ante assessment. 

13  These  represent  additional  elements  of  financial  additionality,  as  generally  assessed  in  the 
economic literature. See for example D’Ignazio, A., Menon, C. (2012), Evaluating credit guarantees 
for SMEs: evidence from Italy. 

14  SMEi  ex-ante  assessment  also  identified  a  catalytic  effect,  under  which  loans  triggered  by  the 
initiative would lead to subsequent additional external financing obtainable by the final recipients. 
This is not considered in the draft simplified intervention logic as such effect was not referenced in 
the literature reviewed by the consultants. 

15  These  enterprises  reported  positive  turnover  growth  in  the  last  six  months  according  to  the 
‘upper boundary’ used to calculate the funding gap in SMEi ex-ante assessment. 

16  SAFE,  September  to  October  2016,  wave  15.  For  more  on  SAFE  methodology,  see  European 
Central Bank (2017), Survey on the access to finance of enterprises  – Methodological information 
on the survey and user guide for anonymised micro dataset, pp. 7-13. The calculation is based on 
data in fi-compass (2018), Financial gap in the EU agricultural sector. 

17 More replies are available to SAFE question Q7ba ‘If you applied and tried to negotiate for a bank 
loan over the past 6 months, what was the outcome?’ for these Member States.  

11 

 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

States is similar, the data indicates a more important role for discouraged enterprises in 
Spain and rejected18 and refused loan applications in Italy. 

Given  the  different  types  of  credit  rationing,  public  guarantees  can  encourage  banks  to 
finance  enterprises  they  would  not  consider  otherwise,  by  reducing  credit  risk,  capital 
absorption19 and alleviating deleveraging20. 

Public guarantees can also encourage more applications from enterprises that would not 
apply despite the need or make enterprises more likely to accept a bank offer they would 
otherwise refuse. 

4,5%

4,0%

3,5%

3,0%

2,5%

2,0%

1,5%

1,0%

0,5%

0,0%

Bulgaria

Finland

Italy

Romania

Spain

Applied but refused because costs too high

Applied but loan rejected

Under  SMEi,  benefits  from  the  subsidised  guarantee  provided  to  financial intermediaries 
are  passed  to  enterprises  by  applying  a  credit  spread  reduction.  This  is  ensured  by 
applying  a  guarantee  fee  to  the  part  of  each  loan  covered  by  the  guarantee,  instead  of 
the  financial  intermediary  standard  credit  risk  premium.  Financial  intermediaries  can 
propose  additional  improvements  on  standard  conditions  for  SMEs,  including  less 
collateral.  On  this  basis,  financial  intermediaries  provide  the  EIF  with  the  expected 
composition and characteristics of the portfolio they intend to build, including the level of 
collateral. 

Source: t33 

18  This  refers  to  enterprises  where  the  loan  application  was  totally  rejected  and  excludes 
enterprises with partial rejections. 

19 As concerns the extent to which guarantees are important to beneficiary banks for risk transfer 
or as instruments of capital management, see Chatzouz, M., Gereben, Á., Lang, F., and Torfs, W. 
(2017),  Credit  guarantee  schemes  for  SME  lending  in  Western  Europe,  EIB  Working  Papers 
2017/02.  This  paper  focuses  on  credit  guarantee  schemes  rather  than  publicly  supported 
guarantees. 

20 For a paper investigating the deleveraging process since the onset of the financial crisis and how 
this affected the provision of credit, see for example Mosk, T. and  Ongena, S. (2013), The impact 
of banking sector deleveraging on investment in the European Union, published as chapter in EIB 
(2013 Investment and Investment Finance in Europe.  

12 

 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

While  this  draft  intervention  logic  is  in  general  also  applicable  to  SMEi  securitisation, 
some key differences need to be considered21: 

  SMEi  securitisation  supports  existing  portfolios  of  SME  and  Small  Mid-Caps  loans, 
leases  or  alternative  debt  finance  agreements  originated  by  selected  financial 
intermediaries.  Priority  is  given  to  transactions  that  include  bank  loans  backed  by 
guarantees from mutual guarantee institutions. Replenishment of existing portfolios 
might be allowed subject to due diligence. 

 

In  addition  to  capital  relief  and  loss  protection  from  synthetic  transactions,  SMEi 
securitisation  can  also  provide  funding  via  a  cash  purchase  of  securitisation 
tranches. Certain tranches within SMEi securitisation transactions are priced below 
market, providing an advantage to financial intermediaries. 

In  exchange  for  receiving  SMEi  support,  financial  intermediaries  generate  an  additional 
portfolio  of  eligible  final  recipient  transactions.  The  EIF  agrees  with  the  financial 
intermediaries  on  a  methodology  to  pass  on  the  full  benefit  of  SMEi  support  to  those 
SMEs  financed  via  the  additional  portfolio.  This  implies  that  the  benefit  is  passed  on  to 
SMEs by reducing, for  each  exposure in the additional portfolio, the  standard credit risk 
premium financial intermediaries would normally charge. 

21  Public  support  to  SME  loan  securitisation  is  often  intended  to  stimulate  market  development. 
However, review of available documents did not identify this as an explicit objective of SMEi Italy. 
This is possibly due to Italy having a relatively  more  mature  SME  loan  securitisation  market. See 
European  Investment  Bank’s  Operations  Evaluation  Division  (2017),  Evaluation  of  the  EIF’s  SME 
Securitisation Activities, 2004-2015. 

13 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

2 PURPOSE AND OBJECTIVES 

SMEi  evaluation  should  address  success  criteria  that  measure  performance,  efficiency, 
effectiveness  and  impact,  based  on  intervention  logic  of  each  SMEi  OP.  Success  criteria 
have been developed based on review of research and evaluations. These are illustrated 
in the  following  tables,  together  with  corresponding  rationale,  evaluation  questions,  and 
tasks22.  Given  possible  constraints  on  the  resources  available  for  the  evaluation,  a 
modular structure has been designed, where success criteria with lower priority have 
been  identified  and  shown  on  dark  grey  background  in  the  tables.  As  presented  in 
the  tables,  the  success  criteria  aim  to  both  disentangle  the  effects  of  SMEi  from  other 
factors  and  to  understand  its  functioning23.  Further  details  on  judgement  criteria, 
indicators, tasks and methods addressing individual evaluation questions are provided in 
sections 3 and 4. 

22 The criteria assume that evaluation of SMEi for any financial resources contributed from COSME 

and Horizon 2020 rest with the European Commission. 

23 See European Commission (2014), The programming period 2014-2020 – Guidance document on 
monitoring  and  evaluation  of  the  European  Cohesion  Fund  and  the  European  Regional 
Development Fund. 

14 

 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

Table 1 SMEi success criteria - Effectiveness 

1 Effectiveness 

Success 
criteria 

Rationale 

Evaluation questions 

1.1 
Achievement of 
expected 
outputs/ 
results 

This assesses the extent that SMEi could achieve 
policy objectives under the relevant investment 
priority. 

EQ1.1 To what extent has SMEi achieved the expected 
output and results, based on programme intervention 
logic? 

Task(s) 

1,2 

1.2 
Appropriate 
target group 
and territorial 
balance 

Academic research shows that the financial 
environment for enterprises differs within 
countries and across target groups. This criterion 
assesses the extent that SMEi could reach the 
territories and target groups most in need. 

1.3 Suitability 
of 
implementation 
structure 

This assesses whether appropriate procedures, 
mechanisms and infrastructure were in place so 
the: (a) target group(s) is reached; (b) interests 
are aligned; (c) benefit from public support is 
passed to supported enterprises; (d) required 
reporting is prepared. 

1.4 Adequate 
implementation 
capacity 

This clarifies the extent that national and local 
institutional capacity was adequate and could 
benefit from EU-level entrusted entities with their 
know-how of guarantees and securitisation 
design. 

EQ1.2 What are the characteristics of SMEi recipients 
and how do they compare to (different groups of) 
potential recipients? 

1,2 

EQ1.3 How do SMEi implementation mechanisms 
promote the desired change? 

EQ1.4 What was the contribution of the EIF know-how 
to ensuring adequate implementation capacity? 

2 

2 

15 

 
 Support to SME Initiative evaluation in the Member States - Final Report 

1 Effectiveness 

Success 
criteria 

1.5 Outreach/ 
relevance 

Rationale 

Evaluation questions 

Task(s) 

This assesses the extent that SMEi could meet 
demand from credit rationed viable enterprises, 
addressing market failure. It considers timing 
and changing market conditions, as well as any 
overlapping or complementarities with other 
public support. 

EQ1.5.1 How has SMEI intervention rationale changed 
from conception to implementation? 

1,2 

EQ1.5.2 What mechanisms were established to adapt 
SMEi support to changing market conditions? 

EQ1.5.3 How does SMEI support fit with similar 
instruments financed with Cohesion policy or national 
resources? 

Lower priority criteria shown on dark grey background 

Table 2 SMEi success criteria - Impact 

2 Impact 

Success criteria  Rationale 

Evaluation questions 

2.1 Financial 
additionality 

This establishes whether enterprises would have 
received a loan from their bank without SMEi 
support. Financial additionality is an important 
issue in the rationale for SMEi which should not 
replace commercial lending (deadweight). 

EQ2.1 To what extent and at what conditions would 
SMEi actual recipients have been financed without SMEi 
support? 

Task(s) 

3 

16 

 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

2 Impact 

Success criteria  Rationale 

Evaluation questions 

2.2 Economic 
additionality 
(Enterprise 
performance) 

This compares performance between SMEi 
supported enterprises and other enterprises. It 
assesses whether a causal link between SMEi 
intervention and longer-term achievements can 
be established, after considering other factors. 
Key variables include enterprise investment, 
sales, exports, job creation and productivity 
(GVA). It also addresses sustainability over time. 

2.3 Project 
additionality24 

This assesses whether enterprises would have 
proceeded with their project without SMEi 
support. It also assesses whether the project 
would have proceeded with the same timing and 
scale. 

2.4 Economic 
benefits 

This estimates economic activity created through 
SMEi support, after accounting for displacement 
i.e., the degree to which improvements in 
recipient enterprise performance resulted in an 
increase in market share at the expense of 
competitors. Ultimately, removing a finance 
constraint should enhance the productivity of 
supported SMEi. 

Lower priority criteria shown on dark grey background 

EQ2.2 How does performance of SMEi recipients 
compare with potential recipients? 

EQ2.3 To what extent would actual recipients of SMEi 
have gone ahead with their project without SMEi 
support? 

EQ2.4 What are the net benefits for the economy, 
once net benefits for SMEi actual recipients are 
aggregated and considering the extent to which the 
growth of SMEi actual recipients displaced other 
enterprises? 

Task(s) 

3 

3 

3 

24 This criterion only applies where SMEs used support for investments in tangible and intangible assets. 

17 

 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

Table 3 SMEi success criteria - Efficiency 

3 Efficiency 

Success criteria  Rationale 

Evaluation questions 

3.1 Cost-benefit  This compares SMEi ‘additional’ benefits and 

costs (see above for more details on economic 
benefits). Where possible, it also assesses 
whether SMEi benefits were at a higher or lower 
cost compared with initial expectations and 
similar financial instruments. 

EQ3.1 How do SMEi economic benefits and costs 
compare to each other? 

4 

3.2 Risk 
optimisation 

This assesses whether SMEi design was suitable 
to the different risk-bearing capacities of risk 
takers and enabled maximum leverage. 

EQ3.2 To what extent was risk allocation appropriate 
considering risk taker capacity for risk? 

4 

Lower priority criteria shown on dark grey background 

18 

 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

3 TASKS 

Based  on  successful  examples  of  evaluations  and  research  on  similar  interventions,  this 
section illustrates the tasks required to address the success criteria and evaluate SMEi in 
the participating Member States. 

Task 1 Mapping and enterprise data survey 

This  task  includes  two  key  components  of  the  methodology  core  layer.  These  are  the 
SMEi mapping and a survey of enterprise micro data. It also includes an overview of 
national financial markets and evolution of SMEi ex-ante assessment key assumptions as 
a further possible activity.  

With  regard  to  the  SMEi  mapping,  the  evaluator  provides  the  main  elements  of  the 
SMEi including:  

 

 

 

 

supported investment priorities and specific objectives; 

type of financial instrument; 

financial intermediaries implementing the initiative; 

signature date of the funding agreement and operational agreements;  

  ESIF  and  programme  amounts  committed  and  paid 

to  SMEi, 

financial 

intermediaries; 

  a description of the main events, including any changes to the funding agreement 

any top-up support 

  ESIF, programme amounts and total financing disbursed to final recipients; 

  a  summary  analysis  of  the  data  reported  to  the  EIF  at  the  level  of  the  individual 

transactions and supported enterprises25; 

  output  indicators  to  which  the  SMEi  contributes,  including  target  and  achieved 

values; 

  management costs and fees; 

  amount attributable to ESIF support paid back to SMEi. 

In addition, the evaluator26 carries out a survey of enterprise micro data available in 
the  Member  State27.  This  identifies  data  sources  can  provide  data  to  control  group 

25  If  not  part  of  EIF  operational  reporting,  this  data  could  be  made  available  to  the  managing 
authorities as part of the evaluation. See Article 16 or Article 18 of the Commission Implementing 
Decision  of  11  September  2014  on  the  model  of  funding  agreement  for  the  contribution  of  the 
European  Regional  Development  Fund  and  the  European  Agricultural  Fund  for  Rural  Development 
to  joint  uncapped  guarantee  and  securitisation  financial  instruments  in  favour  of  small  and 
medium-sized enterprises. 

26  Alternatively,  the  survey  of  enterprise  micro  data  can  be  carried  out  by  the  evaluator  in  their 
technical offer or the managing authority, when preparing the terms of reference of the evaluation. 

19 

 
 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

comparison  of  SMEi  potential  recipients  and  complement  EIF  data  on  SMEi  supported 
enterprises  whenever  needed.  The  survey  also  clarifies  the  feasibility  and  level  of 
detail  that  can  be  delivered  by  each  method  as  well  as  the  specific  approaches 
required. 

As  concerns  administrative  or  commercial  datasets  providing  enterprise  micro  data, 
the data survey should consider: 

  Number of SMEs included in the dataset. Comparison with statistics on the overall 
number of SMEs in each Member State provides the extent of market coverage. 

  Depth of  information available. This should clarify what SME characteristics and 
financial  and  economic  indicator  data  is  available  for  different  types  of  enterprise. 
Extent  of  available  information  can  vary  considerably  within  Member  States, 
depending on reporting obligations of SMEs having different legal status or size. 

  Time period covered by the data, including before and after the intervention and 

available information about enterprises which exited the market. 

The data survey also covers feasibility of matching across data sources. In particular 
where  micro  data  reported  to  the  EIF  includes  an  enterprise  identifier  -  such  as  trade 
register  or  tax  number  -  that  can  be  used  for  automated  matching  with  other  data 
sources.  If  instead  EIF  implementation  data  does  not  include  an  identifier  but  basic 
enterprise information – i.e., enterprise name, address and postal code – additional time 
should be allowed to ensure matching between data sources as part of the evaluation (if 
administrative or commercial datasets are used). 

Where an SME survey is considered as an option to gather enterprise micro data that is 
not covered by EIF reported information, the evaluator clarifies whether a desired sample 
size  can  be  achieved  for  supported  SMEs28,  given  typical  response  rates  of  voluntary 
enterprise surveys.  

As additional possible activity under this task, the evaluator may prepare an overview of 
the  evolution  national  financial  markets.  This  would  provide  background  and  context  to 
assess SMEi effectiveness as well as it could contribute to addressing specific evaluation 
questions  as  detailed  under  task  2.  The  evaluator  may  also  be  required  to  provide  an 
analysis  of  how  SMEi  ex-ante  assessment  key  assumptions  evolved  during  the  relevant 
period.  

More details on the methods under this task are provided in section 4. 

27  Under  this  service  contract,  a  preliminary  analysis  of  SMEi  available  data  was  carried  out  to 
assess  whether  the  evaluation  questions  were  likely  to  be  answerable  -  as  recommended  by  the 
European  Commission  (2008),  Evalsed:  The  Resource  for  the  Evaluation  of  Socio-Economic 
Development Guide on evaluability of evaluation questions.  The preliminary analysis clarified that 
gathering of additional enterprise micro data than reported to the EIF is necessary to answer some 
of the evaluation questions. 

28  In  the  Member  States  participating  to  SMEi,  these  can  be  from  just  some  hundreds  to  tens  of 
thousands.  

20 

 
  
 
 
                                                                                                                                                         
 Support to SME Initiative evaluation in the Member States - Final Report 

Task 2 Assessment of effectiveness 

Evaluation  questions,  judgement  criteria  and  indicators,  as  well  as  methods  under  this 
task  are  given  in  Table  4.  All  criteria  are  important  to  assess  public  support  aimed  at 
improving  access  to  finance,  as  identified  in  the  economic  literature.  However,  given 
possible constraints on the resources available for the evaluation, success criteria and 
related  evaluation  questions  and  judgement  criteria  with  lower  priority  have 
been  identified.  These  are  shown  on  dark  grey  background  in  the  table.  In  addition, 
even  under  success  criteria  having  a  higher  priority,  it  might  not  be  feasible  to  address 
those  judgement  criteria  that  can  only  be  addressed  through  an  SME  survey, 
where this is not allowed by the evaluation budget. These are also shown  on dark grey 
background in the table. 

Most judgement criteria can be addressed through desk analysis and EIF and financial 
intermediary interviews. An SME survey could gather data about basic characteristics 
of  SMEi  actual  and  potential  recipients,  as  well  as  more  detailed  enterprise  information 
including use of other support and external finance. More details on how an SME survey 
and  other  methods  can  contribute  to  answer  the  evaluation  questions  are  provided  in 
section 4. 

21 

 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

Table 4 Evaluation matrix – Effectiveness 

1.1 Achievement of expected output and results 
EQ1.1 To what extent has SMEi achieved the expected output and results, based 
on programme intervention logic? 
Judgement criteria 
SMEi support received 

Indicators 
Number of final recipient transactions 

Method 
Desk analysis 

Amount of finance received through 
SMEi support 

Progress with achieving the 
expected output and 
results 

Comparison of output and results 
achieved values with indicator 
targets 

Desk analysis 

1.2 Appropriate target group and territorial balance 
EQ1.2 What are the characteristics of SMEi recipients and how do they compare 
to objectives and (different groups of) potential recipients? 
Judgement criteria 
Characteristics of SMEi 
actual recipients compared 
to initial objectives, 
potential recipients 

Method 
Desk analysis 

Indicators 
Legal status 

SME survey 

Size29, if a micro, small or medium-
sized enterprise based on 
Commission Recommendation 
2003/361/EC 

EIF interview 

Financial 
intermediary 
interviews 

Age class i.e., based on years since 
establishment30. 

SME status, if active or inactive (if 
administrative data is used) to 
indicate sustainability of effects from 
support 

Business sector, based on NACE 
taxonomy 

1-year probability of default31 

SMEi recipient ownership 
characteristics compared to 
potential recipients 

Characteristics of the owner, 
including sex32 and belonging to a 
minority group32  

SME survey 

Recent change in ownership33 i.e., 
before applying for support 

29 Literature shows that micro enterprises are more likely to be credit rationed. 

30 Literature highlights that banks are usually less prone to lend to younger enterprises. 

31  For default definition,  see Regulation (EU)  No  575/2013 of the  European  Parliament and of  the 
Council of 26 June 2013 on prudential requirements for credit institutions and investment firms and 
amending Regulation (EU) No 648/2012. 

32 Some literature shows that women and minority groups are more likely to need public support to 
access finance. 

33 Literature shows that changes in senior personnel may be perceived as an increased risk of less 
entrepreneurial capability. 

22 

 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

1.2 Appropriate target group and territorial balance 
EQ1.2 What are the characteristics of SMEi recipients and how do they compare 
to objectives and (different groups of) potential recipients? 
Judgement criteria 
Localisation characteristics 
of SMEi actual and 
potential recipients 

Indicators 
Region (Eurostat NUTS2) 

Method 
Desk analysis 

County (NUTS3) 

EIF interview 

Financial 
intermediary 
interviews 

Municipality (LAU2)  

For NUTS3 regions, it is possible to 
add other indicators34: 

Development level – GDP per capita 
(in categories)  

Urbanisation – Predominantly urban, 
intermediate or predominantly rural35 

Accessibility – Classification by 
various taxonomy developed at EU 
level, e.g., ESPON or Eurostat 

Use of external finance by 
SMEi actual and potential 
recipients 
Other public support 
received by SMEi actual 
and potential recipients 
i.e., grant or financial 
instrument  

Enterprise accessed external finance 
or only used internal finance 

SME survey 

Other grant support received 

SME survey 

Other financial support received 

Lower priority criteria are shown on dark grey background 

34  These  indicators  provide information  on  factors  which  could  affect  SME  performance  as  well  as 
access to finance. Literature highlights that companies from deprived zones are usually more likely 
to require public support to access finance than enterprises from more developed territories. 

35  Eurostat  defines  predominantly  urban  NUTS3  regions  as  having  more  than  80  %  of  the 
population living in urban  clusters. An urban  cluster  is a cluster of  contiguous grid cells of 1 km² 
(including diagonals) with a population density of at least 300 inhabitants per km² and a minimum 
population of 5 000 inhabitant. Intermediate NUTS3 regions have more than 50 % and up to 80 % 
of the population living in urban clusters. Predominantly rural NUTS3 regions have at least 50 % of 
the population living in rural grid cells. 

23 

 
 
 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

1.3 Suitability of implementation structure 
EQ1.3 How do SMEi implementation mechanisms promote the desired change? 
Judgement criteria 
SMEi is attractive to 
financial intermediaries 

Indicators 
Rationale behind financial 
intermediaries participation to SMEi 

Method 
Financial 
intermediary 
interviews 

EIF interview 

Desk analysis 

SME survey 

EIF interview  

Desk analysis 

SME survey 

EIF interview 

Desk analysis 

EIF interview 

Implementation 
mechanisms allow 
targeting of viable 
enterprises unable to 
access finance  

Implementation 
mechanisms ensure that 
the benefit from public 
support is passed to final 
recipients 

Implementation 
mechanisms incentivise 
fund manager and financial 
intermediaries performance  

Key parameters of support, including 
guarantee rate and guarantee fee 

Process and factors considered for 
their establishment 

Key parameters of support, including 
guarantee rate and guarantee fee 

Process and factors considered for 
their establishment  

Key parameters of support, including 
guarantee rate and guarantee fee 

Process and factors considered for 
their establishment 

Key parameters of support, including 
management costs and fees, 
penalties for underperformance and 
reallocation of resources among 
financial intermediaries 

Process and factors considered for 
their establishment  

Lower priority criteria are shown on dark grey background 

1.4 Adequate implementation capacity 
EQ1.4  What  was  the  contribution  of  the  EIF  know-how  to  ensuring  adequate 
implementation capacity? 
Judgement criteria 
EIF know-how helped 
ensure adequate 
implementation capacity 

Indicators 
Mechanisms ensuring the transfer of 
capacity to national and local 
institutions 

Method 
EIF interview  

Financial 
intermediary 
interviews 

Lower priority criteria are shown on dark grey background  

24 

 
 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

1.5 Outreach/relevance 
EQ1.5.1  How  has  SMEI  intervention  rationale  changed  from  conception  to 
implementation? 
Judgement criteria 
Extent rationale changed 
from SMEi conception to 
implementation 

Indicators 
Timeline of SMEi establishment and 
implementation 

Method 
Desk analysis 

EIF interview 

Indicators of evolution of national 
financial markets 

Implications for rationale of 
intervention 

Financial 
intermediary 
interviews 

Indicators 
Mechanisms to adapt to market 
changes and extent these were used 

EQ1.5.2 What mechanisms were established to adapt SMEi support to changing 
market conditions? 
Judgement criteria 
Extent SMEi could adapt to 
changed conditions and any 
challenges 
EQ1.5.3  How  does  SMEI  support  fit  with  similar  instruments  financed  with 
Cohesion policy or national resources?  
Judgement criteria 
Extent of overlap or 
synergies with other public 
support  

Indicators 
Comparison between SMEi features 
and main public interventions with 
similar objectives and target groups  

Method 
Desk analysis 

Method 
EIF interview 

EIF Interview 

Lower priority criteria are shown on dark grey background 

Financial 
intermediary 
interviews 

25 

 
 
 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

Task 3 Assessment of impact  

Evaluation  questions,  judgement  criteria  and  indicators,  as  well  as  methods  under  this 
task  are  given  in  Table  5.  All  criteria  are  important  to  assess  public  support  aimed  at 
improving  access  to  finance,  as  identified  in  the  economic  literature.  However,  given 
possible constraints on the resources available for the evaluation, success criteria and 
related  evaluation  questions  and  judgement  criteria  with  lower  priority  have 
been  identified.  These  are  shown  on  dark  grey  background  in  the  table.  In  addition, 
even  under  success  criteria  having  a  higher  priority,  it  might  not  be  feasible  to  address 
those  judgement  criteria  that  can  only  be  addressed  through  an  SME  survey, 
where this is not allowed by the evaluation budget. These are also shown  on dark grey 
background in the table. 

In  terms  of  methods,  enterprise  micro  data  for  impact  assessment  comes  either  from 
administrative  registers  /  commercial  datasets  or  SME  survey,  or  both.  Data  analysis  is 
statistical / econometric or based on a counterfactual approach whenever suitable, as 
this  is  the  most  common  standard  for  impact  evaluation  of  guarantee  support.  In 
particular,  judgement  criteria  related  to  additionality  –  from  EQ2.1  to  EQ2.3  –  can  be 
addressed  by  counterfactual  analysis  or  control  group  statistical  comparison  based  on 
SME  survey,  as  further  detailed  in  section  4.  Economic  additionality  should  be  assessed 
with  reference  to  each  SMEi  OP  intended  outcomes.  Moreover,  project  additionality  can 
only  be  assessed  where  SMEs  used  support  for  investments  in  tangible  and  intangible 
assets.  Proxies  based  on  SME  survey  are  required  to  estimate  net  benefits  for  the 
economy.  More  details  on  how  the  individual  methods  can  contribute  to  answering  the 
evaluation questions are provided in section 4.  

Table 5 Evaluation matrix – Impact 

2.1 Financial additionality 
EQ2.1 To what extent and at what conditions would SMEi actual recipients have 
been financed without SMEi support? 
Indicators 
Judgement criteria 
Financial indicators at the time and 
Different capacity of SMEi 
after receiving SMEi support e.g., 
actual and potential 
value of liabilities (credits and 
recipients to access finance 
debts), creditworthiness36 

Method 
Analysis of micro 
data 

Counterfactual 
analysis  

Ability to get a loan without SMEi 

SME survey 

Ability to meet demands from 
creditors 

Reasons offered by bank for taking 
out SMEi supported finance 

Reasons why unconstrained 
enterprises chose SMEi 

Value and type of collateral 
available 

36 Mariani M., Mealli F., Pirani E. (2013), Gli effetti delle garanzie pubbliche al credito: due misure a 
confronto, IRPET, provides an example of this indicator based on Altman studies. 

26 

 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

Perceived key benefits in terms of 
access to finance for borrowers 
under SMEi 

Financial 
intermediary 
interviews 

2.2 Economic additionality (Enterprise performance) 
EQ2.2  How  does  performance  of  SMEi  recipients  compare  with  potential 
recipients? 
Judgement criteria 
Economic performance 
difference between SMEi 
actual and potential 
recipients  

Indicators 
Variation in financial and investment 
performance indicators e.g., value 
of fixed and current assets, 
liabilities, creditworthiness 

Method 
Analysis of micro 
data 

Counterfactual 
analysis  

Variation in economic performance 
indicators e.g., turnover, gross 
value added, profit, staff costs and 
total costs 

SME self-reported performance 
indicators 

SME survey 

2.3 Project additionality 
EQ2.3  To  what  extent  would  actual  recipients  of  SMEi  have  gone  ahead  with 
their investment without SMEi support? 
Judgement criteria 
Different investment 
capacity of SMEi actual and 
potential recipients 

Indicators 
Variation in investment indicators 
e.g., value of fixed and current 
assets 

Method 
Analysis of micro 
data 

Counterfactual 
analysis  

Extent actual recipients of 
SMEi would have gone 
ahead with their investment 
project 

Share of SMEi actual recipients who 
would have gone ahead with their 
investment project 

SME survey 

Scale, scope and timing in the 
absence of funding 

Lower priority criteria are shown on dark grey background 

2.4 Economic benefits 
EQ2.4  What  are  the  net  benefits  for  the  economy,  once  net  benefits  for  SMEi 
actual recipients are aggregated and considering the extent to which the growth 
of SMEi actual recipients displaced other enterprises? 
Judgement criteria 
Net benefits for the 
economy 

Indicators 
Share of non-displacing37 SMEi 
actual recipient businesses 

Method 
SME survey 

37  This  can  be  estimated  by  considering  the  location  of  their  customer  base,  competition  in  their 
main market and the expectation that competitors would take up current sales over the next year if 

27 

 
 
 
 
 
 
                                                 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

2.4 Economic benefits 
EQ2.4  What  are  the  net  benefits  for  the  economy,  once  net  benefits  for  SMEi 
actual recipients are aggregated and considering the extent to which the growth 
of SMEi actual recipients displaced other enterprises? 
Judgement criteria 

Method 

Indicators 
Aggregated net benefits for SMEi 
actual recipients  

Better understanding by banks of 
the risk profile of SMEs 

Financial 
intermediary 
interviews 

Lower priority criteria are shown on dark grey background 

the enterprise were to cease trading. See for example London Economics (2017), Economic impact 
evaluation of the Enterprise Finance Guarantee scheme. 

28 

 
 
 
                                                                                                                                                         
 Support to SME Initiative evaluation in the Member States - Final Report 

Task 4 Assessment of efficiency 

Evaluation  questions,  judgement  criteria  and  indicators,  as  well  as  methods  under  this 
task  are  given  in  Table  6.  All  criteria  are  important  to  assess  public  support  aimed  at 
easing access to finance, as identified in the economic literature. However, given possible 
constraints on the resources available for the evaluation,  success criteria and related 
evaluation questions and judgement criteria addressing efficiency were given a 
lower  priority.  For  this  reason,  these  are  shown  on  dark  grey  background  in  the 
table. 

In addition, assessment of whether SMEi implies a welfare gain or loss to the economy is 
only  possible  based  on  conditions  to  calculate  net  benefits  described  under  task  3.  EIF 
interview  provides  insight  on  both  cost-effectiveness  and  risk  optimisation  as  further 
detailed in section 4. 

Table 6 Evaluation matrix - Efficiency 

3.1 Cost-effectiveness  

EQ3.1 How do SMEi economic benefits and costs compare to each other? 

Judgement criteria 

Indicators 

There is a welfare gain or 
loss to the economy from 
SMEi 

Benefit to cost ratios 

Tasks/Methods 

EIF interview (for 
cost, benefit from 
EQ2.4) 

Lower priority criteria are shown on dark grey background 

3.2 Risk optimisation 

EQ3.2  To  what  extent  was  risk  allocation  appropriate  considering  risk  taker 
capacity for risk? 

Judgement criteria 

Indicators 

Capital set aside to cover 
losses from disbursed loans  

Method to determine the amount to 
be set aside 

Tasks/Methods 

EIF interview 

Expected versus actual losses 

Timing for release of resources set 
aside 

Allocation of risk tranches 
between ESIF, Horizon 
2020 and COSME, EIF, EIB 

Approach and method to allocate 
risk and actual risk sharing 

EIF interview 

Lower priority criteria are shown on dark grey background 

29 

 
 
 
 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

4 METHODS 

This  section  describes  the  methods  to  carry  out  the  tasks  described  under  section  3. 
Methods address the evaluation questions, aiming to both disentangle the effects of SMEi 
from  other  factors  and  to  understand  its  functioning38  (see  Box  2  for  hypotheses  of  the 
expected  change  in  the  behaviour  of  market  actors).  The  methods  to  assess  ‘why  and 
how it works?’ include desk analysis, SME survey, interviews with EIF and financial 
intermediaries,  while counterfactual  analysis  or  control  group  comparison  based  on 
an SME survey aim to answer the ‘does it work?’ question. 

Figure  2  illustrates  what  methods  address  individual  evaluation  questions,  while  the 
following  tables  clarify  how  methods  work  together  to  address  specific  evaluation 
questions.  As  clarified  by  the  figure,  SME  survey  can  provide  data  and  evidence  to 
address multiple evaluation questions. In particular it can provide: 

  detailed characteristics of SMEs, indications on their possible credit rationing, SME 

views on how SMEi did work; 

  data  to  control  group  comparison,  especially  where  enterprise  micro  data  from 
administrative  sources  or  commercial  dataset  does  not  ensure  a  sufficient 
coverage; 

  proxies of displacement effect to estimate the net benefits for the economy.   

On  the  other  hand,  enterprise  micro  data  from  administrative  sources  or  commercial 
dataset can contribute to assess financial and economic additionality by: 

  providing  detailed  information  on  time  series  which  are  needed  for  counterfactual 

analysis;  

  ensuring higher standards of quality and comparability of information. 

Based on such advantages and disadvantages, it is recommended that SMEi evaluations 
use  statistical  analysis  based  on  SME  survey  and  counterfactual  analysis  based  on 
administrative  micro  data  as  complementary  methods.  However,  one  of  the  two 
methods  can  be  adopted  in  the  case  the  other  is  not  feasible  in  the  specific  context,  or 
where the evaluation budget does not allow for both methods to be implemented. 

Full  details  on  how  the  specific  methods  address  individual  evaluation  questions  as  well 
as  related  judgement  criteria  and  indicators  are  in  the  following  sections  as  well  as  in 
section  3.  While  specific  methods  have  been  identified  to  address  evaluation  questions, 
SMEi  managing  authorities  may  consider  complementing  these  with  other  methods, 
including: 

 

focus  group  and  stakeholder  consultation,  for  instance  to  assess  SMEi 
outreach / relevance, whether the rationale of the initiative has evolved over time 
e.g.,  top  up  budget  allocation,  or  how  SMEi  fits  with  other  public  support 
instruments and how risk is allocated and managed; 

  case studies to elicit the key mechanisms of SMEi theory  of change39 and identify 
behavioural changes at the level of the financial intermediaries and SMEi recipients. 

38 See European Commission (2014), The programming period 2014-2020 – Guidance document on 
monitoring and evaluation of the European Cohesion Fund and the European Regional Development 
Fund. 

39  For  an  example  on  the  use  of  case  studies,  see  European  Commission  (2016),  Financial 
instruments  for  Enterprises  –  Work  page  3  Ex-post  evaluation  of  Cohesion  Policy  programmes 

30 

 
 
 
 
                                                 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

Box 2 Common methodological approach and theory-based impact evaluation 

Theory-based  evaluation40  is  particularly  appropriate  to  find  and  articulate  a  theory 
which  policy  makers  believe  make  a  policy  intervention  effective.  The  intervention 
theory  can  then  be  tested  to  assess  how  and  why  and  in  what  context  it  produces 
intended and unintended effects41.  

By  getting  inside  the  black  box  of  the  intervention,  theory-based  evaluation  illustrates 
the underlying mechanisms that make the intervention work. Mechanisms are not the 
input-output-outcome.  Instead,  they  concern  amongst  others,  beliefs,  desires, 
cognitions,  incentives  and  other  decision-making  processes  that  influence 
behavioural choices and actions. Differently from other ESIF interventions, mechanisms 
for  SMEi  should  consider  the  implementation  role  of  parties  operating  on  a  fully 
commercial  basis  -  as  it  is  also  the  case  of  other  ESIF  support  through  financial 
instruments.  

Evaluation  questions  in  this  document  have  been  developed  based  on  review  of  SMEi 
official  documents  and  relevant  literature,  as  well  as  interviews  with  some  SMEi 
participating  Member  States  and  feedback  from  a  technical  seminar42.  They  reflect  the 
importance of careful design for success of public guarantee schemes, as emphasised in 
the economic literature. Where evaluation questions are designed to fit SMEi programme 
theories across all participating Member States, they may need to  be further developed 
to address the individual programme theories. 

Evaluation questions test hypotheses on the changes in resources and reasoning of 
stakeholders that underlie the success of individual SMEi, by leading them to  different 
choices  than  they  would  make  without  support.  They  address  the  SMEi  management 
structure  and  operating  rules  -  including  on  risk-sharing  and  the  key  features  of  the 
guarantees  provided  -  and  investigate  whether  and  how  the  SMEi  triggers  different 
change  mechanisms  across  different  types  of  financial  intermediaries  and  SMEi 
recipients. In addition, evaluation questions acknowledge that different mechanisms are 
possibly  at  work  depending  on  the  types  of  credit  rationing  and  benefits  passed  to 
enterprises.  They  explicitly  consider  the  context  in  which  SMEi  operates,  including 
specific elements that can be expected to work in close interaction with the mechanisms 
e.g., availability and use  of  other types of guarantees, including business and personal 
guarantees  and  collateral.  Where  the  hypotheses  being  tested  focus  on  contractual 
arrangements establishing rules and setting incentives of the parties, they also call into 
question  the  capacities  of  those  parties  as  well  as  the  intrinsic  trade-offs  of  market-

2007-2013  focusing  on  the  European  Regional  Development  Fund  (ERDF)  and  the  Cohesion  Fund 
(CF). 

40  For  guidance,  FAQs  and  reference  materials  and  resources  on  the  theory-based  impact 
evaluation approach, see ec.europa.eu/regional_policy/en/policy/evaluations/guidance. 

41 On this basis, theory-based evaluation also provides a narrative linking the specific intervention 
to  the  policy  specific  context,  allows  for  generalisation  to  other  settings  and  timeframes,  and 
supports  policy  understanding  and  communication  to  stakeholders.  See  Riché  M.  (2013),  Theory 
Based Evaluation: A wealth of approaches and an untapped potential, European Commission. 

42  Participants  to  the  seminar  included  representatives  of  SMEi  managing  authorities,  European 
Commission officials and two external discussants.  The  seminar was held  in  Brussels on March 3, 
2020. 

31 

 
                                                                                                                                                         
 Support to SME Initiative evaluation in the Member States - Final Report 

based instruments.  

Key hypotheses being tested are inter alia: 

  Administrative  capacity  and  stakeholder  involvement  allow  to  design  SMEi 
according  to  the  expected  needs  of  national  financial  markets  over  the  relevant 
period. 

  Risk  takers  agree  on  conditions  that  suit  their  risk-taking  capacity  while  allowing 

to improve access to finance for credit-rationed viable enterprises. 

  Terms  and  conditions  set  in  the  funding  agreement  ensure  EIF  mandate  as  fund 

manager is aligned with SMEi intended results. 

  EIF  procedures  for  selecting  financial  intermediaries  as  well  as  terms  and 
conditions of operational agreements ensure financial intermediaries: (a) have an 
interest to participate to SMEi, due to risk-sharing and reduced regulatory capital 
charges;  (b)  have  the  capacity  and  incentives  to  implement  SMEi  by  providing 
finance to eligible enterprises that are viable but credit rationed. 

  Operational  agreements  also  have  sufficient  flexibility  to  ensure  unexpected 
changes  in  the  national  financial  markets  do  not  prevent  particular  mechanisms 
from being triggered; 

 

Increased finance or better financing conditions encourage SMEi recipients to build 
capability  that  would  not  have  built  otherwise  and  can  achieve  the  intended 
change. 

Source: t33 

32 

 
 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

Figure 2 How methods address evaluation questions (overview) 

Source: t33 

33 

 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

Table 7 How methods work together to address individual evaluation questions 
(effectiveness)  

Effectiveness 

EQ 

Role of methods 

EQ1.1 

  Annual Implementation Report (AIR) and EIF data analysis provides 

information on the extent to which SMEi achieved expected output and 
results in the respective participating Member State. 

EQ1.2 

  EIF data provides basic features of SMEi recipients and the support 

provided, e.g., credit amounts, 1-year probability of default, maturity. 

  SME survey can provide complementary details on SMEi recipient 

enterprises e.g., owner characteristics, and information on potential 
recipients.  

  EIF and financial intermediary interviews complement information on 
SMEi recipients and support provided, where detailed data is not 
available. 

EQ1.3 

  SME survey provide details on SMEi implementation e.g., SMEi 

awareness, timing of loan approval. 

  EIF and financial intermediary interviews provide different perspectives 
on how SMEi worked e.g., implementation mechanisms and conditions, 
attractiveness to financial intermediaries. 

EQ1.4 

  EIF and financial intermediary interviews provide different perspectives 

on EIF contribution to build implementation capacity. 

EQ1.5.1 

  AIR and EIF data analysis informs on the timing of SMEi set-up and 

implementation.  

  EIF and financial intermediary interviews provide insight into market 

change and implications for SMEi rationale. 

EQ1.5.2 

  EIF interview provides insight into how SMEi was capable to adapt to 

market change. 

EQ1.5.3 

  AIR data analysis gives details on other similar programmes i.e., 

Cohesion data. 

  EIF and financial intermediary interviews give details on any synergies 

or overlaps with other similar instruments, under ESIF or using other 
national or EU resources. 

Source: t33 

Table 8 How methods work together to address individual evaluation questions 
(impact) 

Impact 

EQ 

Role of methods 

EQ2.1 

  Counterfactual analysis provides an assessment of financial additionality.  

  Depending on suitability of micro data, SME survey can provide an 

alternative or complementary assessment. 

34 

 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

Impact 

EQ 

Role of methods 

  Financial intermediary interviews provide complementary information. 

EQ2.2 

  Counterfactual analysis provides an assessment of economic 

additionality.  

  Depending on suitability of micro data, SME survey can provide an 
alternative or complementary assessment, based on ‘self-reported’ 
benefits. 

EQ2.3 

  Counterfactual analysis provides an assessment of project additionality 

based on balance sheet investment indicators. 

  SME survey provides perceptions on scale, scope and timing of projects 

without SMEi support (where SMEi support has been used for 
investment) 

EQ2.4 

 

If carried out, SME survey provides proxies for displacement.  

  Financial intermediary interviews give any improved understanding of 

SME risk profile. 

Source: t33 

Table 9 How methods work together to address individual evaluation questions 
(efficiency) 

Efficiency 

EQ 

Role of methods 

EQ3.1 

  Benefits are from EQ2.4.  

  EIF interview provides SMEi costs.  

EQ3.2 

  EIF interview provide information to assess suitability of risk allocation. 

Source: t33 

35 

 
 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

4.1 Desk analysis 

This  section  illustrates  the  methods  under  desk  analysis,  including  analysis  of  AIR  and  EIF  data 
(4.1.a),  enterprise  micro  data  (4.1.b),  financial  markets  and  SMEi  ex-ante  assessment  key 
assumptions (4.1.c). 

4.1.a Analysis of AIR and EIF data 

Analysis will be based on:  

  AIR data, including based on Article 46 of Regulation (EU) No 1303/201343; and 
  EIF information and data. 

Table 10 details what evaluation questions AIR data can help address. The analysis uses 
time  series  based  on  annual  submissions  to  clarify  implementation  progress  over  the 
relevant  period.  As  concerns  information  addressing  SMEi  timeline,  this  may  need  to  be 
complemented with further data identifying other key implementation steps e.g., date of 
signature of the operational agreements with financial intermediaries.  

Table 10 Contribution of AIR data to evaluation questions 

AIR data  
Output and results indicator(s), target and 
achieved values 

Total value of new debt finance created by 
SMEi 

Number of final recipients supported by the 
financial product, by size type 

Date of completion of the ex-ante assessment 
Date of signature of the funding agreement 

Programme contributions committed with final 
recipients 

Programme contributions paid to final 
recipients  

Evaluation question 
EQ1.1 To what extent has SMEi 
achieved the expected output and 
results, based on programme 
intervention logic? 

EQ1.2 What are the characteristics of 
SMEi recipients and how do they 
compare to (different groups of) 
potential recipients? 

EQ1.5.1 How has SMEI intervention 
rationale changed from conception to 
implementation? 

This includes comparison between SMEi and 
other comparable financial instruments across 
many of the variables reported on the basis of 
Article 46 CPR 

EQ1.5.3 How does SMEI support fit 
with similar instruments financed with 
Cohesion policy or national resources? 

 Source: t33 

43  See  also  European  Commission,  Annotated  template  for  reporting  on  financial  instruments 
according to Article 46 Common provisions regulation. The dataset includes additional information 
relevant to the evaluation general module, as described below.  

36 

 
 
 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

EIF  information  includes  call  for  expression  of  interest  documentation.  Review  of  this 
documentation  gives  the  terms  of  SMEi  support,  thus  providing  insight  into  the 
implementation mechanisms established to promote the desired change (see EQ 1.3). 

The  analysis  also  considers  details  on  the  individual  supported  enterprises  gathered  by 
the EIF, including44: 

  Enterprise name; 
  Address, Postal Code, Region (NUTS2); 
  Date of establishment; 
  Sector (NACE245); 
  Number of employees; 
  Total turnover (EUR); 
  Total assets (EUR); 
  SME internal scoring/rating (Rating class), if applicable; 
  Financial Intermediary rating model, if applicable; 
  1-year probability of default. 

This  data  contributes  to  addressing  EQ1.2  and  should  be  available  for  enterprises 
supported by SMEi guarantee option as well as for enterprises on the additional portfolio 
generated under the securitisation instrument46. 

Moreover,  the  analysis  should  also  cover  the  key  features  of  final  recipient  transactions 
and modifications, payment demand and loss recovery schedules, which are also reported 
by  financial  intermediaries  to  the  EIF.  Maturity  of  the  finance  provided  to  SMEs  is 
especially  important  as,  given  all  other  things  being  equal,  risks  generally  increase  for 
longer maturities.  

44 This list is based on review of SMEi indicative term sheets. It should be noted that data reported 
to  the  managing  authority  may  be  more  limited  –  see  Article  16  of  Commission  Implementing 
Decision  of  11  September  2014  on  the  model  of  funding  agreement  for  the  contribution  of  the 
European  Regional  Development  Fund  and  the  European  Agricultural  Fund  for  Rural  Development 
to  joint  uncapped  guarantee  and  securitisation  financial  instruments  in  favour  of  small  and 
medium-sized  enterprises.  Additional  information  may  be  obtained  based  on  duties  stipulated 
under Article 18 of the same Commission Implementing Decision. 

45 NACE is the statistical classification system of economic activities in the European Union. 

46  See EIF,  Annex  II to the Open Call for Expression  of Interest to select Financial Intermediaries 
under  the  SME  Initiative  Italy  –  Securitisation  Instrument  –  Indicative  SMEi  transactions  term 
sheet. 

37 

 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

Figure 3 Example of new financial recipient transactions reporting requirements 

Source: EIF, SMEi Spain Indicative direct guarantee term sheet 

4.1.b Analysis of micro data 

Data  survey  (see  section  3  –  Task  1)  confirms  the  availability  of  suitable  micro  data  on 
SMEi  supported  enterprises  and  enterprises  potentially  eligible  as  final  recipients  based 
on SMEi objectives. On this basis, analysis of micro data contributes to addressing EQ1.2 
and is also the basis for counterfactual analysis (see section 4.5). 

Based  on  a  review  of  academic  research  and  evaluations,  the  most  relevant  sources  of 
micro data in SMEi countries  have been identified47.  Information sourced in the banking 
sector e.g., credit registers, could be confidential and potentially not disclosed/disclosable 
to  evaluators.  In  this  case,  the  feasibility  to  have  some  analyses  carried  out  directly  by 
the banking sector could be investigated. 

Box 3 Main sources of enterprise micro data in SMEi Member States 

Bureau van Dijk data bases include information on around 21 million companies across 
Europe.  It  can  be  used  to  research  individual  companies,  search  for  companies  with 
specific profiles and for analysis. In particular, Orbis-Amadeus is a comprehensive, pan-
European database – see details below on coverage of SMEi Member States - containing 
accounting information for both publicly traded and privately held companies.  

CERVED dataset provides financial accounts for the universe of Italian firms that have 
the  legal  structure  of  limited  liability  corporations.  It  provides  classified  financial 
statements;  that  is,  the  balance  sheets  of  the  firms  processed  by  the  CERVED  to 
ensure  accounting  consistency  overtime  and  across-firms.  The  dataset  also  includes 
non-classified  financial  accounts.  These  are  in  principle  more  similar  to  the  actual 
balance sheets used by the bank at the time of the application, although banks may also 
use provisional financial statements. CERVED also provides official and non-official data 

47  Enterprise  micro  data  used  in  the  reviewed  literature  is  in  most  cases,  and  except  for  credit 
registers, from commercial providers. Commercial databases maintain and enrich information from 
public  sources,  such  as  Kaupparekisteri  in  Finland,  InfoCamere  in  Italy  or  Central  de  Balances  in 
Spain.  Some  literature  in  Spain  use  data  made  available  by  a  Spanish  financial  institution  for 
research purposes. 

38 

 
 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

on private partnerships and sole proprietorships, which are  widespread legal structures 
for very small firms. 

Credit registers. In Italy, the credit register of the Bank of Italy collects data at the 
firm  level  on  financial  variables,  such  as  loans,  either  granted  or  disbursed  by  banks, 
bad  loans  and  interest  rates.  This  archive  was  set  up  for  surveillance  purposes.  Only 
loans exceeding an EUR 30 000 threshold are included, except for bad loans which are 
included regardless of their amount. Thus, this data fails to include the very small firms, 
which  might  borrow  for  amounts  below  that  threshold.  In  Spain,  the  central  credit 
register  of  the  Bank  of  Spain  contains  detailed  information  on  all  bank  credit  to  non-
financial 
including  credit  amount,  maturity, 
creditworthiness  and  the  existence  of  collateral.  In  addition,  the  dataset  includes  the 
fiscal  identity  of  both  borrower  and  lender,  enabling  to  establish  a  matched  bank-firm 
dataset. 

institutions  above  EUR  6000, 

Other datasets include, in Italy, the MET (Monitoraggio Economia e Territorio) research 
centre unique database, which builds on an extensive sample survey of Italian industrial 
firms ensuring 3-digit NACE coverage. MET survey maps financial constraints, as well as 
business  structures  and  strategies  in  terms  of  investments,  R&D,  innovation  and 
internationalization  processes,  also  including  firms  with  less  than  10  employees.  MET 
survey  provides  information  on  the  innovation  performance  and  relevant  specific 
intangibles  e.g.,  networking  capacity,  internationalisation  processes,  which  are  not 
covered by balance sheets. The survey sample covered over 23 000 enterprises in 2015 
and 2017. 

Open  data  sources  can  also  provide  relevant  microdata.  For  example,  the  Romanian 
Ministry of Finance database48 includes information on enterprises that has been already 
used  for  evaluation  purposes  e.g.,  the  evaluation  of  support  to  microenterprise 
development  –  Key  Area  of  Intervention  4.3  of  2007-2013  ERDF  Regional  Operational 
Programme. This can be integrated with the commercial Listafirme.ro database. 

Source: t33 

Example of financial indicators available from a commercial data source is given in Figure 
4. 

Figure 4 Selected financial indicators extracted from the Orbis Database 

1 Balance sheet  
Label and definition 

  Current liabilities. Current liabilities of the company 
  Current ratio. Current assets / Current liabilities 
  Fixed assets. Total amount (after depreciation) of non-current assets 
  Loans. Short term financial debts  
  Long term debt. Long term financial debts 
  Material costs. Detail of the purchases of goods. No services 
  Shareholders’ funds. Total equity 
  Tangible fixed assets. All tangible assets such as buildings, machinery, etc. 
  Total assets. Total assets (Fixed assets + Current assets) 
  Working capital. Indicates how much capital is used by day-to-day activities = 

48 See data.gov.ro/dataset 

39 

 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

Stocks + Debtors – creditors 

2 Business characteristics 
Label and definition 

  Bureau Van Dijk’s (BvD) independence indicator. BvD Independence 

indicator, which differentiates companies according to their ownership structure 

  Listed/Delisted/Unlisted. Whether the company is listed, unlisted or has been 

delisted 

  NACE Rev. 2 division code. NACE Rev. 2 main division code 
  NACE Rev. 2 main section. NACE Rev. 2 main section description 
  Number of employees. Total number of employees included in the company’s 

payroll 

  Number of patents. Number of patents owned by the company 
  Number of trademarks. Number of trademarks owned by the company  
  Peer Group Size. Size of the BvD standard peer group 
  Standardised legal form. Standardised legal form 

3 Income Statement 
Label and definition 

  Added value. Profit + Depreciation + Taxation + Interests paid + Cost of 

employees 

  Cash flow. Profit + Depreciation 
  Cost of Employees. Detail of all the employees costs of the company 
  EBITDA. Operating profit + Depreciation 
  Gross profit. Operating revenue – Cost of goods sold 
  Interest paid. Total amount of interest charges paid for shares or loans 
  Operating revenue (Turnover). Total operating revenues (Net sales + Other 

operating revenues + Stock variations) 

4 Ratios 
Label and definition 

  P/L before tax. Operating profit + financial profit 
  Profit margin. (Profit before tax / Operating revenue) * 100 
  Liquidity ratio. (Current assets – Stocks) / Current liabilities 
  ROA using P/L before tax. (Profit before tax / Total assets) * 100 
  ROE using P/L before tax. (Profit before tax / Shareholders funds) * 100 
  Solvency ratio (Asset based). (Shareholders’ funds / Total assets) * 100 

Source: Asdrubali and Signore (2015)49 

Examples of advantages and disadvantages of commercial datasets are illustrated in Box 
4. 

Box 4 Examples of advantages and disadvantages of commercial datasets  

At  international  level,  Bureau  van  Dijk  ORBIS  database  provides  administrative 
information  on  non-listed  SMEs  at  firm-level.  Amadeus  is  the  ORBIS  European  subset. 
The  main  sources  of  this  database  are  official  business  registers  from  Chambers  of 
Commerce, company annual reports, newswires, and webpages.  

Kalemil-Özcan  S.  et  alii  (2015)50  identified  key  advantages  of  using  Amadeus  Orbis  at 
international and EU level as it:  

49  Asdrubali,  P.,  Signore,  S.  (2015),  The  Economic  Impact  of  EU  Guarantees  on  Credit  to  SMEs 
Evidence from CESEE Countries, EIF Working Paper 2015/29. 

40 

 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

  Contains  more  information  than  Census  and  other  international  databases, 
because  it  covers  balance-sheets  and  financial  performance  (e.g.  debt,  equity, 
assets, or bank loans) and non-listed companies. 

  Ensures a more frequent data coverage than a Census, i.e. with more data points 

over time. 

  Provides data with a detailed industrial classification up to 4-digits NACE code for 

each SME.  

  Organises  public  data  in  a  standard  global  form  ensuring  easier  international 

comparison.  

The following table, elaborated from the aforementioned paper, provides an overview of 
the Amadeus coverage for five of SMEi Member States51. 

Member 
State 

Number  of  enterprises 
(estimate) 

Source 

Bulgaria 

Less than 10% of the 
active companies 

Creditreform 

Finland 

120 000 

Suomen Asiakastieto Oy 

Italy 

900 000 

Romania 

500 000 

Not specified in the paper (but AIDA is the 
national source of Amadeus) 

Chamber of Commerce and Industry of 
Romania 

Spain 

776 000 

Informa 

This paper also identifies some challenges of Amadeus52:  

  There is a reporting lag of roughly two years, meaning that a firm’s filing in 2010 

will appear in the database in 2012.  

  Certain  companies  are  removed  from  the  database  if  there  is  no  reporting  for 

some time, even if the firm continues operating without reporting.  

  Data  cleaning  and  checking  procedures  have  to  be  carried  out,  as  it  is  however 

required for any data analysis.  

  AMADEUS is a commercial database with relevant costs. 

Source: t33 

50 Kalemil-Özcan S. et alii (2015), How to Construct Nationally Representative Firm Level Data from 
the ORBIS Global Database. 

51 Malta is covered by the database, but the paper has not reviewed this specific country. 

52 The paper also proposes solutions to overcome some of the drawbacks. 

41 

 
 
 
 
                                                                                                                                                         
 Support to SME Initiative evaluation in the Member States - Final Report 

Based on available data, this method compares – samples of – SMEi actual and potential 
recipients  across  structural  features  as  well  as  financial  and  economic  indicators. 
Significance  of  differences  between  actual  and  potential  recipient  characteristics  is 
verified using statistical testing such as F-statistics and p-values. 

In addition, the method should propose a data weighting approach that aligns actual and 
potential recipients as  much as possible based on basic characteristics such as business 
sector, age, legal status and localisation, as suggested by literature. This enables actual 
recipients to be  ‘matched’ to potential recipients with similar characteristics that did not 
receive  SMEi  support.  Descriptive  statistics  are  proposed  to  compare  actual  to  potential 
recipient characteristics adjusted by this weighting. An ad hoc verification index provides 
indications of group similarities after weighting. 

42 

 
 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

4.1.c  Analysis  of  financial  markets  and  SMEi  ex-ante  assessment  key 
assumptions 

This method provides the overall context for the evaluation and contributes to addressing 
EQ1.5.1. It prepares an overview of national financial markets, in particular loan/portfolio 
guarantees. Box 5 includes a preliminary structure for this overview, including details on 
data availability from the ECB and other relevant sources allowing for comparison across 
Member States. The analysis is complemented with other indicators published by national 
authorities  and  stakeholders,  such  as  central  banks  or  associations  of  financial 
institutions e.g., as concerns evolution at sub-national level. It also includes a description 
of key regulatory changes during the relevant period, especially as concerns bank capital 
requirements. 

Box  5  Preliminary  structure  for  an  overview  of  national 
guarantees markets 

loan/portfolio 

1.  Loan volumes and borrowing costs 

  Outstanding loans and cost-of-borrowing for non-financial corporations 

The European Central Bank datasets on the volume of outstanding loans include only the 
Euro  area  countries,  from  2003  to  2019,  based  on  the  Monetary  Financial  Institutions 
(MFI)  interest  rate  statistics.  Outstanding loans  are  classified  based  on  loans  maturity, 
and on loans vis-à-vis the Euro area and to the Euro area. 

The  cost-of-borrowing  for  non-financial  corporations  is  based  on  the  MFI  interest  rate 
statistics. It represents a weighted average of rates on short-term and long-term loans 
to  non-financial  corporations.  Data  are  available  for  the  Euro  area  countries.  The 
available European Central Bank data covers the period from 2003 to 2019 for the Euro 
area members. 

  Small loans to non-financial corporations, new business volumes (nominal and as 

share of total) 

The  European  Central  Bank  data  covers  the  Euro  area  countries,  from  2003  to  2019. 
Data  are  also  available  for  Romania  from  2010,  while  Bulgaria  is  not  included.  The 
indicator is based on the MFI interest rate statistics. Loans are classified as small (<EUR 
0.25 million), medium-sized (EUR 0.25 – EUR 1 million), large (> EUR 1 million). 

 

Interest rates by loan size and maturity 

The European Central Bank datasets include data for the Euro area countries from 2003 
to  2019.  Data  for  Romania  and  Bulgaria  are  also  available  from  2007.  The  indicator  is 
based  on  the  MFI  interest  rate  statistics.  Three  different  loan  size  categories  are 
considered,  as  mentioned  above.  Interest  rate  data  are  further  broken  down  according 
to  loan  maturity,  where  interest  rates  on  loans  with  a  maturity  less  than  three  years 
serve as a proxy for short term lending, 3 to  5 years for medium term lending and 10 
years and more for long term lending. 

2.  SME financing from a supply perspective 

  Net changes in credit standards applied to the approval of loans or credit lines for 

43 

 
 
 
 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

SMEs and large enterprises 

The  indicators  are  based  on  the  European  Central  Bank  Lending  Survey  Statistics.  The 
available data covers the Euro area members, from 2003 to 2019. 

  Factors contributing to changes in credit standards for SMEs 

The  indicators  are  based  on  the  European  Central  Bank  Lending  Survey  Statistics.  The 
available data covers the period from 2003 to 2019 for the Euro area members. 

3.  SME financing from a demand perspective 

  Sources of external financing for SMEs 

The  European  Central  Bank  data  covers  the  Euro  area  countries,  from  2014  to  2019. 
Data for Bulgaria and Romania is available from the European  Commission for relevant 
waves of the survey. 

  Share of SMEs ranking access to finance as a highly important issue 

The  European  Central  Bank  provides  data  from  2009  to  2019  for  the  Euro  area 
countries. Data for Bulgaria and Romania is available from the European Commission for 
relevant waves of the survey. 

  Factors driving the availability of external financing to SMEs 

The  European  Central  Bank  provides  data  from  2009  to  2019  for  the  Euro  area 
countries. Data for Bulgaria and Romania is available from the European Commission for 
relevant waves of the survey.  

4.  SME guarantees53 

  Volume of outstanding guarantees in portfolio, nominal and scaled by GDP 

Data is available for the members of Association Européenne du Cautionnement (AECM) 
from 2015 to 2018. Malta is not covered. 

  Volume of new guarantees granted, nominal and by GDP 

Data is available for the AECM members from 2015 to 2018. Malta is not covered. 

  Number of SME beneficiaries 

Data is available for the AECM members from 2015 to 2018. Malta is not covered. 

Source: t33 

Analysis of financial markets also enables a quantitative analysis of the evolution of key 
assumptions  in  the  ex-ante  assessment54.  This  implies  updating  individual  assumptions 

53  Data  availability  as 
from 
aecm.eu/communication/statistics/ 

reports  and  statistics  produced  by 

the  AECM,  at: 

54 SMEi ex-ante assessment include assumptions: (1) that are general and were used to evaluating 
SMEi options: these concern regulatory aspects, SMEi required critical mass, conditionalities to be 
applied to contractual arrangements and design of the instrument – see Annex 1 to Chapter 2; (2) 
underlying  the  leverage  calculations  –  see  Annex  2  to  Chapter  2;  (3)  used  in  the  funding  gap 

44 

 
 
 
 
                                                 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

used in SMEi ex-ante assessment for funding gap calculation55. To estimate the potential 
unmet  SME  demand  for  finance,  SMEi  ex-ante  assessment  uses  data  from  SAFE  of  the 
European  Commission  and  the  European  Central  Bank56.  Based  on  this  data,  SMEi  ex-
ante assessment:  

  estimates  the  number  of  enterprises  not  obtaining  a  loan,  while  being  financially 

viable; 

  multiplies  this  number  by  the  average  enterprise  loan  amount  to  estimate  the 

funding gap. 

SMEi  ex-ante  assessment  method  defines  two  boundaries  for  financially  viable 
enterprises: 

 

 

the lower boundary covers enterprises with turnover growth of at least 20% in the 
previous three years i.e., high-growth enterprises; 

the  upper  boundary  covers  enterprises  with  positive  turnover  growth  in  the 
previous six months. 

Estimating unsuccessful enterprises (see also Box 1) is based on: 

 

 

 

rejected transactions, where the lender did not make an offer to the applicant;  

the offer by the finance provider was rejected by the applicant, for instance due to 
the high cost (high interest rate); 

lack  of  applications,  where  final  recipients  did  not  apply  for  financing  because  of 
expectation to be rejected. 

Table 11 details the variables used by SMEi ex-ante assessment funding gap calculation 
which would need updating under this method. 

estimation process; (4) concerning the funding gap coverage. The consultants considered (3) to be 
the key assumptions referred to in the tender specifications. The ex-ante assessment also provides 
analyses of SME difficulties in accessing finance and the expected evolution of SME financing gap. 
In this document, such analyses are covered under the overview of national financial markets. 

55  European  Commission  (2013),  Ex-ante  assessment  of  the  EU  SME  Initiative,  SWD  (2013)  517 
final. 

56 See ec.europa.eu/growth/access-to-finance/data-surveys_en 

45 

 
 
 
 
 
                                                                                                                                                         
 Support to SME Initiative evaluation in the Member States - Final Report 

Table  11  Data  sources  and  variables  used  by  SMEi  ex-ante  assessment  to 
calculate the funding gap 

Variable 

Enterprises 

Source 

Eurostat 

Financially 

Lower boundary: 

Eurostat 

viable 

share of high growth 

enterprises 

enterprises 

Year 

2010 

2010 

Upper boundary: 

SAFE, waves 2009 

Average 

share of enterprises 

and 2011 

2009- 2011 

with positive 

turnover growth in 

the last six months 

Unsuccessful 

Enterprises that 

SAFE, waves 2009 and 2011 

enterprises 

applied 

Enterprises rejected 

SAFE, waves 2009 and 2011 

Enterprises refused 

SAFE, waves 2009 and 2011 

Enterprises 

SAFE, waves 2009 and 2011 

discouraged 

Average 
enterprise 
loan size 

Source: fi-compass 

BvD's Orbis Database of 
Company information on 
liabilities, and the BACH-ESD 
Database 

Average  2009- 
2011 

Average  2009- 
2011 

Average  2009- 
2011 

Average  2009- 
2011 

Average  2010- 
2011 

46 

 
 
  
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

4.2 SME survey 

This  method  gathers  enterprise  micro  data  on  actual  and  potential  SMEi  recipients, 
allowing  for  control  group  comparison.  Under  the  paradigm  of  theory-based  impact 
evaluation,  this  method  complements  analysis  of  administrative  micro  data  and  helps 
address the ‘how’ and ‘why’ SMEi works. It also contributes to the estimation of benefits. 

In particular, this method could contribute to answering: 

  EQ1.2 What are the characteristics of SMEi recipients and how do they compare to 

(different groups of) potential recipients? 

  EQ1.3 How do SMEi implementation mechanisms promote the desired change? 

  EQ2.1  To  what  extent  and  at  what  conditions  would  SMEi  actual  recipients  have 

been financed without SMEi support? 

  EQ2.2 How does performance of SMEi recipients compare with potential recipients? 

  EQ2.3 To what extent would actual recipients of SMEi have gone ahead with their 

investment without SMEi support? 

  EQ2.4  What  are  the  net  benefits  for  the  economy,  once  net  benefits  for  SMEi 
actual recipients are aggregated and considering the extent to which the growth of 
SMEi actual recipients displaced other enterprises? 

Table 12 provides dimensions a survey questionnaire could address to complement data 
from  administrative  or  commercial  datasets,  or  where  necessary  information  is  not 
available from those sources.  

47 

 
 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

Table 12 Possible dimensions of the survey questionnaire57 

1 Profile of enterprise  

  Business profile 

  Business  practices, 

including 

formal  business  practices  and  cutting-edge 

technology 

  Ownership  characteristics,  including  age,  management  experience,  qualification, 

underrepresented owners 

2 Finance-seeking and application process 

  Finance-seeking  process,  including  reasons  for  seeking  finance,  awareness  of 

SMEi, alternative sources of finance, number of applications  

  Applications,  including  when  SMEi  was  first  discussed  with  lender,  bank  service 

satisfaction  

  Sources of external advice  

3 Financial and project additionality 

  Self-reported  financial  additionality,  including  ability  to  get  a  loan  without  SMEi 
support (deadweight), reasons why unconstrained businesses chose SMEi, reasons 
given by bank for taking out an SMEi loan, value and type of collateral available to 
borrowers, pressure on cashflow and ability to meet demands from creditors 

  Self-reported project additionality i.e., if the enterprise would have not proceeded 

with their project without the finance, or not with the same timing or scale 

4 Benefits and contribution 

  Self-reported benefits and loan finance contribution 

  Location  of  the  customer  base  and  competition  in  main  markets,  which  helps 

assess any displacement58 

Source: t33 

An example survey questionnaire would include questions as illustrated in Table 13. Most 
questions expect single-option replies, unless explicitly mentioned in the table. 

For  SMEi  recipients,  the  survey  would  focus  on  the  years  since  they  received  SMEi 
support.  If  SMEi  support  was  received  over  multiple  years,  data  analysis  would  need  to 
provide  detail  by  cohorts.  Additionally,  where  a  single  enterprise  received  support  in 
multiple years, the survey would need to focus on a specific year e.g., the year in which 
the largest amount of finance was drawn. More generally, formulation of some questions 
depends  on  the  timing  the  questionnaire  is  submitted  compared  to  when  SMEi  support 
was  provided,  thus  may  need  to  be  adapted  to  the  specific  situations  in  SMEi  Member 
States. 

57 Dimensions of the proposed survey questionnaire and specific questions build on various sources 
and  in  particular  on  Allinson,  G.,  Robson,  P.,  and  Stone,  I.  (2013),  Economic  evaluation  of  the 
Enterprise Finance Guarantee scheme. 

58  See  for  example  London  Economics  (2017),  Economic  impact  evaluation  of  the  Enterprise 
Finance Guarantee scheme. 

48 

 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

Table 13 Example survey questions 

1 General characteristics of the enterprise 

Dimension59 

1.1 How would you characterise your enterprise? 

[Subsidiary, branch, autonomous profit-oriented enterprise, a non-profit 
enterprise] 

1.2 In which region is your enterprise located? [Based on NUTS] 

1.3 What is the main activity of your company? [Based on NACE] 

1.4  How  many  persons  does  your  enterprise  currently  employ  in  full 
time or part time in country at all locations? [Range-based; headcount, 
excluding  unpaid  family  workers  and  freelancers  working  regularly  for 
enterprise] 

1.5 In which year was your enterprise registered? [Year] 

1 

1 

1 

1 

1 

1.6 What was the annual turnover of your enterprise in country in year?  

1 

[Range-based; currency conversion for non-euro countries] 

1.7  Did  your  enterprise  receive  any  [other,  for  SMEi  recipients]  public 
support during the last n years?  

1.8  Which  formal  business  practices  are  used  in  your  enterprise? 
[Multiple options]  

1.9  How  intensive  do  you  believe  competition  to  be  in  your  enterprise 
main markets60?  

1.10 Where are your current customers based? 

1.11 Who owns the largest stake in your enterprise?  

1.12  What  experience  do  owners  have  of  business  management? 
[Range-based, years of experience]  

1.13  What  is  their  level  of  educational  attainment?  [Based  on 
qualification framework] 

1.14 Over start year – end year, how much did your enterprise grow on 
average per year? [Range-based] 

1 

1 

4 

4 

1 

1 

1 

1 

2 Finance-seeking and application process 

Dimension 

2.1  Have  you  applied  for  credit  financing  in  year?  [Final  template  to 
indicate the year]   

2.2 What was main reason for seeking external finance?  

2 

2 

2.3  What  was  the  reason  working  capital  was  sought?  [Depending  on  2 

59 According to dimensions in Table 12. 

60 This question helps assess any displacement effects. See for example London Economics (2017), 
Economic impact evaluation of the Enterprise Finance Guarantee scheme. 

49 

 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

answer to 2.2] 

2.4 What business and personal collateral was available at the time your 
enterprise applied for finance? 

2.5  Was  your  enterprise  experiencing  any  pressure  on  cashflow  and 
ability  to  meet  demands  from  creditors,  at  the  time  it  applied  for 
finance? 

2.6 From what sources did you know about SMEi?  

2.7  How  many  applications  for  finance  did  you  submit  before  securing 
SMEi support?  

2.8 When was SMEi first discussed with the bank? 

2.9  What  reasons  were  offered  by  the  bank  for  taking  out  SMEi 
guaranteed finance?  

2.10 How long did it take before the bank reaching a decision?  

3 

3 

2 

2 

2 

2 

2 

2.11 Did the financing decision time have any impact on your business? 

2 

2.12 How clear was the explanation by the bank that you remained fully 
liable for the loan? 

2.13  How  clear  was  the  explanation  by  the  bank  that  the  interest  rate 
was  below  the  rate  your  enterprise  would  have  been  charged  without 
SMEi support? 

2 

2 

2.14 Did your enterprise use any external sources of information, advice 
or support to help raising finance? 

2 

3 Financial and project additionality 

Dimension 

3.1 In your opinion, would other external finance or a loan without SMEi 
guarantee have been available to you? 

3.2 If that was likely, for what reason did you choose SMEi, rather than 
opting for other finance? 

3.3 How likely is it is that would you have gone ahead with your project 
without SMEi support? 

3.4  Would  its  scale,  scope  and  timing  have  been  the  same  without 
funding? 

3 

3 

3 

3 

4 Benefits and contribution 

Dimension 

4.1  Over  the  past  n  years,  how  much  did  your  enterprise  grow  on 
average per year in terms of employment? 

4.2  Over  the  past  n  years,  how  much  did  your  enterprise  grow  on 
average per year in terms of turnover? 

4.3  Over  the  past  n  years,  how  much  did  your  enterprise  grow  on 
average per year in terms of profitability? 

4.4  Over  the  past  n  years,  how  much  did  your  enterprise  grow  on 
average per year in terms of labour productivity? 

4.5  Over  the  past  n  years,  how  much  did  your  enterprise  grow  on 
average per year in terms of exports? 

4 

4 

4 

4 

4 

50 

 
 Support to SME Initiative evaluation in the Member States - Final Report 

4.6  Which  (if  any)  do  you  believe  to  be  the  direct  benefits  to  your 
enterprise  as  a  result  of  having  received  finance?  [Separately  for 
employment,  turnover,  survival,  new  or  improved  products  or  services, 
new  or  improved  processes,  value  added  per  employee,  reduced  costs, 
increased export] 

4 

4.7  To  what  extent  it  would  have  otherwise  been  possible  for  your 
enterprise  to  achieve  similar  business  outcomes?  [Including  different 
timing] 

4 

Source: t33 

Table 14 indicates which enterprise groups each questionnaire dimension could target61. 
Literature review shows that an expected average interview duration could be around 25 
minutes for SMEi recipients and 15 minutes for potential recipients. 

Table 14 Questionnaire dimensions and enterprise target groups 

Section 

SMEi final 
recipients 

Non recipients 
using loan 
finance 

Non recipients 
not using loan 
finance 

1. General characteristics of the 
enterprise 

2. Finance-seeking and SMEi 
application process 

3. Financial and project 
additionality 

4. Benefits and contribution 

* Excluding SMEi-related questions  

  * 

  * 

The  sample  of  potential  SMEi  recipients  would  need  to  be  drawn  from  the  general 
business  population  by  matching  SMEi  recipients  basic  characteristics  e.g.,  in  terms  of 
size,  business  age,  broad  business  sector.  For  analysis,  the  achieved  sample  of  SMEi 
actual  and potential  recipients  should  be  weighted  by  basic  characteristics  to  reflect  the 
population of SMEi actual recipients overall. 

Within potential SMEi recipients62,  comparison  with particular groups  of  enterprises may 
provide  specific  insight.  For  example,  enterprises  who  did  and  did  not  apply  for  finance 

61  Based on Allinson,  G.,  Robson, P., and  Stone,  I.  (2013), Economic evaluation of the Enterprise 
Finance Guarantee scheme. 

62  Literature  clarifies  that  unsuccessful  SMEi  applicants  would  be  the  most  suitable  control group, 
minimising selection bias. See, for example, in Martini, A., and Bondonio, D. (2012), Counterfactual 
impact  evaluation  of  Cohesion  Policy:  impact  and  cost-effectiveness  of  investment  subsidies  in 
Italy.  The  authors  clarify  that  unsuccessful  applicants  share  with  recipients  of  support  ‘the  same 
desire  to  invest,  which  is  an  important  proxy  of  unobservable  such  as  business  strategies  and 

51 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
                                                 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

are obvious comparison groups. Within enterprises that did apply for finance, those that 
received  some,  but  not  all,  of  the  funds  they  applied  for  (volume  rationed),  could  be 
another comparison group63.  

The  method  should  determine  a  sample  size  to  ensure  the  required  accuracy  and 
confidence  level.  This  should  consider  an  expected  response  rate,  which,  based  on 
reviewed  literature,  could  be  at  around  15-20%  for  actual  recipients  and  5-10%  for 
potential  recipients64.  When  identifying  comparison  groups,  the  method  should  consider 
that  enterprises  applying  for  external  finance  are  only  a  fraction  of  the  total  business 
population, which may imply using disproportional stratified sampling. 

managerial abilities, as well as specific market trends to which the firms are exposed’. Nonetheless 
it is understood that SME apply for finance at financial intermediaries rather than for SMEi support. 
In this sense, there are no unsuccessful SMEi applicants. 

63 However, questions in Table 13 do not address the outcome of the loan application. 

64  This  refers  to  achieved  interviews  as  a  share  of  usable  sample  and  it  is  only  indicative,  as 
response rates would depend on a number of factors, including the specific country and timing of 
the  interviews.  However,  research  literature  indicates  that  even  small-scale  interventions  in 
deprived  areas  can  generate  (micro)  business  respondents  that  are  capable  of  understanding 
standard  evaluation  terminology.  See  for  example,  Cowling,  M.  (2019),  LDBG  Loan  and  Grant 
Funding Recipients - Draft Evaluation Report. 

52 

 
 
 
                                                                                                                                                         
 Support to SME Initiative evaluation in the Member States - Final Report 

4.3 EIF interview 

Under  the  paradigm  of  theory-based  impact  evaluation,  this  method  helps  address  the 
‘how’ and ‘why’ SMEi works. In particular, it could contribute to addressing: 

  EQ1.2 What are the characteristics of SMEi recipients and how do they compare to 

(different groups of) potential recipients? 

  EQ1.3 How do SMEi implementation mechanisms promote the desired change? 

  EQ1.4  What  was  the  contribution  of  the  EIF  know-how  to  ensuring  adequate 

implementation capacity? 

  EQ1.5.1  How  has  SMEi  intervention  rationale  changed  from  conception  to 

implementation? 

  EQ1.5.2  What  mechanisms  were  established  to  adapt  SMEi  support  to  changing 

market conditions? 

  EQ1.5.3  How  does  SMEi  support  fit  with  similar  instruments  financed  with 

Cohesion policy or national resources? 

  EQ3.2  To  what  extent  was  risk  allocation  appropriate  considering  risk  taker 

capacity for risk? 

An example template for semi-structured interview is illustrated in Table 15. 

Table 15 Example template for EIF interview 

Question 

1  What  was  the  process  to  establish  the  key  parameters  of  the  support 
provided e.g., guarantee rate, guarantee fee? 

2  What  factors  were  considered  to  decide  a  specific  guarantee  rate?  How 
does  the  rate  differ  from  other  guarantees  provided  by  the  EIF  in  the 
country?  

EQ 

EQ1.3 

EQ1.3 

3 What factors were considered to decide a specific guarantee fee? How does 
this relate to the cover fees for risk takers? 

EQ1.3 

4  What  factors  were  considered  to  decide  on  minimum  and  maximum 
portfolio volume for each financial intermediary?  

EQ1.3 

5  How  did  the  conditions  offered  by  lenders  under  SMEi  expression  for 
interest  differ  from  their  standard  policy  e.g.,  in  terms  of  probability  of 
defaults of SME borrowers, loan maturities, required collateral?  

EQ1.3 

6 To what extent and for what reasons did the conditions offered by lenders 
during  implementation  differ  from  their  initial  objectives  under  SMEi 
expression of interest?  

EQ1.3 

7 How do penalties provide incentives to financial intermediaries to reach the 
agreed volumes?  

EQ1.3 

8 How do you ensure eligibility of expenditure and what if any were the main 
challenges with ensuring compliance with relevant rules?  

EQ1.3 

53 

 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

Question 

9  How  did  you  ensure  that  financial  intermediaries  have  the  necessary 
capacity to manage ESIF support? 

10 What was the contribution of the different financial intermediaries to the 
achievement of SMEi objectives? E.g. addressing different market segments, 
geographies, or just complementing each other on the same markets.  

EQ 

EQ1.4 

EQ1.3 

11  How  did  the  financial  market  change  from  SMEi  establishment  to 
implementation and what implications for SMEi rationale of intervention?  

EQ1.5.1 

12 What were the implications of the low interest rate environment on SMEi 
implementation?  

EQ1.5.1 

13 What is the envisaged total agreed for management costs and fees?  

EQ3.1 

14  What  amount  was  set  aside  to  cover  losses  from  guaranteed  loans  and 
how  did  you  estimate  that  amount  was  necessary?  What  is  the  timing  for 
these resources to be released? 

EQ3.2 

15 Who were the risk takers and did they include any promotional banks?   

EQ3.2 

16 What are the risk cover fees for risk takers? 

17 Given risk tranches and actual losses, what have been the implications for 
the capital set aside by the individual risk takers? 

EQ3.1 

EQ3.2 

18  Have  you  identified  improvements  to  risk  sharing,  given  the  different 
policies and risk-bearing capacity of the guarantors? 

EQ3.2 

19 What synergies or overlaps with other private or publicly supported SME 
guarantee  instruments?  Did  the  fact  that  part  of  SMEi  resources  are  not 
subject  to  State  Aid  allowed  for  higher  benefit  for  the  SMEs  than  other 
comparable public support?  

EQ1.5.3 

Source: t33 

54 

 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

4.4 Financial intermediary interviews 

Under  the  paradigm  of  theory-based  impact  evaluation,  this  method  helps  address  the 
‘how’ and ‘why’ SMEi works. In particular, it could contribute to addressing:  

  EQ1.2 What are the characteristics of SMEi recipients and how do they compare to 

(different groups of) potential recipients? 

  EQ1.3 How do SMEi implementation mechanisms promote the desired change? 

  EQ1.4  What  was  the  contribution  of  the  EIF  know-how  to  ensuring  adequate 

implementation capacity? 

  EQ1.5.1  How  has  SMEI  intervention  rationale  changed  from  conception  to 

implementation? 

  EQ1.5.3  How  does  SMEI  support  fit  with  similar  instruments  financed  with 

Cohesion policy or national resources? 

  EQ2.1  To  what  extent  and  at  what  conditions  would  SMEi  actual  recipients  have 

been financed without SMEi support? 

  EQ2.4  What  are  the  net  benefits  for  the  economy,  once  net  benefits  for  SMEi 
actual recipients are aggregated and considering the extent to which the growth of 
SMEi actual recipients displaced other enterprises? 

An example template for semi-structured interview is illustrated in Table 16. 

Table 16 Example template for financial intermediary interview 

Question 

1 How important was the use of SME credit guarantees for your bank at the 
time  you  applied  for  SMEi?  What  types  of  guarantees  were  being  used  e.g., 
mutual or public, loan-by-loan or portfolio-type? 

2 What were the main reasons for using guarantees? 

3  What  was  the  rationale  behind  your  decision  to  apply  for  SMEi  support? 
[Including risk-sharing and capital relief motives, as well as the possibility to 
offer improved financing conditions to clients and reaching potential clients] 

EQ 

EQ1.3 
EQ1.5.3 

EQ1.3 
EQ1.5.3 

EQ1.3 
EQ1.5.3 

4 As it concerns capital management, do you think SMEi allows better capital 
relief than it can be obtained from other credit guarantee schemes available 
in your country?  

EQ1.3 

5  Do  you  think  SMEi  guarantee  to  be  especially  suitable  for  specific  eligible 
applicant enterprises? 

EQ1.3 

6  In  your  view,  what  are  the  main  reasons  that  could  make  the  conditions 
offered  to  SME  during  implementation  differ  from  initial  objectives  under 
SMEi expression of interest?  

EQ1.2 

7 How would you assess the claim management process under SMEi in terms 
of efficiency and transparency? E.g., timing of payment of claims  

EQ1.3 

8  In  your  opinion,  how  difficult  is  it  for  a  bank  to  manage  SMEi  and  what 
could be the main challenges?  

EQ1.3 

55 

 
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

Question 

9 Did your bank have previous experience of dealing with ESIF support? How 
did interaction with EIF help to ensure the necessary capacity? 

10  In your opinion, how do the  costs for administration of SMEi compare to 
other  credit  guarantees  used  by  your  bank?  E.g.,  ensure  eligibility, 
compliance with state aid rules, publicity requirements 

EQ 

EQ1.4 

EQ1.3 

11  In  your  view,  what  are  the  key  implications  of  a  low  interest  rate 
environment for the capacity to disburse SMEi-supported credit?  

EQ1.5.1 

12 In your view, what are the key benefits in terms of access to finance for 
borrowers  under  SMEi  e.g.,  in  terms  of  collateral,  probability  of  default  of 
SME borrowers, loan maturity? How did that compare to other types of credit 
guarantees used by your bank?  

EQ2.1 

13  In  your  opinion,  to  what  extent  can  SMEi  contribute  to  a  better 
understanding by participating banks of the risk profile of SMEs?  

EQ2.4 

Source: t33 

56 

 
  
 
 
 Support to SME Initiative evaluation in the Member States - Final Report 

4.5 Counterfactual analysis 

Based  on  data  reported  to  the  EIF  on  SMEi  recipients  and  additional  micro  data  (see 
section  4.1),  counterfactual  analysis  contributes  to  addressing  evaluation  questions 
assessing SMEi impact (see section 3 for details): 

  EQ2.1  To  what  extent  and  at  what  conditions  would  SMEi  actual  recipients  have 

been financed without SMEi support? 

  EQ2.2 How does performance of SMEi recipients compare with potential recipients? 

  EQ2.3 To what extent would actual recipients of SMEi have gone ahead with their 

investment without SMEi support? 

Box 6 Counterfactual approaches 

Counterfactual  approaches  for  impact  evaluation  are  particularly  suitable  for  evaluating 
the  effectiveness  of  credit  guarantee  schemes  as  stated  by  the  World  Bank  toolkit65. 
Counterfactual  evaluation  of  the  effect  includes  estimating  the  difference  between  the 
outcome  variable  after  the  intervention  and  the  counterfactual  value  if  the  intervention 
had not been implemented. The underpinning principle behind this analysis is to quantify 
the policy effect as the difference between the (factual) result of the variable after policy 
implementation  and  the  counterfactual  result  if  the  provision  had  not  been  adopted.  As 
with  any  non-experimental  method  based  on  available  information  and  a  lack  of 
experimental counterfactual data is not observable, caution shall be applied to reduce the 
risk of selection bias. This tries to minimise the problem of non-null differences in initial 
conditions to give a reliable estimate of the counterfactual effect. 

The  main  counterfactual  methods  include  regression  discontinuity  design,  propensity 
score matching, and difference-in-difference estimates. 

Source: t33 

This method builds on four pillars, as illustrated in Figure 5 and described in more details 
below. 

Figure 5 Pillars of the module for quasi-experimental designs 

Assessment of 

data availability

Choice of specific 
methods

Assessment of 
limits and risks

Estimated time 
and budget

Source: t33 

65  World Bank (2018), Toolkit for impact  evaluation  of public credit guarantee schemes for SMEs, 
World Bank Group. 

57 

 
 
 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

Data availability. Use of this method requires to assess the availability of sufficient and 
suitable  data,  that  is  comparable  between  treated  and  control  units,  as  further  detailed 
under section 3. 

Choice of specific methods. Choice of the methodological approach requires assessing 
the suitability of specific counterfactual techniques based on SMEi specificities, as well as 
the  identification  of  the  appropriate  outcome  variables  of  the  econometric  model.  The 
methods  include  regression  discontinuity  design,  propensity  score  matching  and 
difference-in-difference estimation66. 

Box 7 Counterfactual methods considered for suitability 

Regression  discontinuity  design  (RDD)  is  used  to  assess  interventions  that  have  a 
continuous  eligibility  index  with  a  clearly  defined  cut-off  score  to  determine  who  is 
eligible and who is not. This approach can only be applied if information about scoring is 
available  and  can  be  used  for  the  estimates.  RDD  takes  advantage  of  existing 
programme  rules,  thus  can  be  used  without  changing  programme  design.  It  can  be  a 
retrospective tool as it does not rely on random assignment. The RDD strategy exploits 
discontinuity around the cut-off score to estimate the counterfactual.  

Propensity Score Matching (PSM) is a non-experimental approach that can be used 
to  identify  a  control  group  that  is  statistically  equivalent  to  the  treatment  group.  The 
idea behind matching is to compare each firm in the treatment group to a control group 
firm that is very similar. As there are many dimensions (firm size, profitability, leverage, 
urban-rural  location,  etc.)  which  the  evaluator  can  use  to  match  firms,  PSM  can 
incorporate many different characteristics. PSM essentially uses statistical techniques to 
construct  an  artificial  control  group  by  identifying  for  each  SME  being  reviewed  a  non-
treatment  SME  that  has  the  most  similar  characteristics  possible.  PSM  combines 
measures  into  a  single  score,  the  propensity  score,  which  represents  the  predicted 
probability of participating in the credit guarantee scheme. Firms with similar propensity 
scores  have  a  similar  likelihood  of  receiving  the  intervention,  thus  can  be  compared 
across  the  treatment  and  control  groups.  The  estimate  generally  uses  a  logit  model. 
Various estimators match observations as such as Nearest Neighbour Matching, Radius 
Matching,  Kernel  Matching  and  Stratification  Matching.  The  impact  of  the  intervention 
will then be measured as the difference in outcomes between the treated group and the 
control  group.  Literature  review  shows  a  wide  range  of  options  for  measuring  the 
propensity  score.  In  some  cases,  this  is  estimated  separately  by  enterprise  year  of 
registration or specific eligibility conditions e.g., if more calls for application. Non-eligible 
enterprises are excluded automatically from the control group. 

The  difference-in-difference  (DID)  method  compares  changes  in  the  outcome  of 
interest  over  time  between  the  population  enrolled  in  a  programme  (treatment  group) 
and the population that is not (control group). Use of the DID estimator requires data on 
the  outcomes  of  interest  for  both  the  treatment  and  control  groups  for  periods  before 
and after the intervention.  

66  For  an  alternative  approach  see  D’Ignazio,  A.,  Menon,  C.  (2017),  Evaluating  credit  guarantees 
for  SMEs:  evidence  from  Italy.  The  paper  proposes  an  alternative  approach  to  regression 
discontinuity  design  and  propensity  score  matching  based  on  the  use  of  instrument  variable 
procedure. 

58 

 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

The  key  assumption  of  DID  analysis  is  that  without  treatment  the  change  in  outcomes 
for  the  treatment  group  would  be  identical  to  the  change  in  outcomes  for  the  control 
group.  While  this  assumption  is  not  formally  testable,  its  validity  should  always  be 
carefully  examined  to  ensure  the  DID  impact  is  not  biased.  If  data  are  available  for 
several years before the treatment, then an easy way to assess the validity of the trend 
equality  assumption  is  whether  pre-treatment  trends  were  equal  between  the  two 
groups.  It  is  also  useful  to  control  for  baseline  characteristics  between  the  treatment 
and  control  groups.  As  in  the  case  of  PSM,  literature  review  shows  that  eligibility 
features can be used to define the control group67. 

Source: t33 

Literature review allows identifying outcome variables that are used more frequently in 
econometric  models  measuring  the  net  effects  of  public  guarantees.  A  first  set  of 
outcome  variables  relates  to  financial  additionality  i.e.,  the  different  capacity  of  SMEi 
recipients to access finance, and includes variables such as the ratio between long-term 
and  short-term  debt,  financial  debt  costs  /  total  debt,  amount  of  loans,  interest  rates, 
probability  of  bad  loans,  trade  debts,  or  creditworthiness  indexes  as  a  z-score.  The 
second  category  is  about  project  and  economic  additionality  as  defined  within  the 
framework of SMEi evaluation and includes variables such as the amount of tangible and 
intangible  assets,  total  investments,  turnover,  sales,  value  added,  number  of  full-time 
equivalent  (FTE)  employees,  productivity68.  The  appropriate  outcome  variables  can  be 
selected based on the individual SMEi intervention logic (see Section 1). 

Limits  and  risks  of  estimation  bias.  Limits  and  risks  of  selected  counterfactual 
methods should be clarified based on consideration of the data available and the specific 
context of SMEi support. 

Box 8 Limits and risks of estimation bias 

There is a selection bias in evaluating the impact of a programme if the mean outcome 
of  units  which  participated  in  the  programme  differs  from  non-supported  units  even  in 
the  absence  of  support.  This  difference  could  be  due  to  a  pre-existing  difference  in  key 
characteristics  of  both  groups  which  also  determine  group  performance.  Selection  bias 
can  substantially  affect  and  distort  estimates  of  programme  impact.  It  may  arise  when 
evaluators decide to use programme non-beneficiaries as a control group. Understanding 
this  requires  noting  that  a  decision  on  programme  participation  can  be  based  on 
unobserved factors that are correlated with the outcomes. For instance, firms may decide 
to  participate  in  a  programme  or  not  based  on  preferences,  former  performance, 

67 See the multi-stage scoring system in De Blasio, G., De Mitri, S., D’Ignazio, A., Finaldi Russo, P., 
Stoppani,  L.  (2017),  Public  guarantees  on  loans  to  SMEs:  and  RDD  evaluation,  Bank  of  Italy 
working papers. 

68  On  the  impact  of  credit  to  productivity  see  Gatti  R.,  Love  I.  (2006),  Does  access  to  credit 
improve  productivity?  Evidence  from  Bulgarian  firms,  World  Bank.  Moreover,  see  Castillo  L.  L., 
Guasch J.L. (2012), Overdraft facility policy and firm performance: an empirical analysis in eastern 
European Union industrial firms, Policy Research Working Paper 6101. 

59 

 
 
 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

a 

certain 

performance 

characteristic 

opportunity  costs  and  expected  gains.  Furthermore,  programmes/measures  are  not 
assigned  randomly  to  beneficiaries  but:  i)  are  designed  to  target  specific  beneficiaries 
with 
underperforming 
producers/enterprises/areas, etc.), or ii) include eligibility conditions which only apply to 
certain  types  of  economic  units  e.g.,  the  best  enterprises.  Selection  of  ‘unobservables’ 
(e.g., taking all SMEi non-beneficiaries as a control group) makes it difficult to establish a 
credible  counterfactual.  Those  not  participating  in  the  programme  are  generally  an 
inadequate  comparison  for  those  participating,  since  it  is  difficult  to  tell  whether  the 
differences  in  outcomes  of  the  two  groups  are  due  to  differences  in  unobservable 
characteristics  or  to  programme  support.  Using  unsuccessful  applicants  as  a  control 
group helps to minimise the problem, but this is not always possible or easy. 

(e.g., 

The evaluation should be designed to reduce / avoid the possible risk of estimation bias 
related  to  the  deviations  from  the  conditional  independence  assumption,  also 
known  as  ‘unconfoundedness’  or  ‘selection  on  observables’.  This  assumption implies  the 
conditional  independence  of  potential  outcomes  and  treatment  assignment  given 
observables, and thus means that the selection into treatment can be only attributed to 
the factors which are observable and observed by the researcher / evaluator. In order to 
minimise the risk of deviations, the evaluator should review available empirical evidence 
and literature and conduct appropriate and dedicated analysis69. 

Another  possible  distortion  is  the  self-selection  bias.  This  may  appear  if  firms  that 
anticipated  participation  in  the  programme  already  adjusted  their  performance  prior  to 
the start of the programme e.g., to comply with programme eligibility criteria. In such a 
situation,  even  if  the  control  group  was  very  similar  to  programme  participants, 
comparisons  of  both  groups  just  before  participation  could  lead  to  a  significant  control 
bias. The important consequence for ex-post evaluation is that this type of bias should be 
recognised  and  reduced  or  eliminated  (if  possible)  before  the  impact  assessment  is 
undertaken.  

A  homogeneous  treatment  effect  is  where  the  programme  effect  is  assumed  to  be 
constant  across  all  individuals  /  units.  The  homogeneous  treatment  effect  means 
outcomes  for  programme  participants  and  non-programme  participants  are  two  parallel 
curves only differing in level. Although such an effect would greatly facilitate analysis of 
programme  impacts,  assuming  homogeneous  treatment  effect  is  inconsistent  with 
empirical evidence and leads to numerous estimate biases.  Heterogeneous treatment 
effects  assume  that  support  impact  varies  across  individuals/units  (a  possible  effect  of 
an  observable  component  or  as  a  part  of  unobservables).  Yet,  contrary  to  the 
homogenous treatment effect, this does not enable extrapolation to all population strata 
of  evaluated  units.  Furthermore,  in  such  a  situation  (including  selecting  unobservables) 
applying  popular  econometric  OLS  estimators  (even  after  controlling  for  observable 

69  In  the  case  of  PSM,  the  analysis  could  inspect  how  the  observations  are  distributed  across  the 
propensity  score  common  support  and  how  sensitive  the  estimates  are  with  respect  to  the 
utilisation of observations  in the tails of  the  common support (see for further details Black D. A., 
Smith  J.A.    (2004),  How  robust  is  the  evidence  on  the  effects  of  college  quality?  Evidence  from 
matching,  Journal  of  applied  econometrics).  Another  option  from  the  literature  is  to  carry  out  a 
sensitivity  analysis  assessing  whether  (and  to  what  extent)  the  estimated  average  treatment 
effects are robust to possible deviations from the conditional independence assumption (see Ichino 
A., Mealli F. and Nannicini T. (2007), From Temporary Help Jobs to Permanent Employment: What 
Can We Learn from Matching Estimators and their Sensitivity?, Journal of Applied Econometrics). In 
the  case  of  Regression  Discontinuity  Design,  see  the  aforementioned  paper  from  De  Blasio  et  alii 
(2017). 

60 

 
                                                 
 Support to SME Initiative evaluation in the Member States - Final Report 

differences  in  characteristic)  would  be  inconsistent.  For  instance,  with  RDD  the  ‘fuzzy 
approach’ is used to check, analyse and consider non-linearity. PSM uses many variables 
also referring to turnover and employment (expressed as dummies) to reduce the risk of 
heterogeneous effects. 

Source: t33 

Estimated  time  and  budget.  The  workload  required  for  an  evaluation  using  quasi-
experimental  design  is  estimated  to  be  of  approximately  25  to  50  person-days  of  a 
statistical/econometric expert. This only includes the time required to decide on the most 
appropriate  techniques,  prepare  the  data  for  analysis,  run  calculations  testing  various 
model  specifications,  report  on  outcomes  of  the  analysis.  It  excludes  data  collection 
including  data  gathering  and  any  efforts  needed  to  ensure  matching  between  multiple 
data sources. Required workload depends on a number of factors including the number of 
SMEi actual recipients and timing of support. 

61 

 
 
 
 
Getting in touch with the EU 

In person 

All over the European Union there are hundreds of Europe Direct information centres. You can find the address 
of the centre nearest you at: https://europa.eu/european-union/contact_en 

On the phone or by email 

Europe Direct is a service that answers your questions about the European Union. You can contact this service: 

– by freephone: 00 800 6 7 8 9 10 11 (certain operators may charge for these calls), 

– at the following standard number: +32 22999696 or  

– by email via: https://europa.eu/european-union/contact_en 

Finding information about the EU 

Online 

Information about the European Union in all the official languages of the EU is available on the Europa website 
at: https://europa.eu/european-union/index_en 

EU publications 

You  can  download  or  order  free  and  priced  EU  publications  at:  https://publications.europa.eu/en/publications. 
Multiple  copies  of  free  publications  may  be  obtained  by  contacting  Europe  Direct  or  your  local  information 
centre (see https://europa.eu/european-union/contact_en). 

EU law and related documents 

For  access  to  legal  information  from  the  EU,  including  all  EU  law  since  1952  in  all  the  official  language 
versions, go to EUR-Lex at: http://eur-lex.europa.eu 

Open data from the EU 

The EU Open Data Portal (http://data.europa.eu/euodp/en) provides access to datasets from the EU. Data can be 
downloaded and reused for free, for both commercial and non-commercial purposes. 

62 

 
 
 
 
 
 
K
N
-
0
2
-
2
1
-
5
0
9
-
E
N
-
N

       ISBN: 978-92-76-36159-6