Study on e-procurement measurement and benchmarking Markt 2011/097/C : lot 1 : public procurement performance indicators : D4 final study report.
Date: 2016-06-22
Institutions: Directorate-General for the Internal Market and Services
Subjects: benchmarking, cross-frontier data flow, electronic document, electronic government, information technology, public contract

Study on e-Procurement 
Measurement and 
Benchmarking MARKT 
2011/097/C 

Lot 1 – Public Procurement 
Performance Indicators 

D4- Final Study Report- Final  

m
o
c
.
i
g
-
c
d

i
.

w
w
w

.

1
8
8
7
8
8
9
8
0
5
.
F

.

0
0
4
4

.

.

5
3
9
8
0
5
P.

A
S
U
1
0
7
1
0
A
M

,

m
a
h
g
n
m
a
r
F

i

t
e
e
r
t
S
n
e
e
p
S
5
:
s
r
e
t
r
a
u
q
d
a
e
H

June 27, 2013 

The opinions expressed in this study are those of the authors and do not 
necessarily reflect the views of the European Commission 

l

a
b
o
G

l

  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
The information and views set out in this study are those of the authors and do not necessarily 
reflect  the  official  opinion  of  the  Commission.  The  Commission  does  not  guarantee  the 
accuracy of the data included in this study. Neither the Commission nor any person acting on 
the  Commission’s  behalf  may  be  held  responsible  for  the  use  which  may  be  made  of  the 
information contained therein. 

DOI: 10.2780/94897 

ISBN: 978-92-79-29926-1 

© European Union, 2013. 

Reproduction is authorised provided the source is acknowledged. 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Author(s) 

Gabriella Cattaneo, Rosanna Lifonti, Stefania Aguzzi, Marcello 
Bardellini, IDC 

Con Sadee, Thamar Tax, Capgemini Consulting 

Deliverable 

D4 – Final Study Report  

Date of delivery 

June 27, 2013 

Version 

3.0 

Addressee officers  

Tudor Ion 

Policy Officer – Desk Officer – Electronic Procurement  

European Commission 

Directorate General Internal Market and Services 

Unit C4 – Economic Analysis and e-Procurement 

Office: SPA2 06/040  

Tel: +32-2-29.68524 

Email: tudor.trestieni.ion@ec.europa.eu 

Contract ref. 

MARKT/2011/097/C4/ST/OP LOT1 

3 

 
 
 
 
 
 
2. 4. 

2.5. 

Table of Contents 

E xe cut iv e   Sum ma r y  

8 

Overview .............................................................................................................................. 8 
Towards the full adoption of e-Procurement ........................................................................ 8 
Feasibility and Availability of Indicators ............................................................................... 8 
The Plan for the Pan-European e-Procurement Measurement System .............................. 10 

1. 

Int rodu ct ion  

1. 1.  Ab out  th is   rep o rt  

1. 2.  O bj ect iv e s  an d  st ruc t ur e o f th e  r epo rt  

1. 3.  St ru ctu re  of  th e  r epo rt  

1. 4.  Ke y  Con c ept s  and  F r am ew or k of   An a l ys is  

1. 5.  Su mm ar y  of  the  I ndi c at o r  S ys te m  

2. 

M ain r e su lts  of  th e T ri al  

2. 1.  De s cr ipt ion  of  the  t ri al  

2. 2.  Su mm ar y  of  R es ult s  

12 

12 

12 

13 

13 

15 

18 

18 

21 

2.2.1. 
2.2.2. 
2.2.3. 

Assessment of availability ..................................................................................... 21 
Availability of data for Take-up indicators ............................................................. 21 
Availability of data for Performance indicators ...................................................... 24 

2. 3.  Ca lc ul ati on  of th e  In dic ato r s  

T ake- up  of  e - P ro cu re ment  i n  V alu e   -   E U  2 0 11- 2 0 12  

25 

25 

EU level Performance Indicators ................................................................................... 30 
E-Submission Ease of Access Indicator ............................................................... 30 
E-Submission Transparency Indicator .................................................................. 32 

2.5.1. 
2.5.2. 

2. 6. 

Le ss ons  L e ar ned  f ro m th e  tr ia l  

37 

Level of participation and motivation ..................................................................... 37 
2.6.1. 
2.6.2.  Data Collection Process ........................................................................................ 40 

3. 

As s e s sm ent  of  th e  in dic ato r  s ys t em  

3. 1.  Ap p ro ac h to  f ea s ibi li t y  and  qu a lit y  a ss e ss ment  

3. 2.  As s e s sm ent  of  T ak e - up I ndi c ato r s  

45 

45 

47 

Assessment of Top-level Take-up Indicator of e-Procurement ............................. 47 
3.2.1. 
Assessment of Take-up Indicator by type of purchase ......................................... 48 
3.2.2. 
Assessment of Take-up Indicator by type of procedure ....................................... 48 
3.2.3. 
Assessment of Take-up Indicator above/below the EU threshold ........................ 49 
3.2.4. 
3.2.5. 
Assessment of Take-up Indicator by electronic procedure ................................... 49 
3.2.6.  Main Barriers to Take-up indicators ...................................................................... 50 
Indicator of take-up of e-Procurement by SMEs ................................................... 51 
3.2.7. 
3.2.8. 
Indicator of take-up of e-Procurement by Cross-Border suppliers........................ 52 
3.2.9.  Main Barriers to Take-up indicators for SMEs and Cross-Border suppliers ......... 53 

3. 3.  As s e s sm ent  of   Pe rfo rm an ce  i ndi c ato r s  

55 

3.3.1. 
3.3.2. 
3.3.3. 
3.3.4. 
3.3.5. 

Assessment of the e-Submission Price Reduction Indicator ................................ 55 
Barriers to the e-Submission Price Reduction Indicator ....................................... 56 
Assessment of the e-Submission Reduction of Litigation Indicator ...................... 57 
Barriers to the e-Submission Reduction of Litigation Indicator ............................. 58 
Assessment of the e-Tendering Efficiency Improvement Indicator....................... 59 

4 

 
3.3.6. 
3.3.7. 
3.3.8. 
3.3.9. 
3.3.10. 

Barriers to the e-Tendering Efficiency Improvement Indicator .............................. 59 
Assessment of the Ease of Access Indicator ........................................................ 60 
Barriers to the Ease of Access Indicator ............................................................... 62 
Assessment of the Transparency Indicator ........................................................... 62 
Barriers to the Transparency Indicator .............................................................. 63 

3. 4.  Av a i l abi lit y  of  Ad d iti ona l  Dat a  fr om  oth e r  publ ic   sou rc e s  

63 

3. 5.  As s e s sm ent  of   Sc al a bil it y,  Co mp ar ab il it y  and  R ep r es ent at iv en es s  64 

3.6. 

Assessment of scalability ...................................................................................... 64 
3.5.1. 
3.5.2.  Main barriers to scalability and representativeness .............................................. 65 
Conclusions on the revision of the System of Indicators ............................................... 66 
3.6.1.  Revision of Take-up indicators .............................................................................. 66 
3.6.1.  Revision of Take-up indicators by SMEs and Cross-border Suppliers ................. 67 
3.6.2.  Revision of Performance indicators ...................................................................... 68 
3.6.3.  Update of the System of Indicators ....................................................................... 69 

4. 

Dev elo pm ent  of  a  P a n - Eu ro pe an   Pe rf or ma nc e  me a su re me nt  S y ste m
72 

4. 1.  O v erv i ew  

4. 2.  Sug ge st ed   Ap pr oa ch  

72 

74 

4.3. 

Governance of the Measurement System ..................................................................... 75 

4. 4.  Roa dm ap  of t he  M ea s ur em ent   S yst em  

77 

4. 5.  Roa dm ap  of  Ind ic at or s  

77 
Plan for the 1st year ............................................................................................... 77 
Plan for the Roll-out of the Measurement system ................................................. 79 

4.5.1. 
4.5.2. 

4. 6. 

Imp le me nta tio n  Pl an   for  th e n e xt M ea su r e ment  C yc l e  

81 

4.6.1.  Overview and main activities................................................................................. 81 
4.6.2.  Coverage of the EU 28 .......................................................................................... 84 
Approach to Contracting Authorities ..................................................................... 85 
4.6.3. 

4. 7.  Con clu s ion s  

An n ex  

Def in itio ns  of  P ro cu r e m ent  Co nc ept s  

Def in itio ns  on  T ak e  u p of  e - P ro cu r em ent  

Def in itio ns  on   Pe rfo r man c e o f th e  pl atfo r m  

Def in itio ns  a dd ed  du rin g t he  T ri al  

86 

88 

88 

91 

93 

94 

5 

 
 
 
Figures 

Figure 1 - The e-Procurement Process Phases: Pre- and Post-Award ......................................... 14 

Figure 2 - The e-Procurement Ecosystem: Main Stakeholders ..................................................... 15 

Figure  3  -  Estimate  of  take-up  of  e-Procurement  in  the  EU,  2011-2012: 
high-low range (€ Billion) ............................................................................................................... 29 

Figure 4 - Ease of Access Indicator for the Trial Platforms ............................................................ 31 

Figure  5  –  Trial  results:  Transparency  Indicator  average  score  by 
Platform .......................................................................................................................................... 36 

Figure 6 - Governance and structure of the Measurement System ............................................... 76 

Tables 

Table 1 - Summary of Take-Up Indicators ...................................................................................... 16 

Table 2 - Summary of Performance Indicators .............................................................................. 17 

Table 3 - List of Trial Platforms by level of e-Submission solution ................................................. 20 

Table 4 - Level of technology maturity of e-Submission solutions ................................................. 20 

Table  5  -  Classification  of  criteria  of  Data  availability  and  correlated 
barriers ........................................................................................................................................... 21 

Table 6 - Summary View of Availability of data for take-up indicators ........................................... 23 

Table 7 - Summary View of Availability of data for Performance Indicators .................................. 24 

Table 8 – Low Range Estimate of EU level Take-up (€ Billion) ...................................................... 29 

Table 9 - High Range estimate of EU level Take-up ...................................................................... 29 

Table  10  –  Ranking  of  Trial  Platforms  by  Level  of  Ease  of  Access 
Indicator ......................................................................................................................................... 30 

Table 11 - Data collection for Transparency Indicator: level of availability 
of information by type (100% = Full availability) ............................................................................ 34 

Table 12 - Transparency Indicator - Measurement Scale .............................................................. 35 

Table 13 Trial results: Transparency Indicator scores by platform and by 
category ......................................................................................................................................... 36 

Table 14 - Correspondence to plan of the trial participants ........................................................... 37 

Table 15 Characteristics of the Trial Participant ............................................................................. 39 

Table 16 – Data collection process: main steps ............................................................................. 40 

Table 17 - Assessment of Feasibility of Take-up Indicators - I ....................................................... 48 

Table 18 - Assessment of Feasibility of Take-up Indicators - II ...................................................... 49 

Table 19 - Main Barriers to Take-up Indicators .............................................................................. 51 

Table 20  - Assessment of take-up indicators  by  SMEs and cross-border 
suppliers ......................................................................................................................................... 53 

Table  21  -  Main  Barriers  to  Take-up  Indicators  for  SMEs  and  Cross-
border suppliers ............................................................................................................................. 55 

Table 22 - Assessment of the e-Submission Price Reduction Indicators....................................... 56 

Table 23 - Main Barriers to e-Submission Price Reduction Indicator ............................................ 57 

6 

 
 
Table 24 - Assessment of e-Submission Reduction of Litigation Indicator .................................... 58 

Table 25 - Main Barriers to e-Submission Reduction of Litigation Indicator .................................. 58 

Table 26 - Assessment of e-Tendering Efficiency Improvement Indicator ..................................... 59 

Table 27 - Main Barriers to e-Tendering Efficiency Improvement Indicator ................................... 60 

Table 28 - Assessment of Ease of Access and Transparency Indicators ...................................... 62 

Table 29 - Main Barriers to Ease of Access Indicator .................................................................... 62 

Table 30 - Assessment of Transparency Indicator ......................................................................... 62 

Table 31 - Main Barriers to Transparency Indicator ....................................................................... 63 

Table 32 - Additional Data collection needs for Take-up indicators – Data 
needed from MS or Eurostat .......................................................................................................... 64 

Table 33 - Additional data collection needs for Performance indicators ........................................ 64 

Table 34 - Hierarchy of the System of Indicators ........................................................................... 65 

Table 35 - Revision of take-up indicators ....................................................................................... 66 

Table  36  -  Revision  of  take-up  indicators  by  SMEs  and  Cross-border 
Suppliers ........................................................................................................................................ 68 

Table 37 - Revision of Performance Indicators .............................................................................. 69 

Table 38  - Updated Take-Up Indicators......................................................................................... 70 

Table 39 - Updated e-Procurement Performance Indicators ......................................................... 71 
Table  40  -  List  of  e-Procurement  Indicators  to  be  measured  –  1st  Year 
Plan ................................................................................................................................................ 79 

Table 41 List of e-Procurement Indicators to be measured – Roll-out Plan .................................. 80 

Table 42 – Indicative workplan timing for next measurement cycle ............................................... 84 

7 

 
  
E X E C U T I V E   S U M M A R Y  

Overview 

in  2012-2013.  The  report  presents 

This  report  presents  the  main  results  of  the  study  on  e-Procurement 
Measurement  and  Benchmarking  carried  out  by  IDC  and  Capgemini 
take-up  and 
Consulting 
performance indicators designed by the study team, the assessment of 
their feasibility and value added based on a proof-of-concept trial, and a 
high-level implementation  plan for the development of a  Pan-European 
e-Procurement Performance Measurement system. The report presents 
also  an  estimate  of  the  current  level  of  take-up  of  e-Procurement  in 
Europe. 

the 

The  main  goal  of  this  plan  is  to  measure  the  performance  of  the  e-
Procurement  systems,  and  the  achievement  of  the  main  goals  of  e-
Procurement policies, first of all the smooth migration from paper-based 
to  electronic  procedures  across  Europe.  The  development  of  this  e-
Procurement  Performance  Measurement  System  promises  to  do  so, 
and  to  build  a  community  of  stakeholders  able  to  take  advantage  of 
these data to achieve fully the benefits of e-Procurement adoption.  

Towards the full adoption of e-Procurement 

An  important  result  of  our  study  is  the  discovery  of  the  still  marginal 
take-up of e-Procurement in most countries. According to our estimate, 
the  overall  value  of  e-Procurement  in  the  EU  in  2012  was  between 
€185-220 Bn, corresponding to the take-up of around 12% of the value 
of  total  public  procurement,  with  a  growth  rate  from  the  previous  year 
around  8%.    However,  only  5  Member  States  (MS)  (the  UK,  Lithuania, 
Sweden,  Portugal  and  Ireland)  present  take-up  rates  substantially 
higher  than  10%,  while  several  MS  are  around  or  lower  than  1%. This 
means also that public procurement authorities pay  little  attention to  e-
Procurement,  or  consider  it  only  an  alternative  service  channel,  rather 
than  a  business  process  with  high  benefits  of  efficiency  and 
effectiveness.  

Feasibility and Availability of Indicators 

There  are  relevant  availability  and  feasibility  barriers  to  the  systematic 
measurement of e-Procurement indicators. They include: 

  Technology  barriers:  because  of  the  high  number  of  e-
Procurement  platforms  with  low  technology  maturity,  where 
there is low visibility on value data and type of contracts; 

  Organizational  barriers  at  platform  and  country  level:  

monitoring  and  data  collection  on  e-Procurement  is  under-
developed, at platform level, country level, and EU level. Private 
platforms tend to have more advanced monitoring systems than 
public platforms, but are not authorized to communicate data to 
third parties. 

  Policy  and  regulatory  barriers:  the  transparency  of  public 
procurement  flows,  particularly  at  regional  and  local  level,  is 
quite  low  in  most  countries,  with  Contracting  Authorities  (CA) 

8 

 
protecting their autonomy and uninterested to share information 
and  data.  There  is  little  systematic  monitoring,  and  very  little 
good quality, comparable data about public procurement on key 
aspects  such  as  SMEs  wins.  This  prevents  the  collection  of 
complementary  data  needed  to  benchmark  the  indicators  and 
does  not  create  incentives  for  CA  to  collaborate  to  the 
measurement system.  

  The  fragmentation  of  the  e-Procurement  landscape  by  country 
means  that  each  MS  follows  its  own  development  path,  and 
there  are  difficulties  in  finding  common  denominators,  building 
and agreeing on comparable indicators.  

Given  these  problems,  our  recommendation  is  to  focus  the  system  of 
indicators on a short list of priorities, privileging indicators on take-up in 
value  and  volume,  as  shown  by  the  following  table.  We  suggest  also 
including an indicator on the take-up of e-Ordering, which seems to be 
than  pre-award  e-
practiced  by  a  higher  number  of  countries 
Procurement. Performance indicators in the short term should be limited 
to  two  aspects  easily  measurable  (ease  of  access  and  transparency) 
and relevant for usability. The other Performance indicators need further 
research.  

The e-Procurement Indicators 

Policy Goals 

EU Top level Take-up indicators 

Full transition to e-
Procurement 

E-Procurement take-up in value (% of total public procurement)  

E-Procurement take-up in volume (% of total number of 
contracts) 

Full participation of SMEs 
to e-Procurement 

E-Procurement take-up by SMEs in value and volume (over 
total public procurement)  

Enablement of cross-
border e-Procurement 

E-Procurement take-up by cross-border economic operators (in 
value and volume (over total public procurement) 

Policy Goals 

Other EU Take-up indicators 

Full transition to e-
Procurement 

E-Procurement take-up in value and volume by type of 
purchase (% supplies, services, public works)  

E-Procurement take-up in value and volume above/below the 
EU threshold (% above/below) 

E-Auction take-up in value and volume (total value and number 
of contracts) 

E--Ordering take-up in value and volume (total value and 
number of orders) 

Policy Goals 

EU Top level Performance indicators 

Improve ease of access of 
public procurement 

E-Submission Ease of access indicator (based on 
authentication requirements) 

Improve transparency of 
public procurement 
process 

E-Submission Transparency Indicator (based on the type of 
information and communication given to buyers and suppliers) 

Policy Goals 

EU Top level Performance indicators –  requiring further 

9 

 
 
Improve Effectiveness of 
public spending 

Improve efficiency of 
procurement processes 

research 

E-Submission Price Reduction Indicator (average difference 
between the asking price (reserve price) and the award price of 
contracts, as a measure of savings)  

E-Submission Reduction of Litigation indicator (average 
number of appeals for e-Procurement contracts) 

E-Tendering Efficiency Improvement  indicator for Contracting 
Authorities (based on the average time needed for the pre-
award tendering process) 

E-Tendering Efficiency Improvement  indicator for economic 
operators (based on the average time needed for the pre-
award tendering process) 

The Plan for the Pan-European e-Procurement Measurement System 

Based  on  these  considerations,  in  order  to  overcome  the  main 
availability  barriers,  we  suggest 
the  Measurement  System 
development strategy should be articulated as follows: 

that 

  Aim  for  measurement  of  the  updated  list  of  indicators 
selected by this study, with priority to take-up indicators in the 
short  term.  Further  work  should  be  done  on  performance 
indicators to improve their feasibility. 

formalized  by 

  Set  up  data  collection  for  the  basic  indicators  through  a 
standardized  contract  award  notice, 
the 
European  Committee  for  Standardization  (CEN),  and  therefore 
accepted  by  all  the  MS  as  a  national  standard.  The  contract 
award notice is a mandatory requirement for all public contracts 
above  the  EU  threshold  and  will  be  updated  in  view  of  the 
adoption of the new Public Procurement Directives. This can be 
complemented  by  continuing  data  collection  with  Panels  of 
platforms and Contracting Authorities.  

 

Implement ways to incentivise and reward the collaboration 
of  platforms  and  Contracting  Authorities, 
for  example 
providing  personalised  “dashboards”  and  statistical  reports 
based  on  the  data  collected.    This  will  be  supported  with  the 
launch  of  a  web  portal  acting  as  a  key  reference  point  of  the 
Measurement system, providing access to all the data collected, 
publishing the results of the indicators measurement, supporting 
communication  and  promotion  initiatives  of  the  Measurement 
system.  

The use of a CEN standard as the main pillar of the data collection for 
the indicators presents several advantages: 

 

 

It  will  solve  the  availability  and  scalability  problems,  since  the 
data  should  in  principle  cover  the  entire  universe  of  contracts 
awarded above the EU threshold; 

It  will  provide  common  standard  definitions  for  measuring  e-
Procurement  at  national  level,  which  may  in  the  future  be 
extended by MS to contracts of value below the EU threshold.  

The  suggested  roadmap  of  the  Measurement  system  is  articulated  in 
the following way: 

10 

 
  A measurement cycle (indicatively from the fall of  2013 to the 

fall of 2014) in which: 

o  DG  MARKT  and  the  Benchmarking  Organization  will 
work  with  CEN  to  develop  the  standardized  contract 
award notice;  

o 

In  parallel,  the  Benchmarking  organization  will  set-up 
and  organize  the  complementary  data  collection  with 
the  Panels  and  produce  the  “Landscaping”  and  “State 
of Play” reports. 

  The  roll-out  of  the  Measurement  System  (indicatively  from 
2015) when the data collection will be fully implemented through 
the TED forms. This leads to two possible scenarios: 

  DG  MARKT  will  become  directly  responsible  for  the 
elaboration  of  indicators  and  production  of  “state  of 
play”  reports,  based  on  TED  data,  as  well  as  the 
publication of results on the web and the interaction with 
platforms  and  Contracting  Authorities.  In  this  scenario 
the  role  of 
the  Benchmarking  Organization  could 
disappear or be greatly reduced.  

  DG MARKT may decide to continue to use a contractor 
to  manage  the  Measurement  system,  design  and  test 
new  indicators,  run  the  web  portal,  and  manage  the 
interaction  with 
the  stakeholders,  continuing  also 
complementary  data  collection  activities  with  Panels  of 
platforms  and  Contracting  Authorities.  In  this  scenario 
the  role  of  the  Benchmarking  Organization  will  remain 
central for the Measurement system.  

The main goal of this plan is not measurement for measurement's sake, 
but the assessment of progress towards relevant policy priorities, those 
targeted by the new Public Procurement Directives. The development of 
this  e-Procurement  Performance  Measurement  System  promises  to  do 
so, and to build a community of stakeholders able to take advantage of 
these  data  and  understanding  to  achieve  fully  the  benefits  of  e-
Procurement adoption.  

11 

 
1 .  I N T R O D U C T I O N    

1.1.  About this report 

This  deliverable  is  the  Final  Report  (D4)  of  the  Study  “e-Procurement 
Measurement  and  Benchmarking  –  Lot  1  –  Public  Procurement 
Performance  Indicators”  entrusted  by  the  European  Commission, 
Directorate  General  Internal  Market  and  Services,  Unit  Economic 
Analysis  and  e-Procurement  (DG  MARKT  hereinafter),  to  IDC  and 
Capgemini Consulting.  

The  deliverable  is  without  prejudice  to  the  offer  made  by  IDC  and 
Capgemini Consulting for this project. The document may be updated at 
the request of the Commission to ensure that the objectives of the study 
are achieved, within the limits established by the contract. 

The main goal of this project is to design, set up and test a systematic 
and comparable monitoring capacity of the e-Procurement infrastructure 
and  performance  across  Europe.  The  project  includes  two  main 
Workstreams: 

1.  Mapping and measuring the e-Procurement landscape in Europe; 

2.  To  design, 
recommend  a  set  of  e-Procurement 
Performance indicators for a Pan-European Performance Measurement 
System.  

test  and 

This  deliverable  completes  the  achievement  of  the  objectives  of 
workstream  2,  building  on  the  results  of  D3  "Performance  Indicators 
Report".  

1.2. Objectives and structure of the report 

More specifically, the main objectives of Workstream 2 are the following:  

  To  identify  the  main  requirements  of  the  key  stakeholder 
categories  for  monitoring  and  performance  indicators  of  e-
Procurement; 

  To define the criteria for the measurement of the performance of 
e-Procurement  systems,  based  on  the  key  success  factors/ 
barriers,  taking  into  account  the  different  points  of  view  of  the 
main stakeholder categories; 

  To design a set of idealized indicators (an indicator system) with 

the following characteristics:  

o  To  be  able  to  measure  the  performance  (take-up, 
efficiency  and  effectiveness)  of  the  e-Procurement 
systems, at present and in time, feeding trend analysis; 

o  To be based on data generated from the e-Procurement 
systems  themselves  (either  automatically,  or  by  hand 
with a reasonable effort), produced through sustainable 
information flows; 

o  To  be  able  to  measure  the  achievement  of  the  main 
goals  of  e-Procurement  policies,  first  of  all  the  smooth 
migration  from  paper-based  to  electronic  procedures 

12 

 
across  Europe;  but  also    the  level  of  participation  of 
SMEs and cross-border bidders in Public Procurement.  

o  To  be  linked  with  the  assessment  of  the  key  success 
factors  and  barriers  to  e-Procurement  take-up,  able  to 
provide  insights  for  practical  and  policy  action  by  main 
stakeholders.  

The  deliverable  D3  "Performance  Indicators  Report"  achieved  these 
objectives. This report addresses the other objectives of Workstream 2 
and more specifically: 

  To test and fine-tune the indicator system, through a trial to be 
organized  with  the  collaboration  of  a  group  of  e-Procurement 
platforms in a sample of Member States (proof-of-concept trial); 

  To  review  the  design  of  the  measurement  process  and  of  the 
indicator  system,  based  on  the  findings  from  the  trial  and  the 
feedback from the participating platforms;   

  To  propose  a  high-level  implementation  plan  for  the  roll  out  of 
this  Pan-European  e-Procurement  Performance  Measurement 
system,  indicating  the  actions  to  be  taken  by  the  main 
stakeholders.  

1.3. Structure of the report 

The report is structured as follows: 

  Chapter  1  presents  the  main  goals  of  the  report,  the  key 

concepts and framework of analysis; 

  Chapter  2  presents  the  main  results  of  the  trial  about  the 
availability of data for the indicator system and the lessons 
learned about the feasibility of the data collection process; 

  Chapter  3  presents  the  assessment  of  the  feasibility, 
scalability  and  reliability  of  the  indicators,  based  on  the 
results  of  the  trial  and  the  results  of  the  previous  research 
on the Landscape and State of Play of e-Procurement in the 
EU 27. It presents also the update of the Indicator System, 
based on the results of the assessment; 

  Chapter  4  presents 

the  suggested  approach 

the 
development 
the 
implementation  plan  and  the  suggested  roadmap  to  its 
execution. 

a  measurement 

capacity, 

to 

of 

  The Annex includes: 

o  The list of definitions used in the Trial  

o  The Trial results report 

1.4. Key Concepts and Framework of Analysis 

This study is based on the following definitions.  

13 

 
According  to  the  proposed  Directive  on  Public  Procurement1,  public 
procurement is: 

The  purchase  or  other  forms  of  acquisition  of  public  works, 
supplies  or  services,  by  one  or  more  contracting  authorities  from 
economic  operators  chosen  by  those  contracting  authorities, 
whether or not the public works, supplies or services are intended 
for a public purpose. 

Within  the  scope  of  this  study,  we  define  electronic  procurement  as 
follows:  

E-Procurement refers to the use of electronic communications and 
transaction  processing  by  government  institutions  and  other 
public sector organizations when buying supplies and services or 
tendering  public  works.  This  includes  the  replacement  of  paper-
based procedures through the e-Procurement chain.  

The e-Procurement process covers two main phases (Figure 1): the pre-
award  phase  and  the  post  award  phase.  Pre-award  comprises  all  the 
sub-phases of e-Procurement until the award of the contract (publication 
of notices, access to tender documents, submission of bids, evaluation 
of the proposals and the award of the contract). Post-award comprises 
all  the  sub-phases  of  procurement  after  the  award  of  the  contract 
(ordering,  invoicing  and  payment).  The  switch  over  from  traditional, 
paper-based  processes  to  electronic  processes  may  be  end-to-end,  or 
limited to some phases of the process.  

Figure 1 - The e-Procurement Process Phases: Pre- and Post-Award  

Source: IDC 2012 

Within  the  framework  of  this  study,  and  in  agreement  with  the  EC 
Green  Paper  on  e-Procurement,  we  will  define  as  the  minimum 
package  for  an  e-Procurement  offering  the  electronic  provision  of 
the following services:  

•  E-Notification: that is the online publication of tender notices; 

•  E-Access:  that  means  the  online  access  to  tender  documents 
24/7 in an automatic manner; 

•  E-Submission: that means the online submission of tenders. 

Since  the  majority  of  procurement  contracts  are  processed  through  a 
mix  of  online  and  offline  phases,  we  will  use  the  same  definition  as  a 
threshold for the identification and measurement of contracts processed 
online.  

The definition of e-Procurement take-up is the following: 

1 Proposal for a Directive on Public Procurement, Brussels 20-12-2011, COM (2011) 896 
final 

14 

Pre-AwardPost-Awarde-NotificationE-Accesse-Invoicinge-Paymente-Submissione-Evaluatione-Awarde-Ordering 
 
                                                      
The  take-up  of  e-Procurement  is  measured  as  the  number  and 
value  of  public  procurement  contracts  that  have  been  processed 
electronically,  as  a  minimum  up  to  and  including  at  least  e-
Submission,  as  a  proportion  of  the  total  public  procurement 
includes  all 
contracts  processed 
procurement contracts, above and below the EU threshold.  

in  the  same  period.  This 

The  measurement  of  take-up  can  be  segmented  by  type  of  economic 
(SMEs  and  cross-border  suppliers) 
the 
to  monitor 
operators 
achievement of important EU policies for e-Procurement adoption. 

However,  a  simple  description  of  the  electronic  process  phases  is 
insufficient to reflect the complexity of the e-Procurement system, which 
is  essentially  focused  on  the  interaction  between  public  buyers  and 
suppliers through an interactive buy-and-sell cycle, exploiting electronic 
technologies.  This  is  the  framework  of  analysis  developed  in  order  to 
map the e-Procurement landscape, illustrated in the following figure.  

Figure 2 - The e-Procurement Ecosystem: Main Stakeholders 

Source: IDC 2012 LMEs = Large and Medium Enterprises; SMEs = Small and 
Medium Enterprises 

The Figure 2 above shows (in a simplified manner) the structure of the 
e-Procurement  ecosystem,  which  is  centred  around  the  relationship 
the  e-
between  public  buyers  and  suppliers, 
Procurement  entities,  who  act  as  specialised  intermediaries  providing 
the e-Procurement services.  

intermediated  by 

1.5. Summary of the Indicator System 

The  Table  1  and  Table  2  below  present  the  list  of  the  main  indicators 
included  in  the  Indicator  System  tested  in  the  trial  and  the  main  policy 
goals  they  aim  to  measure.  The  indicators  are  classified  in  3  main 
groups: 

  Top level take-up indicators (the most important indicators of e-

Procurement take-up in value and in volume); 

15 

Advertising PortalsE-Procurement Platforms / Pre-AwardE-Procurement Platforms / Post-AwardNational AuthoritiesFederal AuthoritiesRegional AuthoritiesLocal AuthoritiesCentral Procurement Bodies DOMESTIC SUPPLIERSLMEsSMEsLMEsSMEsFOREIGN SUPPLIERSREGULATIONPROCUREMENT PROCEDURESTYPE OF PURCHASETECHNOLOGYELECTRONIC PROCEDURESELECTRONIC TOOLSCONTRACTING AUTHORITIES AND ENTITIESVALUE THRESHOLDCentral Purchasing Bodies 
 
  Other  take-up  indicators  (segmenting  the  take-up  indicator  in 
value and in volume according to several variables, like type of 
purchase, procedure, electronic procedure and EU threshold); 

  Performance 

indicators, 
performance of the e-Procurement system.  

focused  on 

the  benefits  and 

Table 1 - Summary of Take-Up Indicators 

Policy Goals 

EU Top level Take-up indicators 

E-Procurement take-up in value (% of total public procurement)  

Full transition to e-Procurement 

E-Procurement take-up in volume (% of total number of 
contracts) 

Full participation of SMEs to e-
Procurement 

E-Procurement take-up by SMEs in value and volume (over 
total public procurement)  

Enablement of cross-border e-
Procurement 

E-Procurement take-up by cross-border economic operators (in 
value and volume (over total public procurement) 

Policy Goals 

Other EU Take-up indicators 

Full transition to e-Procurement 

E-Procurement take-up in value and volume by type of 
purchase (% supplies, services, public works)  

E-Procurement take-up in value and volume above/below the 
EU threshold (% above/ below)  

E-Procurement take-up in value and volume by type of 
procedure (Competitive Dialogue, Framework Agreement, Mini 
Competition, Open Procedure, Negotiated Procedure, 
Simplified Procedure)  

E-Procurement take-up in value and volume by type of 
electronic procedure: e-Auction, DPS, e-Marketplace (value 
and number of contracts)  

Source: D3, Performance Indicators Report IDC 2013 

16 

 
 
 
Table 2 - Summary of Performance Indicators 

Policy Goals 

EU Top level Performance indicators 

Improve Effectiveness of public 
spending 

Improve efficiency of 
procurement processes 

E-Submission Price Reduction Indicator (average difference 
between the asking price (reserve price) and the award price of 
contracts, as a measure of savings)  

E-Submission Reduction of Litigation indicator (average 
number of appeals for e-Procurement contracts) 

E-Tendering Efficiency Improvement  indicator for Contracting 
Authorities (based on the average time needed for the pre-
award tendering process) 

E-Tendering Efficiency Improvement  indicator for economic 
operators (based on the average time needed for the pre-
award tendering process) 

Improve ease of access of public 
procurement 

E-Submission Ease of access indicator (based on 
authentication requirements) 

Improve transparency of public 
procurement process 

Transparency Indicator of e-Procurement for buyers and 
economic operators (based on the type of information and 
communication given to buyers and suppliers) 

Source: D3, Performance Indicators Report IDC 2013 

17 

 
 
2 .  M A I N   R E S U L T S   O F   T H E   T R I A L  

2.1.  Description of the trial 

This chapter summarizes the main results of the proof-of-concept trial of 
the  e-Procurement  Measurement  System  held  from  January  to  April 
2013.  The 
trial  was  designed  and  executed  by  Capgemini 
Consulting. The participants of the trial  were members (and associated 
platforms) of the ‘Advisory Panel for the development of a measurement 
system of e-Procurement in the EU’.  

The main objectives of the trial were: 

  To  test  the  feasibility  and  sustainability  of  the  process  of  data 

collection, analysis and evaluation; 

  To  test  and  fine-tune  the  feasibility,  reliability  and  scalability  of 

the designed indicators; 

  To  provide  insights  for  the  development  of    an  implementation 
plan for the indicator measurement system (roll-out) across the 
EU. 

The  method  chosen  for  the  trial  was  a  web  survey  based  on  a  closed 
questionnaire collecting the basic data needed for all the elaboration of 
the "Take-up and Performance Indicators" presented in Deliverable 3 of 
this study. 

The  survey  was  addressed  to  a  group  of  e-Procurement  platforms 
previously recruited through the Advisory Panel and follow-up activities. 
A detailed trial report is attached to this report.  

The 

following 

18 

 
Table 3 shows the list of platforms participating to the trial with the level 
of  e-Submission  as  identified  above.  From  now  on  we  will  use  the 
country or region name abbreviation to identify the platform.  

The  level  of  maturity  of  the  technical  solution  for  e-Submission  proved 
relevant  for  the  availability  of  data,  so  it  is  reported  for  each  platform. 
The definition of the different levels is presented in Table 4 below.  

19 

 
Table 3 - List of Trial Platforms by level of e-Submission solution 

 N 

Short 
Name 

Name 

Type of Platform 

Level of  e-
Submission  

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

AT 

BE 

DE 

EE  

ES 

CLO 

FR 

National Public e-Procurement 
Platform (auftrag.at) 

Public National 

Federal Government e-Procurement 
Platform  (national level) 

Public National 

National Public Platform - 
Procurement Agency of the Federal 
Ministry of the Interior (eVergabe) 

Public National 

National Public Platform - Public 
Procurement Register  

Public National 

National Public Platform PLACE & 
CONECTA  

Public National 

Regional Public e-Procurement 
Platform (Cloudia) 

Public Regional 

National Public Platform - PLate-
forme des Achats de l'État (PLACE)  

Public National 

LOM  

Regional Public Platform Central 
Procurement Agency of the Region 
Lombardy 

Public Regional 

NL 

National Public Platform TenderNed  

Public National 

PL 

National Public e-Procurement 
Platform - Electronic Auctions 

Public National 

PT  

Portal BASE 

Public National 

12 

SCT  

Scotland 
Service 

eCommerce 

Shared 

Public Regional 

Source: Capgemini 2013 

Table 4 - Level of technology maturity of e-Submission solutions 

Level 

Description of e-Submission solution 

4 

1 

1 

4 

0 

4 

1 

4 

4 

2 

4 

2 

0 

1 

2 

3 

4 

The platform does not enable e-Submission 

The tender is submitted as an email attachment sent (pdf) to the platform 

The tender is submitted through data entry into the platform  

The tender is submitted through an export/import function to 
excel/access/etc. programmes. The user can input the data offline and re-
import into the platform system 

The tender is submitted through a fully interactive process, allowing 
advanced complete transactions with the e-Procurement platform. The 
platform provides guidance on the preparation and uploading of the tender 
for all its components. 

Source: Capgemini 2013 

20 

 
2.2. Summary of Results  

2.2.1. Assessment of availability  

The presentation of the results of the trial in the following paragraphs is 
based on the following variables: 

  The  level  of  availability  of  the  data  is  presented  in  a  4  steps 
scale  (see  Table  5  below)  which  specifies  the  reason  why  the 
data  is  or  is  not  available.  This  scale  will  lead  to  the 
identification  of  the  barriers  preventing  availability,  as  shown  in 
the table. Notice that the scale of availability is progressive: the 
data must first be available, then collected, then communicated 
to  the  benchmarking  organization.  One  cannot  discuss  the 
willingness to communicate the data, if it is not available in the 
first place.  

  The  level  of  maturity  of  the  e-Submission  solution  used  by  the 
platform is relevant for the assessment of availability of barriers. 
We have asked the platforms to classify themselves,  based  on 
the maturity levels identified in the Table 4. The platforms using 
only level 1 or 2 usually cannot provide data on contracts value, 
because  they  do  not  have  visibility  on  them.  The  more 
advanced  solutions  allow  such  visibility  (the  value  data  is 
entered in the system) but the platform may not collect it.  

The combination of the two criteria leads to the assessment of barriers.  

Table  5  -  Classification  of  criteria  of  Data  availability  and  correlated 
barriers 

Data Availability -  Criteria  

Barriers 

Available 

Data is available, collected, and can 
be delivered to the benchmarking 
organization 

Not available  

Data not visible to the platform / e-
Submission only level 1  

Technical barrier (to be overcome it requires 
upgrading of the system) 

Not collected  

Data potentially available but not 
collected  

Organizational barrier (to be overcome it 
requires the implementation of a process in 
the management of the platform) 

Planned 

Data will be collected in the future 

Not authorised  

Data available, collected, but 
confidential - cannot be delivered to 
the benchmarking organization  

Policy Barrier (to be overcome it requires a 
decision by the policy maker). This is 
normally a problem for the private platforms, 
while CPB can decide autonomously.  

Source: IDC 2013 

2.2.2. Availability of data for Take-up indicators 

The summary view of availability of data for take-up indicators shown in 
Table  6  below  is  calculated  based  on  the  percentages  of  our  small 
sample  of 12  platforms (notice that 8% means 1  platform). However, it 
provides a quick snapshot of the situation.  

The data needed for our system of indicators is available from no more 
than  half  the  sample  of  trial  participants  in  the  best  cases,  usually  for 
volume  indicators,  and  often  much  less  than  that.  Not  only  there  is  a 

21 

 
 
 
general problem with the basic availability of data (when the technology 
solution limits the visibility of the value data, most specifically when bids 
are  sent  as  PDF  attachments  and  the  platform  only  plays  an 
intermediary role). There is also an organizational problem when data is 
available but is not collected (particularly relevant for SMEs and cross-
border  suppliers).  Encouragingly,  a  few  platforms  are  planning  more 
advanced  monitoring  systems  (see  planned  availability).  The  level  of 
monitoring  increases  with  the  technology  maturity  of  the  solution 
implemented. 

22 

 
Table 6 - Summary View of Availability of data for take-up indicators  

Availability of basic data for Take-up Indicators 

Available 

Not 
available  

Not 
collected  

Planned 

Not 
offered 

TOT2 

Total Value of Contracts awarded by platform  

Total Value of Contracts awarded by platform with e-
Submission 

50% 

42% 

8% 

25% 

Top Take-up Indicator in Value 

Top Take-up Indicator in Volume 

Total Number of Contracts awarded by platform  

Total Number of Contracts awarded by platform with 
e-Submission 

75% 

50% 

8% 

8% 

Top Take-up Indicator in Value - SMEs 

25% 

17% 

17% 

25% 

17% 

17% 

0% 

17% 

0% 

0% 

0% 

0% 

100% 

100% 

100% 

100% 

Total Value of Contracts awarded to SMEs by 
platform 
Total Value of Contracts awarded  with e-
Submission to SMEs by platform 

8% 

8% 

8% 

75% 

8% 

0% 

100% 

25% 

50% 

17% 

0% 

100% 

Top Take-up Indicator in Volume - SMEs 

Total Number of Contracts awarded to SMEs by 
platform 

Total Number of Contracts awarded with e-
Submission to SMEs by platform 

8% 

8% 

8% 

75% 

8% 

0% 

100% 

17% 

58% 

17% 

0% 

100% 

Top Take-up Indicator in Value - Cross-Border suppliers 

Total Value of Contracts awarded to cross-border 
economic operators by platform 
Total Value of Contracts awarded  with e-
Submission to cross-border economic operators by 
platform 

50% 

17% 

33% 

0% 

0% 

100% 

33% 

33% 

25% 

8% 

0% 

100% 

Top Take-up Indicator in Volume - Cross-Border economic operators (EO) 

Total Number of Contracts awarded to cross-border 
EO by platform 

Total Number of Contracts awarded with e-
Submission to cross-border EO by platform 

42% 

17% 

42% 

0% 

0% 

100% 

33% 

25% 

33% 

8% 

0% 

100% 

Take-up Indicator by Type of Purchase (supplies, services, public works) - with e-Submission 

Total Value of contracts awarded by type of 
purchase  
Total Number of contracts awarded by type of 
purchase  

33% 

33% 

33% 

25% 

25% 

33% 

8% 

8% 

0% 

0% 

100% 

100% 

Take-up Indicator by Type of Procedure  

Total Value of contracts awarded with e-Submission 
by type of procedure 

Total Number of contracts awarded with e-
Submission by type of procedure* 

31% 

29% 

39% 

2% 

0% 

100% 

31% 

31% 

39% 

0% 

0% 

100% 

Take-up Indicator by Type of Electronic Procedure - with e-Submission - Value 

Total Value of contracts awarded with e-auction 

Total Value of contracts awarded with DPS 

Total Value of contracts awarded with e-Marketplace 

25% 

0% 

0% 

25% 

25% 

25% 

25% 

25% 

25% 

8% 

8% 

8% 

Take-up Indicator by Type of Electronic Procedure - with e-Submission - Volume 

Total Number of contracts awarded with e-auction 

50% 

Total Number of contracts awarded with DPS 

Total Number of contracts awarded with e-
Marketplace 

8% 

0% 

17% 

17% 

17% 

25% 

25% 

25% 

0% 

8% 

8% 

17% 

42% 

42% 

17% 

42% 

100% 

100% 

100% 

100% 

100% 

42% 

100% 

2 Percentages are rounded up without decimals – totals in units are always 100% 

23 

 
                                                      
2.2.3. Availability of data for Performance indicators 

The  summary  view  of  availability  of  data  for  performance  indicators 
shown  in Table  7  below  is  calculated  based  on  the  percentages  of  our 
small sample of 12 platforms (notice that 8% means 1 platform).  

The  level  of  availability  is  very  different  by  type  of  indicator.  For  the  e-
Submission  Price  Reduction  indicator  and  the  Reduction  of  Litigation 
indicator  there  is  a  very  low  level  of  availability,  with  the  main  reason 
being  that  the  data  is  not  there.  This  shows  that  there  are  relevant 
barriers for the measurement of these indicators.  

the  other  extreme,  both 

At 
Transparency indicator reach 100% availability easily.  

the  Ease  of  Access 

indicator  and 

The e-Tendering Efficiency Improvement indicator seems to be midway 
between  these two groups of indicators,  with half the sample declaring 
the  availability  of  the  data  needed.    However,  the  quality  of  the  data 
actually collected (as will be discussed in the next chapter) is debatable. 
This also may require a reflection on the indicator itself.  

Table 7 - Summary View of Availability of data for Performance Indicators 

Availability of basic data for Performance 
Indicators 

Available 

Not 
available  

Not 
collected  

Planned 

TOT3 

E-Submission Price Reduction Indicator 

Data on reserve price and average award 
price for all the contracts processed by the 
platform without e-Submission 

Data on reserve price and average award 
price for all the contracts processed by the 
platform with e-Submission 

8% 

42% 

50% 

0% 

100% 

25% 

25% 

50% 

0% 

100% 

E-Submission Reduction of Litigation Indicator 

Average number of appeals per contract 
processed with e-Submission  

Average number of appeals per contract 
processed  without e-Submission 

8% 

67% 

25% 

0% 

100% 

8% 

58% 

33% 

0% 

100% 

e-Tendering efficiency improvement indicator for Contracting Authorities 

Average time needed by a CA to prepare and 
implement an e-Tendering process including 
e-Submission, according to the requirements 
posed by the platform 

42% 

33% 

25% 

0% 

100% 

E-Submission Ease of access indicator 

 Type of e-authentication required to access 
the platform and its main services (e-
Notification, e-Submission) per type of supplier 
(domestic vs. cross-border) 

100% 

0% 

0% 

0% 

100% 

E-Submission Transparency indicator 

Type of information provided to economic 
operators and buyers, based on a check list 
provided by the benchmarking organization 

100% 

0% 

0% 

0% 

100% 

3 Percentages are rounded up without decimals – totals in units are always 100% 

24 

 
 
                                                      
2.3. Calculation of the Indicators  

In our original plan, we had hoped to be able to calculate the indicators 
thanks  to  the  results  of  the  trial.  Unfortunately,  the  limited  level  of 
availability of data means that we have too few data points to be able to 
calculate  all  the  indicators  with  an  acceptable  level  of  reliability.  Based 
on the trial results and our research, we are able to calculate only three 
indicators, as follows: 

1.  Top level indicator of take-up in value: Only 5 platforms out of 12 
provided this data. However, we were able to combine this data with 
other  data  from  our  research  and  public  sources,  in  order  to 
calculate  an  estimate  of  the  value  of  e-Procurement  in  2012  in 
Europe. While for the years 2010-2011 we were able to estimate the 
take-up indicator for each of the EU27 MS (see D2 “State of Play”), 
for the year 2012 this is not possible.  

2.  E-Submission  Ease  of  Access  and  Transparency  Performance 
Indicators:  the  data  required  is  100%  available  so  we  are  able  to 
calculate them based on the trial results.  Clearly, the results of the 
performance indicators cannot be considered valid for the entire EU; 
they  are  only  representative,  at  this  stage,  of  the  trial  group  of 
platforms.    However,  this  measurement  provides  a  good  "proof  of 
concept" of the potential quality and value added of these indicators, 
as shown below.  

2.4. Take-up of e-Procurement in Value - EU 2011-

2012 

The  take-up  value  indicator  is  calculated  for  each  MS  and  then 
aggregated to calculate the EU indicator. The indicator is calculated as 
follows: 

  The  numerator  is  the  absolute  value  of  e-Procurement  by 
MS,  that  is  the  total  value  of  public  contracts  awarded  in  the 
reference period and processed with e-Procurement, up to and 
including  at  least  e-Submission,  but  also  e-Auction  and  e-
Ordering  (if  they  were  used).  This  value  was  estimated  for  the 
years  2010-2011  based  on  public  sources,  interviews  and  a 
dedicated value survey carried out by IDC4. The estimates for 
2012  were  calculated  by  projecting  the  values  of  2011  for 
each MS, leveraging our knowledge of main trends and the 
data collected by the trial.  

  The EU estimate of e-Procurement for 2012 corresponds to 
the aggregated value of  e-Procurement in 24 MS, excluding 
the Netherlands, Latvia and Slovakia for which we did not have 
sufficient  data.  In  4  MS,  e-Submission  was  not  available  yet  in 
this  period  and  the  corresponding  value  is  zero  (Bulgaria, 
Slovenia,  Greece  and  Hungary).    Compared  to  2011,  this 
estimate  adds  Luxemburg  where  the  national  public  platform 
started  offering  e-Submission  in  2012. This  is  the  “minimum  or 
in 
low 

presented 

estimate” 

range 

4  The  estimate  process  is  explained more  in  detail  in  Deliverable  D2  "State  of  Play  of  e-
Procurement in Europe" 

25 

 
                                                      
Figure 3 and Table 8 below.  

  The denominator of the indicator is the Eurostat estimate of 
the value of general government procurement for the 24 MS 
indicated  above,  updated  to  20125.  This  value  excludes  the 
Netherlands,  Latvia  and  Slovakia,  since  we  were  unable  to 
estimate 
the  public 
procurement by the countries where e-Procurement is currently 
nil.  This  is  the  denominator  of  the  “low  range  estimate” 
in 
presented 

take-up.  This  value 

includes 

their 

.5  The  General  Government  Procurement    data  by  Eurostat  includes  central,  local,  and 
state governments spending, and social security funds. We have subtracted the value of 
procurement by the State Owned Utilities. This data is sourced from the European System 
of National and Regional Accounts (ESA 95). 

26 

 
                                                      
Figure 3 and Table 8 below.  

  We are aware that this aggregate by Eurostat is larger and less 
accurate  than  the  definition  of  government  spending  used  by 
most countries  when calculating their own  e-procurement take-
up.  But  this  is  the  only  consistent  and  comparable  source  of 
general procurement value across the EU27, so we were forced 
to use it to compare the level of take-up by country.  

  The calculation of the “maximum” or “high range estimate” 
(Table  9)  was  performed  with  a  "sensitivity  analysis" 
approach,  by  pushing  to  the  upper  range  all  the  estimates, 
which had a margin of uncertainty, building on our estimates for 
2010-2011.  This  was  based  on  calculating  higher  estimates  of 
e-Procurement  value  in  2012  for  Belgium,  Denmark,  France, 
Finland, Germany, Italy, the UK, and adding an estimate for the 
Netherlands  (where  the  public  platform  started  offering  e-
Submission in 2012). Therefore, the high range estimate for the 
EU concerns 25 MS, excluding only Latvia and Slovakia.  

The results of this estimates are presented in Figure 3. The overall value 
of  e-Procurement  in  24  MS  in  the  EU  in  2012  is  estimated  between 
€185 Bn (low range  estimate) and  €220 Bn (high range  estimate). The 
corresponding  level  of  take-up  on  general  government  procurement  is 
(
between 

12.5% 

11.5% 

and 

27 

 
Figure  3).  In  2011,  the  overall  value  of  e-Procurement  was  between 
€170  Bn  and  €203  Bn,  with  take-up  in  a  range  between  10.6%  and 
11.7%. These estimates are, in our opinion, quite reliable for most  MS. 
The  gap  between  the  low  range  and  high  range  estimates  is  limited  to 
1% take-up, which reflects the solidity of the estimate.  

From  the  point  of  view  of  the  calculation  method,  this  experience 
indicates  that  the  data  collection  from  a  sample  of  platforms  is 
insufficient to lead to a sound and reliable estimate at the EU level, even 
if the panel of platforms is extended.  

28 

 
Figure 3 - Estimate of take-up of e-Procurement in the EU, 2011-2012: high-
low range (€ Billion)  

Source: IDC 2013 

Table 8 – Low Range Estimate of EU level Take-up (€ Billion) 

TOTAL EU – 24 MS (No 
NL, SK, LV) 

2010 

2011 

Growth 
2011/2010 

2012 

Growth 
2012/2011 

General gov procurement   

1.606.622 

1.602.172 

-0.3% 

1.615.784 

+0.8% 

E-Procurement Value 

149.957 

170.069 

13.4% 

185,091 

8.8% 

Take-up  

9.3% 

10.6% 

13.7% 

11.5% 

7.9% 

Source: IDC March 2013  

The  growth  rate  of  e-Procurement  between  2011  and  2012  in  the  low 
range  estimate  (Table  8)  is  approximately  8.8%  in  value,  while  take-up 
increased  by  7.9%  only.  The  economic  crisis  does  not  seem  to  have 
accelerated  the  adoption  of  e-Procurement  as  much  as  it  could  have 
been  expected,  given  its  potential  benefits  of  efficiency  and  control  of 
public spending. 

Notice  also  that  since  e-Procurement  is  still  marginal  (around  or  under 
10%  of  public  spend  in  most  countries),  the  tendering  of  a  few  large 
public contracts and other contingency factors are sufficient to push up 
or down the total level of take-up from one year to another.  

Table 9 - High Range estimate of EU level Take-up 

TOTAL EU – 25 MS (No 
LV, SK) 

2010 

2011 

Growth 
2011/2010 

2012 

Growth 
2012/2011 

General gov procurement         1,742,226 

1,738,680 

-0.2% 

1.754.947 

0.9% 

E_Procurement Value  

173,950 

202,719 

16.5% 

219,746 

8.4% 

Take-up  

10.0% 

11.7% 

16.8% 

12.5% 

7.4% 

Source: IDC 2013 – Higher estimates compared to low range estimate for 
BE, DE, DK, FI, FR, IT, NL, UK 

29 

05010015020025020112012170185203220HighLowTake-up  2011: 11.7%Take-up  2011: 10.6%Take-up  2012: 12.5%Take-up  2012: 11.5% 
 
 
2.5. EU level Performance Indicators 

2.5.1. E-Submission Ease of Access Indicator 

The simplification  and ease of use  benefits of e-Procurement is one  of 
the main drivers of the transition from traditional to electronic processes. 
We have designed one indicator focused on the usability of the methods 
of  authentication  required  by  the  platforms,  which  is  one  of  the  most 
critical aspects of e-Procurement usability. The indicator is as follows: 

  The  e-Submission  Ease  of  Access  indicator  measures  the 
level of ease of access to e-Submission services measured 
through the type of electronic authentication required. This 
indicator  cannot  be  measured  with  an  absolute  value,  but  is 
based  on  a  semantic  scale  corresponding  to  a  rising  scale  of 
authentication requirements.  

  Measurement  scale:  in  the  suggested  semantic  scale  each 
value  corresponds  to  a  score  and  is  indicatively  defined  as 
follows: 

o  Very  High  =  VH  =  score  5  =  No  authentication 

requirements 

o  High  =  H  =  score  4  =  Request  of  simplified  eID 

(username and password) very easy to have 

o  Medium  =  M  =  score  3  =  Request  of  simplified  eID 
(username  and  password)  based  on  a  request  process 
of low difficulty 

o  Low = L = score 2 = Request of qualified or advanced e-
Signatures based on a request process of low difficulty  

o  Very  low  =  VL  =  score  1  =  Request  of  qualified  or 
advanced  signatures  with  complex  process  of 
acquirement,  or  of  more  complex,  country-based 
qualification methods 

The trial explored the type of authentication required by platforms on the 
basis of these definitions. Fortunately, in this case there is no availability 
problem.  

It is however necessary to clarify in the data collection to which function 
we  refer  (registration  to  the  platform;  e-Notification;  e-Submission) 
because  many  platforms  require  more  secure  authentication  methods 
for e-Tendering. This is the case for example of Cloudia and PLACE, the 
Finnish  and  French  platforms  of  our  sample,  as  shown  in  the  Table  10 
below.  

In addition, to complete the assessment it is necessary to verify with the 
platform  whether  the  process  of  acquiring  the  digital  signature  is 
complex  or  easy,  whether  there  is  a  national  system  which  a  cross-
border  supplier  must  use  or  not.  By  combining  the  trial  results  with 
information  from  our  previous  research6  we  were  able  to  complete  the 
following table. But in the future, this indicator will require more than one 
question to be properly measured.  

Table 10 – Ranking of Trial Platforms by Level of Ease of Access Indicator 

6  Sourced  from  the  database  Census  of  e-Procurement  Platforms,  presented  in  D1  –  e-
Procurement Landscape in Europe 

30 

 
                                                      
Platform 

Score 

Level of Ease of Access 

Authentication Required 

SCT 

NL 

AT 

EE 

CLO 

FR 

ES 

DE 

BE 

LOM 

PL 

5 

4 

3 

3 

3 

3 

3 

3 

2 

2 

2 

Very High 

Not needed 

High 

Simplified e-ID 

Medium 

e-ID with request process 

Medium 

National e-ID 

Medium 

Medium 

Simplified e-ID for registration 

Qualified digital signature for e-Submission 

Simplified e-ID for registration 

Advanced digital signature for e-Submission 

Medium 

Other  

Medium 

Digital Signature 

Low 

Low 

Low 

Qualified digital signature 

Qualified digital signature 

Qualified digital signature 

Source: IDC 2013 

The  same  results  are  shown  in  the  Figure  4  below.  This  seems  to  say 
that only a minority of platforms have high or very high ease of access, 
most of them are at a medium level, and another group have low ease 
of  access  -  which  might  prompt  a  recommendation  to  try  to  improve 
access conditions.   

Figure 4 - Ease of Access Indicator for the Trial Platforms 

 Source: IDC 2013 

31 

0123456SCT NLATEE CLOFRESDEBELOMPLEase of Access IndicatorVery HighHighLowMedium 
2.5.2. E-Submission Transparency Indicator 

Transparency and completeness of information have been identified as 
critical  success  factors  for  the  adoption  of  e-Procurement. To  measure 
this, we have designed a transparency indicator based on the following 
definition:  

the 

indicator  measures 
information  provided 

  The  Transparency 
level  of 
transparency  of 
the 
to  buyers  and 
the  main  e-Procurement  entities 
economic  operators  by 
analyzed  about  the  use  of  their  main  services.  This  indicator 
cannot  be  measured  with  an  absolute  value,  but  has  to  be 
level  of  quality  and 
based  on  an  assessment  of 
completeness  of  the  information  provided  at  the  platform  level, 
which will then be scaled up to the country and EU level. 

the 

The study team developed for the trial a list of typologies of information 
relevant  for  this  indicator.  The  main  categories  considered  are  the 
following:  

  Buyers information, such as FAQ for Contracting Authorities and 

buyer guidelines; 

  Suppliers information, similarly, FAQ for suppliers and suppliers 

guidelines; 

 

Information on platform functionalities, such as user guides and 
technical requirements; 

  Communication  functionalities,  such  as  message  boards  and 

complaint forms/ processes; 

  Language  availability:  we  have  privileged  the  availability  of  a 

second language, usually English 

  Disclosure  of  bidders  and  winners  identity:  in  this  case  full 

disclosure of all information is privileged 

  Helpdesk support, for example whether it is offered only in office 

hours or 24/7.  

For  each  of  these  categories  of  information  we  have  identified  4/5 
options,  if  the  platform  offers  them  all  this  is  considered  100% 
availability. Different mixes of options are scored differently, for example 
in the case of the helpdesk a platform providing support 24/7 rather than 
only  in  office  hours  receives  a  higher  score.  The  ranking  of  results  by 
in 
type 

information 

shown 

of 

is 

32 

 
Table 11 below.  

33 

 
Table 11 - Data collection for Transparency Indicator: level of availability of 
information by type (100% = Full availability) 

Type of Information and service   

Average Availability of 
Information  

Buyers Information 

Suppliers Information 

Information on functionalities 

Communication functionalities 

Language availability 

Disclosure of bidders and winners identity 

Helpdesk support 

Source: IDC 2013 

Looking 

72% 

72% 

57% 

53% 

47% 

46% 

27% 

at 

34 

 
Table  11,  we  notice  that  the  average  availability  of  information  by 
category  (calculated  by  averaging  the  transparency  scores  of  all  12 
platforms) is higher for buyers and suppliers information and lowest for 
helpdesk  support  and  disclosure  (with  some  platforms  providing  the 
information only to the parties actively involved, having placed a bid for 
example). Overall, the level of availability of information is not as high as 
we would have expected.  

The following step was to identify the type of information corresponding 
to  high  or  low  transparency.  For  each  category  of  information,  we 
assigned  a  score  from  1  (very  low  transparency)  to  5  (very  high 
transparency  –  see  Table  12)  to  the  information  provided  by  the 
platforms. The  detailed  definition  of  the  combination  of  answers,  which 
corresponds to the score, is provided in annex. To provide an example, 
in the case of the disclosure of information on bidders and winners, we 
defined the maximum and minimum levels of the measurement scale as 
follows: 

  Very  High  transparency  (score  5):  Full  disclosure  of  all  bidders 
identities after awarding + full disclosure of the winner's identity 
and bid value; 

 

Insufficient  transparency  (score  1):  Disclosure  of  only  winning 
bidder's identity to bidders and buyer 

By full disclosure we mean that the information is made available to all 
parties  (for  example  published  on  the  web);  by  simple  disclosure  we 
mean that the information is sent only  to the participants to the tender. 
Of course the second approach implies lower transparency than the first 
approach.  The  intermediate  scores  are  assigned  on  the  basis  of  a 
similar  logic,  based  on  the  combination  of  disclosure  behaviour  of  the 
platforms.  

These  scores  were  assigned  by  the  study  team  on  the  basis  of  our 
expert  assessment.  Naturally,  the  scoring  scale  should  be  discussed 
and validated again with stakeholders before the final indicator's value is 
universally accepted. This calculation resulted in a list of scores for each 
platform, corresponding to the 7 typologies of information measured.  

Finally,  the  Transparency  Indicator  per  platform  was  calculated  as  the 
average  of  all  the  7  scores.  All  the  information  categories  were 
considered equally important. It is also possible to add weights, in order 
to increase the weight of certain typologies of information. For example, 
it  could  be  argued  that,  to  measure  transparency,  a  high  score  in  the 
disclosure  of  information  about  winners  is  more  important  than  a  high 
score  in  communication  functionalities.  Any  decision  about  weights  in 
these indicators should be discussed with stakeholders and/or based on 
objective elements, such as a survey on user preferences. At this stage, 
lacking these inputs, it is better to avoid the use of weights.  

Table 12 - Transparency Indicator - Measurement Scale 

Score 

Scale 

Description 

Very High 

Very high transparency 

High 

High transparency 

5 

4 

35 

 
3 

2 

1 

Medium 

Medium Transparency 

Low 

Low transparency 

Very Low 

Insufficient transparency 

Source: IDC 2013 

The results about the indicator scores by platform are presented in Table 
13 and Figure 5 below. As shown in the figure, the platforms in the trial 
have  summary  scores  falling  between  medium  and  high  transparency. 
However,  some  platforms  achieve  very  high  transparency  for  some  of 
the  information  categories,  as  shown  in  Table  13.  The  detailed  score 
helps to identify weak and strong points of each platform and what could 
be improved.  

Table  13  Trial  results:  Transparency  Indicator  scores  by  platform  and  by 
category 

BE 

LOM 

AT 

CLO 

FR 

SCT   EE   NL 

ES 

DE 

PL 

Buyers Information 

Suppliers Information 

Information on 
functionalities 

Communication 
functionalities 

Language availability 

Disclosure of bidders 
and winners identity 

Helpdesk support 

5 

5 

3 

5 

4 

3 

2 

5 

5 

3 

5 

2 

5 

2 

4 

4 

3 

5 

2 

5 

3 

4 

4 

2 

5 

4 

5 

2 

5 

5 

5 

2 

4 

3 

2 

5 

5 

4 

4 

2 

2 

3 

4 

4 

2 

3 

2 

4 

3 

4 

4 

3 

4 

2 

2 

3 

4 

4 

2 

2 

2 

3 

4 

4 

4 

3 

2 

2 

1 

3 

4 

4 

2 

2 

2 

3 

2 

Average Score 

3.9 

3.9 

3.7 

3.7 

3.7 

3.6 

3.1 

3.1 

3.0 

2.7 

2.7 

Figure 5 – Trial results: Transparency Indicator average score by Platform 

Source: IDC 2013 

36 

0.00.51.01.52.02.53.03.54.04.55.0BELOMATCLOFRSCT EE NLESDEPLTransparency IndicatorHighMediumLow 
 
  
2.6. Lessons Learned from the trial 

This  paragraph  presents  the  lessons  learned  from  the  trial  and  our 
suggestions to revise and improve the process of data collection, in view 
of a new round of implementation.  

2.6.1. Level of participation and motivation 

The trial sample was never meant to be statistically representative, but 
according  to  our  original  plan,  it  should  have  reflected  the  main 
characteristics of the EU e-Procurement landscape, in terms of typology 
of platform, technology sophistication, e-Procurement models (based on 
the regulatory framework) and geographical balance.  

The actual group of participants, though small  in size (smaller  than  we 
would  have  wished),  responds  to  these  requirements  (as  shown  in  the 
Table 14 below) but for two main aspects: 

  The  typology  of  platforms:  no  private  platform  agreed  to 
participate,  because  they  all  declared  that  they  were  not 
authorized by their Contracting Authority clients to communicate 
their data; 

  The  level  of  sophistication:  the  sample  includes  half  highly 
sophisticated platform, and half platforms with basic technology 
solutions  (based  on  our  classification  of  the  levels  of  e-
Submission). This was done to insure representativeness at the 
geographic level, but reflects also the actual level of maturity of 
technology infrastructures in the EU, which is still not very high.  

The following main considerations follow  in  view of the development of 
the measurement system: 

  There  is  a  need  to  find  more  effective  motivations  to  stimulate 

participation by platforms; 

  The problem of authorization of private platforms to share data 

must be solved ; 

  Since the platforms have different technology maturity, the data 
collection approach must adapt to it (platforms with low maturity 
have no visibility on value data, for example).  

Table 14 - Correspondence to plan of the trial participants 

Plan 

Actual Sample 

Appropriate balance of the main typologies of 
platforms identified by the Census (public platforms 
national/ regional, private platforms)  

All public platforms (of which 2 regional) 
but no private platforms 

Sophisticated platforms with high level of coverage of 
interactive services 

6 platforms at level 4 e-Submission (high 
sophistication);  6 platforms at medium-low 
sophistication (level 0 to 2) 

Appropriate balance of Member States with 
Mandatory/non Mandatory, Centralized/Decentralized 
e-Procurement models, based on the development 
models identified in the landscape mapping 

Good mix: 6 mandatory, 6 voluntary  

Appropriate representativeness of the EU27 in terms 
of population (Large-medium-small Member States, 
following the segmentation anticipated in workflow 1) 

Good mix of large-medium-small countries, 
North-South, East-West. The trial includes 
platforms from the 6 main EU MS 

37 

 
and geography (old-new MS, north-south, east-west).  

Source: IDC 2013 

38 

 
Sophistication - 

Level of e-
Submission 

Regulation of e-
Submission 

4 

1 

1 

4 

0 

4 

1 

4 

4 

2 

2 

2 

Voluntary 

Mandatory in some 
cases 

Mandatory in some 
cases 

Mandatory  take-up 
targets / Planned 

Voluntary 

Voluntary 

Mandatory in some 
cases 

Mandatory in some 
cases 

Voluntary 

Voluntary 

Mandatory for all 

Voluntary 

Table 15 Characteristics of the Trial Participant 

MS 

Platform Name 

Type of 
Platform 

Public 
National 

Public 
National 

Public 
National 

National Public e-

Procurement Platform 
(auftrag.at) 

Federal Government e-
Procurement Platform  
(national level) 

National Public Platform - 

Procurement Agency of the 
Federal Ministry of the 
Interior (eVergabe) 

National Public Platform - 
Public Procurement Register  

Public 
National 

National Public Platform 
PLACE & CONECTA  

Public 
National 

Regional Public e-

Procurement Platform 
(Cloudia) 

National Public Platform - 

PLate-forme des Achats de 
l'État (PLACE)  

Public 
Regional 

Public 
National 

Regional Public Platform 
Central Procurement Agency 
of the Region Lombardy 

Public 
Regional 

National Public Platform 
TenderNed  

Public 
National 

National Public e-

Procurement Platform - 
Electronic Auctions 

National Public Platform 
Portal BASE 

Public 
National 

Public 
National 

Public 
Regional 

AT 

BE 

DE 

EE 

ES 

FI 

FR 

IT 

NL 

PL 

PT 

SCT 

Scotland eCommerce 
Shared Service  

Source: IDC 2013 

39 

 
2.6.2. Data Collection Process 

The  following  paragraphs  discuss  in  detail  the  process  for  data 
collection developed and tested in the trial and the relative findings, as 
an input for a new round of implementation. This process is composed 
of  four  phases  and  eleven  steps  (Table  16).  Not  all  the  activities 
described in the steps were relevant for the trial.  

Table 16 – Data collection process: main steps 

PHASE I Preparation of Data Collection 

Step 1:   Measurement start-up 

Step 2:   Preparation of the data collection  

PHASE II Data Collection 

Step 3:   Recruitment of the platform by e-mail invitation  

Step 4:   Delivery of the data 

Step 5:   Maximization of response rates by e-mail reminder 

Step 6:   Data collection from other sources  

PHASE III Data Analysis 

Step 7:   Control of the collected data  

Step 8:   Data-export, -cleaning and -filing 

Phase IV Reporting on Results   

Step 9:   Development and elaboration of the indicators  

Step 10:   Reporting and presenting the results  

Step 11: Evaluation of the Measurement Process  

Phase 1 - Preparation of the data collection 

Step 1 - Measurement start up 

In  the  first  phase,  the  study  team  and  DG  MARKT  agreed  on  the 
general  approach  to  the  proof-of-concept  trial  and  the  profile  of 
respondents  to  be  contacted.  The  study  team  prepared  and  presented 
to  DG  MARKT  the  data  collection  template  and  the  targeted  sample 
respondents.  

Step 2 - Preparation of the data collection 

The data collection tools, including the web survey tool, are prepared for 
collecting  data. The study team prepares a dedicated  web page  where 
the data for the benchmark can be compiled by the platforms. The study 
team tests the data collection tools.  

The benchmarking organization informs the platforms about the start of 
the  benchmark,  the  procedure  for  supplying  the  necessary  data  by  the 
platforms, the definitions of the data, the timeline of the start, execution 
and  wrap  up  of  the  benchmark.  Furthermore,  a  service  desk  is 
organized  to  support  the  platforms  in  collecting  the  data  and  importing 

40 

 
them  from  their  information  systems  to  the  data  collection  templates.  
Finally, each platform is asked to appoint a dedicated contact person for 
the measurement process. This contact person is basically the only one 
who interacts with the service desk.  

Lessons Learned - Phase 1 Preparation of the data collection 

1  Communication  and  instruction  are  essential  for  a  smooth  measurement.  We 
developed  guidelines  for  the  process  of  data  collection,  analysis  and  evaluation  and  a 
dictionary  with  definitions  of  the  terms  used  in  the  measurement.  These  supporting  tools 
have  proven  to  be  essential.  It  turns  out  to  be  necessary  to  invest  enough  time  in  future 
measurement rounds to explain the process of data collection and especially the definitions 
of  the  terms  to  the  participants  in  the  measurement.  Possibilities  for  bilateral  explanations 
should be strengthened.      

The  study  team  implemented  a  Yammer  workspace  for  communication  in  the  group  of 
participants in the trial and the benchmarking organization. It turned out that this workspace 
was mainly used to make documents available for the participants and for announcements 
by  the  benchmarking  organization.  There  was  not  mutual  communication  between 
participants  on  Yammer.  For  the  short  term  Yammer  remains  a  suitable  solution  for 
communication  between  the  benchmarking  organization  and  the  participants  in  the 
measurement.   

for  data  storage,  data  aggregation  and  analysis.  The 

2 The trial was executed through a web enabled survey, using Surveymonkey combined 
with  Excel 
functionality  of 
Surveymonkey  is  limited.  This  tool  is  suitable  only  for  a  limited  group  of  participants.  For 
future  use  a  more  sophisticated  survey  tool  would  be  necessary,  with  more  advanced 
possibilities  for  checks  on  entered  data  and  design  of  composed  and  multiple  choice 
questions.  This  will  decrease  the  time  necessary  for  checks  on  the  collected  data  and 
correction  of  inconsistencies.  We  advise  to  use  in  the  future  Access  or  even  business 
analytics software like SAS to handle the collected data.  

Phase 2 - Data collection 

Step 3 - Recruitment of the platform by e-mail invitation  

In  this  phase,  the  study  team  sends  a  personalized  e-mail  to  each 
potential participant, with a short explanation, an invitation to participate 
and the location of the website and templates to import the data related 
to the platform for the benchmark.   

The email is addressed to the contact persons of each participant. This 
e-mail  contains  a  personalized  link  to  the  website  where  to  fill  in  the 
requested data. 

Step 4 - Delivery of the data  

The study team sends an email to the contact persons of the platforms 
when  the  data  collection  website  is  open  for  the  platforms  to  fill  in  the 
requested  data.  The  contact  persons  of  the  platforms  now  have  an 
agreed  upon  period  to  fill  in  the  requested  data  by  answering  the 
questions  on  the  data  collection  website. The  service  desk  is  available 
to  help  in  case  of  questions  or  problems  that  cannot  be  answered  or 
solved by the documentation on the website.   

41 

 
Step 5 - Maximization of response rates  

The study team sends e-mail reminders to maximize the response rates 
every two  weeks. In this e-mail the respondents,  who  have  not filled in 
the  templates  on  the  website,  are  reminded  to  do  so.  The  study  team 
monitors the response rate permanently and  in real-time, supported by 
the data collection tools.  

Lessons Learned - Phase 2 Data collection 

1 Overall, the process of data collection works well.  Since the total number of participants 
was  small,  the  study  team  launched  a  second  and  then  a  third  round  of  invitations  to  other 
platforms  selected  for  their  relevance.  We  noticed  that  the  approach  of  the  participants  as  a 
community  in  combination  with  bilateral  communication  by  the  study  team  delivers  the  best 
results. We  will  suggest  using  this  approach  in  the  measurements  in  the  short  term,  the  next 
years  when  the  amount  of  surveyed  organizations  is  still  growing.    There  is  a  need  for  more 
bilateral  briefings, to clarify  definitions  and make sure that the  same  type  of  data  is collected. 
Quality control of the data is essential and this must be well understood by the participants.     

2  The  amount  of  iterations  and  recalls  by  the  study  team  had  little  influence  on  the 
willingness  of  platforms  to  cooperate.  Platforms  have  to  see  an  advantage  for  themselves 
(what is in it for me?) before they commit to the (extra daily) work the Trial requires. Motivation 
is the key to participation. A way to motivate them could be to prepare personalised reports for 
the  participants,  delivering  their  data  benchmarked  with  their  peers  in  their  country,  or  in 
Europe, focusing on different variables of their interest.  

3  Based  on  the  trial  experience,  public  organisations  generally  were  more  open  to 
supporting  the  e-Procurement  measurement  trial  and  submitted  data  more  willingly. 
Private platforms responded positively at first, but then did not participate to the trial. This was 
justified  by  their  inability  to  share  confidential  data,  owned  by  their  clients,  the  Contracting 
Authorities. In addition, there was probably some concern about the amount of work required to 
prepare the data requested, without clear benefits resulting from this effort.  

Step 6 Data collection from other sources  

The calculation of benchmarks requires data that cannot be delivered by 
traditional 
the  platforms 
procurement. These data need to be collected from other sources, such 
as Eurostat, national Procurement authorities, Contracting Authorities.   

for  example  concerning 

themselves, 

Phase 3 - Data analysis 

Step 7 - Quality Control of the data  

A benchmark characterized by validity, reliability and representativeness 
needs  a  correct,  consistent  and  accurate  dataset.  The  platforms 
themselves are responsible in the first place for the quality of the data. 
The data imported by the platforms are controlled on three moments: 

1.  When the contact person collects the data requesting and builds the 

dataset in the back office of the benchmarking organization; 

2.  When  the  data  are  imported  manually  by  the  platform  contact 

person in the templates on the benchmark website; 

3.  When the study team carries out a final check of the data uploaded 

in the web survey interface. 

Some checks are built in on the website interface, where the requested 
platform data are entered. Data can only be imported in a field if it meets 
the  definition  of  that  field,  such  as  the  types  of  records  or  the 
identification  of  empty  cells.  It  is  not  possible  to  submit  any  data  if  it 
does  not  meet  these  definitions.  Only  a  completed  data  collection  can 

42 

 
be  submitted  to  the  central  server  of  the  study  team.  In  addition,  there 
are automated quality checks to detect discrepancies, which are pointed 
out  to  the  participant  in  charge,  who  can  correct/update  the  data.  All 
data input from the platforms is automatically registered in a centralized 
database.  The  study  team  can  ask  platforms  to  provide  evidence  that 
delivered  data  are  correct  in  case  there  are  signals  of  possible 
inconsistencies or mistakes (wrong decimals in a number, for example).  

Step 8 - Exporting, cleaning and filing the data 

This  step  includes  several  elements  of  data  reporting,  enabled  by  the 
online software tool. Once the data collection is completed, the phase of 
data  treatment  starts.  It  consists  of  the  importation  of  data,  the 
implementation of comparisons and the exportation of consolidated data 
to facilitate the validation process for the platform. The raw datasets are 
‘cleaned’  (i.e.  undergo  final  quality  checks)  before  data  are  delivered 
and  analyzed.  Final  quality  checks  include  comparison  with  previous 
data from other sources such as Eurostat or TED.  

Lessons Learned - Phase 3 Data analysis 

1 The quality control phase also highlighted the limited potential of the tool, which did 
not support validation and crosschecks.  

2  In  a  normal  survey,  once  the  respondents  fill  in  the  questionnaire  their  work  is 
done.  In  the  case  of  a  benchmarking  measurement  such  as  ours,  a  final  phase  of 
interaction  with  the  trial  participants  is  needed  to  provide  clarification,  crosschecks  and 
validation of the final data set. This takes time and requires a strong motivation by the  trial 
participants, particularly the contact person.  

Phase 4 - Reporting 

Step 9 - Calculating the indicators 

The collected data per platform define the basic indicators per platform 
and are used to calculate these basic indicators. These basic indicators 
in turn will be used to calculate the take-up and performance indicators 
on member state and EU level. The study team elaborates the collected 
data by: 

  Aggregating the basic level indicators of the sample and scaling up 

to the level of member state and EU; 

  Developing estimates to fill the gaps between the collected data and 
the  market,  by  calculating  the  level  of  representativeness  of  the 
sample; 

  Calculating the take-up indicators and the performance indicators by 
combining  the  collected  data  from  the  platforms  and  the  traditional 
public procurement data. 

Step  10  -  Reporting  the  results  and  presenting  a  Benchmark 
Dashboard 

After  calculating  the  indicators,  the  study  team  prepares  a  benchmark 
dashboard  to  present  clearly  and  coherently  the  main  indicators.  The 
final  presentation  of  the  benchmark  dashboard  can  be  tailored  to  the 
requirements and preferences of the EC. 

Step 11 - Evaluation process 

43 

 
After  each  round  of  measurement,  the  benchmarking  process  is 
evaluated. The evaluation should take place on the quality of the results 
of  the  benchmark  and  the  quality  of  the  process.  This  may  lead  to 
recommendations  for  improving  the  benchmark  or  adjusting  the  key 
performance indicators for the next period. 

Lessons Learned trial Phase 4 Benchmark Dashboard 

1  The  problems  with  the  lack  of  availability  of  the  data  prevented  the  calculation  of 
the  indicators,  with  the  exception  of  those  presented  in  paragraph  2.3.  A  detailed 
assessment of the availability of data and reliability and scalability of indicators is presented 
in the following paragraph. For the same reason, it was not possible to deliver a Benchmark 
Dashboard as assumed in D3 Performance Indicators.   

2 The evaluation of the process is presented in this chapter.  

44 

 
 
3 .  A S S E S S M E N T   O F   T H E   I N D I C A T O R  

S Y S T E M  

3.1. Approach to feasibility and quality assessment 

This  chapter  presents  the  feasibility  and  quality  assessment  of  the 
system of indicators. This assessment is a comprehensive evaluation of 
the  feasibility  and  quality  of  the  indicators  in  the  perspective  of  a 
complete  measurement  system,  which  was  tested  in  the  proof-of-
concept trial. The results of this assessment will lead to the modification 
and fine-tuning of the System of Indicators, presented in paragraph 3.6. 

The main goal  is to  verify  whether the  System of  Indicators satisfies in 
practice  the  high  quality  criteria  targeted  by  the  study  team,  which 
included  feasibility,  reliability  and  clarity,  comparability,  flexibility  and 
representativeness  of  the  balance  of  experiences  across  the  EU  e-
Procurement landscape.  

Feasibility  is  not  a  yes-no  condition,  but  the  outcome  of  a  series  of 
conditions,  including  availability  of  data,  reliability,  value  added  and 
scalability  of  the  indicators,  which  must  all  be  satisfied  at  least  at  a 
minimum level and with a reasonable amount of time and  effort. These 
conditions were tested in the proof-of-concept trial.  

The  quality  and  feasibility  assessment  is  based  on  the  results  of  the 
trial, but it takes also into account what we learned from the Census of 
e-Procurement platforms implemented in the first stage of the study, on 
the  data  collection  implemented  for  the  D2  (e-Procurement  State  of 
Play) and on the study team knowledge of the e-Procurement market in 
general.  

The  main  criteria  of  the  assessment  presented  in  the  following 
paragraphs are the following:  

Availability: this refers to the level of availability of the data needed for 
the specific indicator, in the format required by the indicator. Availability 
was  analysed  in  detail  in  the  previous  chapter,  so  here  we  report  a 
synthetic  assessment  building  on  that  analysis.  Obviously,  without 
availability no indicator can be measured, so this is a necessary but not 
sufficient condition of feasibility.   

Reliability:  it  refers  to  the  quality  of  the  indicator  and  verifies  whether 
the  definition  of  the  indicator  is  clear,  coherent,  consistent,  shared  and 
understandable  by  the  organizations  collecting  the  data;  it  means  also 
that  the  indicator  is  measured  through  a  clear,  coherent  and  objective 
measurement  scale;  and  that  the  necessary  data  is  of  good  quality, 
unbiased and complete. All these aspects were tested in the trial.  High 
reliability  means  that  the  indicator  responds  to  all  these  characteristics 
and  therefore  can  be  consistently  measured  over  time  and  can  be 
aggregated across platforms, countries and at the EU level.  

Value  added:  this  refers  to  the  capability  of  the  indicator  to  measure 
progress  towards  the  policy  objective  in  a  clear  and  objective  way 
and/or to have a high explanatory power of the measured phenomenon; 
it  implies  the  presence  of  objective  benchmarks  to  measure  progress. 
This  is  a  very  important  attribute  of  the  indicator,  because  the  value 

45 

 
added  guarantees  the  ultimate  achievement  of  the  main  goals  of  the 
Measurement System.  

Scalability,  comparability  and  representativeness:  The  System  of 
Indicators is structured as a hierarchy of indicators, calculated collecting 
a  series  of  basic  indicators  from  the  e-Procurement  platforms,  and 
aggregating  them  into  synthetic  indicators  by  measurement  area, 
country, and for the entire EU. To do so, we need to verify the capability 
of  the  indicators  to  be  scaled  up,  that  means  aggregated  and 
extrapolated from the measured sample of platforms to the country level 
and  to  the  EU  level,  maintaining  full  comparability  at  all  levels.  This 
means  also  to  verify  that  the  group  of  platforms  providing  the  basic 
is  sufficiently  representative  of  the  EU  e-Procurement 
indicators 
landscape.  In  this  sense,  scalability  is  a  property  of  the  System  of 
Indicators, rather than of the single indicator. It is a key property of the 
Measurement System.  

The  assessment  of  each  of  these  criteria  comes  from  our  expert 
evaluation,  based  as  much  as  possible  on  objective  data  and  the 
experience  of  our  surveys.  We  have  used  a  semantic  scale  in  three 
levels coded with a simple colour key:  

  High feasibility = Green colour 

  Medium feasibility = Amber  

  Low feasibility = Red 

The 
following  paragraphs  present  our  detailed  assessment  of 
availability, reliability and value added for each indicator, while scalability 
is presented separately.  

In  addition,  we  have  also  examined  in  detail  the  main  barriers 
preventing  or  reducing  the  feasibility  of  each  indicator,  to  feed  into  the 
decisions about the future measurement system.  

The main barriers have been identified as follows: 

  Technology  challenges  =  Barriers  relative  to  the  technical 
impossibility  to  collect  data  for  certain  indicators  (for  example, 
when  the  e-Submission  of  bids  is  based  on  PDF  attachments, 
the platform operator does not have visibility on contract values; 
or  the  process  may  not  register  time  elapsed  between  certain 
actions, targeted by our efficiency improvement indicator).  

  Organizational challenges = Barriers relative to the way the e-
Procurement  process  is  organized,  including  practical  barriers 
(for  example  the  platform  does  not  collect  value  data  of  the  e-
Procurement  contracts,  or  the  smaller  economic  operators  are 
not  classified  as  SMEs).  In  some  cases,  the  monitoring  and 
measuring process is simply not open and collaborative. 

  Regulatory  and  policy  challenges  =  Barriers  relative  to 
relevant  regulation  which  may  prevent  the  measurement  of 
certain  indicators,  for  example  when  national  law  forbids  e-
Auctions. As  anticipated,  a  main  problem  concerns  the  lack  of 
authorization  to  the  platforms  (particularly  private  platforms)  to 
share data  with the measurement organization. In addition, the 
for  public 
lack  of 

transparency  and  open  data  policies 

46 

 
procurement  and/or  e-Procurement  represent  an  obstacle  to 
data collection, monitoring and measurement.  

The level of barriers is estimated with the same  type of semantic scale 
as  the  feasibility  assessment  criteria,  but  with  the  opposite  meaning  in 
the colour code: 

  Low barrier = Green  

  Medium barrier = Amber  

  High barrier = Red 

The combination of the feasibility assessment and of the type and level 
of barriers will  lead us to the decisions on the changes of the indicator 
system.  

3.2. Assessment of Take-up Indicators 

The  following  paragraphs  present  the  assessment  of  each  of  the  main 
take-up indicator. The assessment of barriers is presented in paragraph 
3.2.6.   

3.2.1. Assessment  of  Top-level  Take-up  Indicator  of  e-

Procurement  

As shown in Table 17 below, the indicator of take-up of e-Procurement in 
value and volume is characterised by: 

  Medium  level  availability  for  value  and  volume:  the  value 
data is available only in platforms with high technology maturity 
(e-Submission level 3 and 4, as shown in the previous chapter).  
The  platforms  providing  the  most  advanced  services  are  also 
the  platforms  usually  collecting  data 
is 
understandable,  since  where  the  tender  is  submitted  though 
fully interactive processes, monitoring is easier for the platform. 
The  less  advanced  platforms  are  also  less  likely  to  have 
advanced  monitoring  systems  and  to  collect  volume  data  as 
well.  

in  value.  This 

  Medium  level  reliability  for  value,  high  level  for  volume: 
while  the  definition  of  the  number  of  contracts  is  sufficiently 
clear,  there  is  still  some  ambiguity  in  the  definition  of  award 
contract value. This issue arose during the validation process of 
the value estimates, with some of the MS not participating to the 
trial.  The  problem  is  the  definition  of  the  exact  value  and  the 
moment  in  which  the  value  should  be  measured,  compared  to 
the  moment  when  the  tender  is  launched  and  processed.  In 
addition  there  are  issues  of  potential  double-counting,  if  the 
same  contract  is  processed  with  e-Submission,  e-Auction, 
perhaps  e-Ordering.  The  calculation  of  the  value  of  framework 
contracts 
their 
implementation requires further rounds of awarding (sometimes 
through  so  called  call-off  procedures,  sometimes  through  e-
Ordering).  

definitions, 

separate 

requires 

since 

  High  level  value  added:  both  indicators  are  clear,  simple  and 
easy to communicate, with a clear benchmark (100% uptake is 
the target).  

47 

 
3.2.2. Assessment  of  Take-up  Indicator  by  type  of 

purchase 

As shown in Table 17 below, the take-up indicator by type  of purchase 
(supplies, services, and public works) is characterised by: 

  Medium  level  availability  in  value  and  volume:    the  value 
data is available only in platforms with high technology maturity 
(e-Submission level 3 and 4, as shown in the previous chapter). 
Most platforms do not collect data by type of purchase.   

  High  level  reliability  in  value  and  volume:  the  Public 
Procurement  process  clearly  identifies  the  main  typologies  of 
the 
purchases  (supplies,  services  and  public  works)  so 
definition  of  the  indicator  is  well  understood.  We  must  pay 
attention  to  the  risk  of  double  counting  (since  there  are 
contracts providing both supplies and services, for example).  

  High level value added in value and volume: both indicators 
are  clear,  simple  and  easy  to  communicate,  with  a  clear 
benchmark  (100%  uptake  for  each  type  of  purchase  is  the 
target).  

Table 17 - Assessment of Feasibility of Take-up Indicators - I  

Take-up indicators 

Feasibility and Quality Assessment 

Availability 

Reliability 

Value added 

Take-up of e-Procurement in 

value  

Available in platforms 

Still some ambiguity in 

with high technology 

the definition of award 

maturity 

contract value 

High explanatory 

power, clear target 

Take-up of e-Procurement in 

Available but often not 

volume  

collected 

Variable easy to 

understand and 

unambiguous 

Take-up of e-Procurement by 

type of purchase - in value 

Take-up of e-Procurement by 

type of purchase- in volume  

Source: IDC 2013 

Available in platforms 

Variable easy to 

with high technology 

understand and 

maturity 

unambiguous 

Available in platforms 

Variable easy to 

with high technology 

understand and 

maturity 

unambiguous 

High explanatory 

power, clear target 

High explanatory 

power, clear target 

High explanatory 

power, clear target 

3.2.3.  Assessment  of  Take-up  Indicator  by  type  of 

procedure 

As  shown  in  the  Table  18  below,  the  take-up  indicator  by  type  of 
procedure is characterised as follows: 

  Medium level availability in value and volume:  the value and 
volume data are available only by technology mature platforms, 
and few of them collect these data.  

  Medium  level  reliability  in  value  and  volume:  there  are 
country  differences  and  ambiguities  in  the  definition  and 
regulation of the main procedures, also the reasons why certain 
procedures  are  selected  are  different  country  by  country.  The 
validity and comparability of data is questionable.  

48 

 
  Medium  level  value  added  in  value  and  volume:  This 
indicator  consists  of  the  percentage  breakdown  of  all  the 
contracts processed with  e-Submission by the platform by type 
of procedure. This is  interesting information, but at this level,  it 
does not provide additional information about the level of take-
up. The correlation between the intensity of use of the different 
procedures  and  the  level  of  take-up  is  unclear,  and  we  are 
missing  comparison  benchmarks,  so  the  value  added 
is 
questionable.  

3.2.4. Assessment  of  Take-up  Indicator  above/below  the 

EU threshold 

The take-up indicator above/below the EU threshold, as shown in Table 
18 below, is characterised by: 

  Medium  level  availability  in  value  and  volume:  the  data  is 
available  only  in  platforms  with  high  technology  maturity  and 
advanced measurement systems.    

  High  level  reliability  in  value  and  volume:  the  definition  is 
clear  (even  though  there  are  also  national  thresholds  to  be 
taken  into  account)  and  the  quality  of  the  data,  if  collected,  is 
good.  

  High level value added in value and volume: both indicators 
are  clear,  simple  and  easy  to  communicate,  with  a  clear 
benchmark  (100%  uptake  for  each  type  of  purchase  is  the 
target).  The  indicators  above  the  EU  threshold  can  also  be 
compared with TED data.  

3.2.5. Assessment  of  Take-up  Indicator  by  electronic 

procedure 

As  shown  in  Table  18  below,  the  take-up  indicator  by  electronic 
procedure is characterised by: 

  Medium  level  availability  in  value  and  volume:  the  data  is 
available  only  in  platforms  offering  the  electronic  procedure. 
in 
DPS 
approximately  half  of  the  trial  platforms.  If  e-Auctions  are 
performed,  usually  both  value  and  volume  data  is  available. 
Also data on e-Marketplaces is normally collected.     

rarely  used,  while  e-Auctions  are  available 

is 

  High level reliability in value and volume: the identification of 
electronic  procedures 
is  clear  and  well  understood.  The 
indicator  of  take-up  for  e-Auctions  is  clear  and  reliable.  The 
indicator  of  take-up  for  e-Marketplaces  needs  to  be  well 
explained  (whether  the  indicator  refers  to  e-Ordering  on  the 
marketplace, or the contracts processed).  

  High level value added in value and volume: the indicator is 
clear,  simple  and  easy  to  communicate.  It  is  also  relevant  to 
measure the level of uptake of these procedures.   

Table 18 - Assessment of Feasibility of Take-up Indicators - II 

Take-up indicators 

Feasibility and Quality Assessment 

49 

 
Availability 

Reliability 

Value added 

Take-up of e-

Available in platforms 

and ambiguities in the 

Unclear correlation 

Procurement by type of 

with advanced 

definition/ 

btw procedure and 

procedure - in value 

monitoring systems 

classification of 

take-up 

Country differences 

procedures 

Country differences 

Take-up of e-

Available in platforms 

and ambiguities in the 

Unclear correlation 

Procurement by type of 

with advanced 

definition / 

btw procedure and 

procedure - in volume 

monitoring systems 

classification of 

take-up 

procedures 

Take-up of e-

Available in platforms 

Procurement above/ 

with high technology 

below the EU threshold - 

maturity and 

in value 

measurement systems 

Variable easy to 

understand and 

unambiguous 

Take-up of e-

Available in platforms 

Procurement above/ 

with high technology 

below the EU threshold - 

maturity and 

in volume 

measurement systems 

Variable easy to 

understand and 

unambiguous 

Take-up of e-

Available in platforms 

Procurement by 

with high technology 

electronic procedure - in 

maturity and 

value 

measurement systems 

Variable easy to 

understand and 

unambiguous 

Take-up of e-

Available in platforms 

Procurement by 

offering the service  

electronic procedure - in 

and measurement 

volume 

systems 

Variable easy to 

understand and 

unambiguous 

Source: IDC 2013 

High explanatory 

power, clear target 

High explanatory 

power, clear target 

High explanatory 

power, clear target 

High explanatory 

power, clear target 

3.2.6. Main Barriers to Take-up indicators 

The  main  measurement  barriers  are  common  to  the  take-up  indicators 
analysed  in  the  previous  barriers  and  presented  in  Table  17  and Table 
18. These barriers are summarized in Table 19 below and refer primarily 
to the availability of data. More specifically:  

  High technology barrier: this is due to the still immature level 
of  development  of  the  technology  solutions  used  by  many 
platforms (in our trial sample, about half of them), which make it 
difficult to collect particularly value data. There is a natural trend 
of  evolution  of  the  platforms  towards  technology  maturity  with 
substitution  of  old  solutions  with  more  advanced  ones,  but  this 
problem  will  not  disappear  quickly.  To  deal  with  this  challenge, 
the  benchmarking  organization  could  a)  use  a  reduced  set  of 
indicators  for  the  platforms  with  lower  level  technology  and  b) 
recruit  for  measurement  as  many  advanced  platforms  as 
possible.  

50 

 
 
 
  Medium  organizational  barrier:  this  is  due  to  the  low  level  of 
data  collection  processes.  The  main  challenge  is  encouraging 
platforms  to  set  up  more  advanced  monitoring  and  data 
collection  systems.  The  benchmarking  organization  should 
identify incentives to motivate them to do so.   

  High regulatory/ policy barriers: this is due to national policies 
about  data  protection  and  privacy  of  public  procurement 
contracts.  This  prevents  many  platforms,  particularly  private 
ones,  from  having  visibility  on  value  data  or  other  data  about 
contracts, and/or from monitoring this data and sharing it (even 
in  an  anonymous  way)  with  a  benchmarking  organization.  If 
national governments adopted full transparency  and open data 
principles  on  public  procurement, 
this  barrier  could  be 
overcome.  

Table 19 - Main Barriers to Take-up Indicators  

Technology Barrier 

Organizational Barrier 

Regulatory/ Policy Barrier 

High - Need advanced 
technology solutions 

Medium - Need to 
organize monitoring of 
value and volume data 

High - Private Platforms 
need authorization to 
communicate data 

Source: IDC 2013 

3.2.7. Indicator of take-up of e-Procurement by SMEs  

As 

shown 

in 

51 

 
Table  20  below,  the  take-up  indicator  of  e-Procurement  by  SMEs  is 
characterised by: 

  Low  level  availability  in  value  and  volume:    the  data  is 
available  only  if  the  platforms  classify  economic  operators, 
distinguishing  between  large  and  small  enterprises,  based  on 
the  EC  definition.  Unfortunately,  this  happens  very  rarely.  In 
addition, there is the same problem as with all value indicators, 
only advanced platforms are able to see and collect value data.   

  High  level  reliability  in  value  and  volume:  the  indicator 
measures  the  number  and  value  of  SME  wins  of  contracts 
processed  with  e-submission. The  definition  is  clear  (assuming 
previous  agreement  on  the  definition  of  a  contract  won  or 
awarded) and so is the target to increase SMEs participation to 
the process.  

  High level value added in value and volume: the indicator is 
clear, simple and easy to communicate, providing critical data to 
measure  progress 
full 
participation of SMEs in e-Procurement.  

the  policy  objective  of 

towards 

3.2.8. Indicator  of  take-up  of  e-Procurement  by  Cross-

Border suppliers 

Similarly  to  the  SMEs  indicator,  the  take-up  indicator  by  cross-border 
suppliers is characterised by:   

  Low  level  availability  in  value  and  volume:    the  data  is 
available  only  if  the  platforms  classify  economic  operators, 
classifying  them  by  national  origin.  Unfortunately,  this  happens 
very  rarely.  In  addition,  there  is  the  same  problem  as  with  all 
value  indicators,  only  advanced  platforms  are  able  to  see  and 
collect value data.   

  High  level  reliability  in  value  and  volume:  the  indicator 
measures  the  number  and  value  of  cross-border  wins  of 
contracts  processed  with  e-Submission.  The  definition  is  clear 
(assuming  previous  agreement  on  the  definition  of  a  contract 
won  or  awarded)  and  so  is  the  target  to  increase  SMEs 
participation to the process.  

  High level value added in value and volume: the indicator is 
clear, simple and easy to communicate, providing critical data to 
measure  progress 
full 
participation of cross-border suppliers in e-Procurement.  

the  policy  objective  of 

towards 

52 

 
Table  20  -  Assessment  of  take-up  indicators  by  SMEs  and  cross-border 
suppliers  

Take-up indicators 

Availability 

Reliability 

Value added 

Feasibility and quality Assessment 

Take-up of e-

Available in 

Variable easy to 

Procurement by SMEs in 

platforms with high 

understand and 

value  

technology maturity 

unambiguous 

High explanatory 

power, clear target 

Take-up of e-

Procurement by SMEs  in 

volume  

Available but not 

collected 

Variable easy to 

understand and 

unambiguous 

High explanatory 

power, clear target 

Take-up of e-

Available in 

Variable easy to 

Procurement by cross-

platforms with high 

understand and 

border suppliers in value  

technology maturity 

unambiguous 

High explanatory 

power, clear target 

Take-up of e-

Procurement by cross-

Available but  not 

border suppliers  in 

collected 

volume  

Source: IDC 2013 

Variable easy to 

understand and 

unambiguous 

High explanatory 

power, clear target 

3.2.9. Main  Barriers  to  Take-up  indicators  for  SMEs  and 

Cross-Border suppliers 

The  indicators  on  take-up  by  SMEs  and  cross-border  suppliers  have 
(
common 

measurement 

barriers 

53 

 
Table 21) concerning the availability of data as follows:  

  Medium  technology  barrier:  the  technology  barrier  for  these 
indicators  is  moderate,  since  the  availability  of  data  does  not 
depend on the sophistication of the technology solution.  

  High  organizational  barrier:  this  is  the  most  relevant  barrier, 
since  every  platform  uses  different  criteria  to  register  suppliers 
and  almost  never  this  includes  their  classification  as  SMEs 
and/or  cross-border  suppliers.  To  overcome 
this  barrier, 
platforms  should  change  their  approach  to  the  identification  of 
economic operators, both in the initial registration phase and for 
e-Submission activities. 

  High regulatory/ policy barrier: this barrier is equally relevant 
for all take-up indicators, concerning the lack of authorization for 
private  platforms  and  some  public  ones  to  share  data  on  take-
up  with  the  benchmarking  organization,  as  discussed  in  par. 
3.2.6 above.  

54 

 
Table 21 - Main Barriers to Take-up Indicators for SMEs and Cross-border 
suppliers  

Technology Barrier 

Organizational Barrier 

Regulatory/ Policy Barrier 

Medium - need platforms 
with medium/advanced 
technology solutions 

High - Need to add 
classification criteria of SMEs 
and Cross-border operators to 
registration data of suppliers 

High - Private Platforms need 
authorization to communicate 
data 

Source: IDC 2013 

3.3. Assessment of Performance indicators 

The  following  paragraphs  present  the  feasibility  assessment  of  the  e-
Procurement Performance Indicators identified in Table 2. The results of 
the trial in terms of the basic data availability are presented in Table 7 in 
chapter 2.   

3.3.1. Assessment  of  the  e-Submission  Price  Reduction 

Indicator 

As  shown  in  the  Table  22  below,  there  are  several  problems  with  this 
indicator, characterised by low feasibility because of:  

  Low  level  availability:  the  trial  demonstrated  that  very  few 
platforms  collect  data  on  both  reserve  and  award  prices,  let 
alone  calculate  averages  for  all  prices.  In  addition,  there  is  the 
same problem as with all value indicators, since only advanced 
platforms are able to see and collect value data. The necessary 
data  for  this  indicator  could  possibly  be  more  easily  sourced 
from Contracting Authorities instead of platforms.   

  Low level reliability: this indicator has comparability problems. 
The definition of reserve price may vary between countries and 
procedures:  for  example,  it  needs  to  be  qualified  when 
considering  framework  contracts.  This  issue  was  raised  by 
some  stakeholders  during  interviews  and  discussions,  rather 
than during the trial. In addition, to insure comparability we must 
be sure that reserve and award prices are collected at the same 
stage of the process and for the same procedures.  

  Low  level  value  added:  this  indicator  was  designed  to 
measure the savings generated by the e-Procurement process, 
through the reduction of the  award price (that is, the difference 
between  the  reserve  price  and  the  award  price).  In  terms  of 
information value, the indicator is valuable if the comparability is 
guaranteed.  The 
is  more 
questionable  (e.g.  if  the  reduced  price  actually  represents  a 
saving)  and  may  depend  on  the  type  of  purchase,  and/or  the 
award  criteria.  The  OECD7  for  example  considers  the  price 
reduction  a  very  simplistic  approach  to  the  measurement  of 
performance  of  public  procurement.  Some  stakeholders 
interviewed believe that this indicator is valid only in the case of 
e-Auctions, where it is easy to measure at the start and the end 

interpretation  of 

indicator 

the 

7 OECD (2011), “Centralised Purchasing Systems in the European Union”, Sigma Papers, 
No. 47, OECD Publishing, par.8.9 “ 

55 

 
                                                      
of  the  process.  This  kind  of  issues  are  difficult  to  solve  in  the 
case  of  a  European  benchmark,  which  requires  simple  criteria 
of  assessment  and  comparability  to  be  valid  across  multiple 
countries. The value of the indicator would be increased by the 
use  of  historical  data  (for  example,  the  possibility  to  compare 
price  reductions 
traditional  procurement 
procedures).  Therefore,  the  value  added  of  this  indicator  is 
problematic.  

the  past  with 

in 

Table 22 - Assessment of the e-Submission Price Reduction Indicators 

Performance indicators 

Feasibility Assessment 

Availability 

Reliability 

Value added 

E-Submission Price Reduction Indicator 

Data on reserve price and 

Unclear, lack of 

average award price for all 

Low availability, 

Unclear definition 

benchmarks - price 

the contracts processed by 

the platform without e-

not often 

collected 

Submission 

Data on reserve price and 

of reserve/award 

reduction impact 

price 

depends on 

circumstances 

Unclear, lack of 

average award price for all 

Low availability, 

Unclear definition 

benchmarks - price 

the contracts processed by 

the platform with e-

Submission 

Source: IDC 2013 

not often 

collected 

of reserve/award 

reduction impact 

price 

depends on 

circumstances 

3.3.2. Barriers  to  the  e-Submission  Price  Reduction 

Indicator 

Given its low feasibility assessment, the  e-Submission Price Reduction 
indicator presents high barriers as follows (Table 23):  

  High  technology  barrier:  this  indicator  cannot  really  be 
measured  unless  the  platform  is  sufficiently  advanced  to  have 
visibility on value data and be able to monitor all price data.  

  High  organizational  barrier:  the  specific  data  collection  for 
reserve  and  award  prices  must  be  organized  ad  hoc,  as  very 
few platforms do.  

  High  regulatory/  policy  barriers:  as  for  all  the  take--up 
indicators,  many  platforms,  particularly  private  ones,  are  not 
authorised  to  collect  data  on  take-up  and  share  it  (even  in  an 
anonymous way) with a benchmarking organization.  

56 

 
 
Table 23 - Main Barriers to e-Submission Price Reduction Indicator 

Technology Barrier 

Organizational Barrier 

Regulatory/ Policy Barrier 

High - Need advanced 
technology solutions 

High - Need to define basic 
concept and organize data 
collection 

High - Private Platforms need 
authorization to communicate 
data 

Source: IDC 2013 

3.3.3. Assessment  of  the  e-Submission  Reduction  of 

Litigation Indicator 

The  e-Submission  Reduction  of  Litigation  indicator  measures  the 
average  number  of  appeals  per  year  for  all  contracts  awarded  with  e-
Submission  and the  difference  with the average number of appeals for 
contracts processed with traditional procurement. As shown in Table 24 
this indicator has mixed feasibility, because of the following reasons: 

  Low  level  availability:    the  trial  demonstrated  that  platforms 
have no visibility on litigation, which  usually occurs in the post-
award  phase  and  is  managed  by  the  Contracting  Authorities. 
This is essentially a demand-side, not a supply side indicator.  

litigations 

  Medium  level  reliability:  the  indicator  itself  is  clear  and  well-
understood.  There  is  however  a  benchmarking  problem:  to 
measure whether there is improvement in the indicator, we need 
either an historical series (at least two measurements in time) or 
an  external  benchmark  (average 
for  contracts 
processed with traditional procurement).  This is difficult to find, 
particularly  at  EU  level.    The  comparability  between  countries 
and contracts may also create problems: the number of appeals 
in  one  year  may  refer  to  the  contracts  previously  awarded, 
depending  on  how  fast  contracts  are  processed  and  litigations 
managed  in  different  MS.  Finally,  the  frequency  of  litigations 
may  vary  depending  on  the  nature  of  different  procedures  or 
purchases.    This  requires  a  great  deal  of  specifications  and 
qualifications  for  the  data  collection  and  inevitably  reduces  the 
feasibility of the indicator.  

  High level value added: the reduction of litigation is one of the 
e-Procurement  benefits  most  often  mentioned  and  a 
performance  criterion  very  close  to  the  heart  of  many  e-
Procurement  operators.  Therefore,  this  is  potentially  a  very 
useful indicator, measuring progress towards a better efficiency 
of procurement processes.  

57 

 
Table 24 - Assessment of e-Submission Reduction of Litigation Indicator 

Performance indicators 

Feasibility Assessment 

Availability 

Reliability 

Value added 

E-Submission Reduction of Litigation Indicator 

Low availability - visible by 
CA not platforms 

Difficult to measure 
comparability 

High explanatory 
power, clear target 

Low availability - visible by 
CA not platforms 

Difficult to measure 
comparability 

High explanatory 
power, clear target 

Average number of 
appeals per contract 
processed with e-
Submission  
Average number of 
appeals per contract 
processed  without e-
Submission; 

Source: IDC 2013 

3.3.4. Barriers 

to 

the  e-Submission  Reduction  of 

Litigation Indicator 

The main barriers for this indicator concern the following factors: 

  Low  technology  barrier:  data  collection  is  not  influenced  by 

the technology solution. 

  High  organizational  barrier:  the  main  challenge  for  this 
indicator  concerns  the  organization  of  data  collection,  which 
must  be  sourced  from  Contracting  Authorities.  This  type  of 
indicator is not usually collected, so strong motivations would be 
required.  

  Medium  regulatory/  policy  barrier:  setting  up  monitoring  and 
measurement  processes  for  this  indicator  would  probably 
require  support  by  policy  authorities  overseeing  public 
procurement  processes,  perhaps  driven  by  a  transparency 
policy  aiming  at  monitoring  the  evolution  of  litigation  and  its 
causes.   

Table 25 - Main Barriers to e-Submission Reduction of Litigation Indicator 

Technology Barrier 

Organizational Barrier 

Regulatory/ Policy Barrier 

Low technology barrier 

Source: IDC 2013 

High: Contracting Authorities 
must set up ad-hoc 
monitoring   

Medium: requires involvement 
and motivation of Contracting 
Authorities probably inspired by 
transparency policies   

58 

 
3.3.5. Assessment 

the 
Improvement Indicator 

of 

e-Tendering 

Efficiency 

This indicator measures the average time spent to prepare and process 
a  tender  in  the  pre-award  phase  (mandays  or  hours)  by  Contracting 
Authorities  and  separately  by  economic  operators,  as  a  proxy  of  the 
efficiency  benefits  of  e-Procurement.  The  trial  tested  it  only  for 
Contracting Authorities,  but  the  main  issues  are  likely  to  be  similar  for 
economic operators.  

This  indicator  has  rather  low  feasibility  (Table  26),  because  of  the 
following reasons: 

  Low level availability: the trial demonstrated that this indicator 
is  difficult  to  collect,  since  it  requires  monitoring  the  time spent 
by  Contracting  Authorities  and/or  economic  operators  on  the 
system. This is not always recorded, or recordable.  

  Low level reliability: the trial revealed that the definition of the 
basic indicator is not sufficiently clear. There is a need to specify 
well the starting and ending point of the measurement, to insure 
comparability. The time spent may vary depending on the nature 
of different procedures or purchases. There is also a problem of 
measurement scale (days, hours...). 

  Medium level value added: efficiency benefits are important to 
measure, so this is potentially a  very useful indicator. However, 
there  is  a  lack  of  benchmarks  allowing  evaluation  (how  much 
time  is  efficient,  how  much  is  too  much?).  In  addition,  simply 
measuring time spent from the start  of the  process might even 
(more  efficient  Contracting  Authorities  or 
be  misleading 
suppliers  will  start  early,  others  at  the  last  minute:  so  the 
measurement  of  time  spent  must  be  well  qualified).  This 
indicator  requires  further  study  and  analysis,  perhaps  through 
case studies, to define the measurement scale and  identify the 
most feasible data collection methodology.  

 Table 26 - Assessment of e-Tendering Efficiency Improvement Indicator 

Performance 
indicators 

E-Tendering Efficiency 
Improvement  indicator 
for Contracting 
Authorities  

Source: IDC 2013 

Feasibility Assessment 

Availability 

Reliability 

Value added 

Low availability 
difficult to collect 

Ambiguous definition, 
unclear measurement 
scale 

High potential value 
added, but lack of 
benchmarks and 
difficult comparability 

3.3.6. Barriers 

to 

the 

e-Tendering 

Efficiency 

Improvement Indicator 

From  the  point  of  view  of  barriers,  this  indicator  presents  the  following 
problems (Table 27): 

  Medium  technology  barrier:  the  e-Procurement  technology 
solution  must  allow  recording  the  time  spent  to  implement 
certain  tasks  in  the  pre-award  process:  this  is  not  possible  if 
some phases are implemented offline (for example, if the tender 
is sent as a PDF attachment). 

59 

 
  High  organizational  barrier:  the  indicator  requires  modelling 
the pre-award process and defining the optimal time interval  to 
perform  the  pre-award  tasks. The  data  collection  must  also  be 
set  up  and  organized  to  collect  the  relevant  data.  These 
activities are not easy to design and implement.  

  Low  regulatory/  policy  barrier:  there  is  no  specific  policy  or 

regulatory barrier for this indicator.  

Table 27 - Main Barriers to e-Tendering Efficiency Improvement Indicator 

Technology Barrier 

Organizational Barrier 

Regulatory/ Policy Barrier 

Medium technology 
barrier 

High: requires reliable 
measurements of the 
pre-award process tasks 

Low: not a policy or 
regulatory problem   

Source: IDC 2013 

3.3.7. Assessment of the Ease of Access Indicator 

The  Ease  of    Access  indicator  measures  the  level  of  usability  of  the 
platform,  based  on  the  authentication  requirements  imposed  by  the 
provider 
service 

60 

 
Table 28). This indicator is characterised by high feasibility as follows: 

  High 

the 

level  availability: 

trial  demonstrated 

the 
necessary  data  is  available  and  easy  to  collect. Authentication 
requirements  must  be  publicised  to  all  the  users  of  the 
platforms,  even  if  sometimes  those  concerning  e-Submission 
are not immediately visible.  

that 

  High level reliability: the indicator is clear and well understood. 
The  trial  confirmed  that  authentication  requirements  are  clearly 
identified by the platforms, even if they are different by phase of 
the process (typically, much lower for registration and access to 
the  platform,  higher  for  e-Submission).  There  is  a  need 
therefore to qualify the indicator by phase, rather than asking a 
generic question for the whole process. The measurement scale 
(which  type  of  requirements  correspond  to  which  level  of  the 
scale)  were  broadly  confirmed  by  the  trial  but  require  further 
reflection and validation together with the main stakeholders.  

  High  level  value  added:  this  indicator  has  very  high  value 
added  since  improving  the  ease  of  access  to  platforms  is  a 
for  SMEs  and  cross-border 
(particularly 
policy  objective 
operators). There is also a clear target to achieve.  

61 

 
Table 28 - Assessment of Ease of Access and Transparency Indicators 

Performance indicators 

Feasibility Assessment 

Availability 

Reliability 

Value added 

E-Submission Ease of access indicator 

Type of e-Authentication required to 
access the platform and its main 
services (e-Notification, e-Submission) 
per type of supplier (domestic vs. 
cross-border 

Source: IDC 2013 

High availability, easy 
to collect 

Clear indicator, 
presence of 
benchmark 

High explanatory 
power, clear target 

3.3.8. Barriers to the Ease of Access Indicator 

There  are  no  particular  technological,  organizational  or  regulatory 
barriers to this indicator.  

Table 29 - Main Barriers to Ease of Access Indicator 

Technology Barrier 

Organizational Barrier 

Regulatory/ Policy Barrier 

Low technology barrier 

Low barrier  

Low barrier  

Source: IDC 2013 

3.3.9. Assessment of the Transparency Indicator 

The Transparency indicator is characterised by high feasibility as shown 
by the Table 30 below:  

  High 

level  availability: 

the 

trial  demonstrated 

that 

the 

necessary data is available and easy to collect.  

  Medium  level  reliability:  the  objective  of  the  indicator  is  clear 
and  well  understood.  However,  this  is  a  composite  indicator, 
calculated  as  the  average  transparency  of  7  categories  of 
information and communication services offered by the platform. 
The scoring scale is based on the study team evaluation of what 
is  the  minimum  level  of  transparency  necessary  for  a  correct 
use  of  the  platform,  and  what  is  the  excellent  level,  for  each 
information category, as explained in par. 2.5.2. This evaluation 
is based on our research and the results of the Lot2 analysis of 
good practices and users' requirements in terms of information. 
However,  the  solidity  and  acceptance  of  the  indicator  would 
improve if the scoring were followed by a validation phase with 
stakeholders, which was not foreseen in the trial.  

  High  level  value  added:  this  indicator  has  very  high  value 
added  and  a  clear 
information 
target,  since 
transparency  of  the  procurement  process  is  a  policy  objective 
both  for  the  effectiveness  and  efficiency  of  the  e-Procurement 
process,  and  in  a  wider  sense  in  the  context  of  the  open 
government policies.  

improving 

Table 30 - Assessment of Transparency Indicator 

62 

 
 
Performance indicators 

Feasibility Assessment 

Availability 

Reliability 

Value added 

Transparency Indicator of e-Procurement  for buyers and suppliers 

Type of information provided to 
suppliers and buyers, based on 
a check list provided by the 
benchmarking organization 

Source: IDC 2013 

High availability, easy 
to collect 

Needs consensus on 
the measurement 
scale 

High explanatory 
power, clear target 

3.3.10. 

Barriers to the Transparency Indicator 

There  are  no  particular  technological,  organizational  or  regulatory 
barriers to the measurement of this indicator.  

Table 31 - Main Barriers to Transparency Indicator 

Technology Barrier 

Organizational Barrier 

Regulatory/ Policy Barrier 

Low technology barrier 

Low barrier  

Low barrier  

Source: IDC 2013 

3.4. Availability  of  Additional  Data  from  other 

public sources 

The  assessment  of  availability  of  data  in  this  chapter  is  focused  on  e-
Procurement platforms as the main source of data.  

However,  to  maximise  the  value  added  of  the  indicators  and  to  insure 
comparison  of  performance  and  benefits  between 
traditional 
procurement  and  e-Procurement,  the  Measurement  System  needs 
additional data which must be sourced from other public sources, at the 
MS level or at EU level, for example from Eurostat.  

The list of data needed from other sources is presented in the following 
Table  32  and  Table  33.  At  the  moment,  these  data  are  not  available 
across the EU and in a comparable way, even if some MS collect some 
of these data.  

63 

 
Table  32  -  Additional  Data  collection  needs  for  Take-up  indicators  –  Data 
needed from MS or Eurostat  

Type of data – related to the reference measurement period (1 year) 

Possible Source  

Total number of public procurement contracts awarded – by MS and at EU 
level  

MS and/or  Eurostat 

Total value of public procurement contracts awarded – by MS and at EU 
level 

MS and/or  Eurostat 

Total number of public procurement contracts awarded to SMEs – by MS 
and at EU level  

MS and/or  Eurostat 

Total value of public procurement contracts awarded to SMEs – by MS and 
at EU level 

MS and/or  Eurostat 

Total number of public procurement contracts awarded to cross-border 
economic operators – by MS and at EU level  

MS and/or  Eurostat 

Total value of public procurement contracts awarded to cross-border 
economic operators – by MS and at EU level 

MS and/or  Eurostat 

Total number of public procurement contracts awarded by type of purchase 
(supplies, services, public works) – by MS and at EU level  

MS and/or  Eurostat 

Total value of public procurement contracts awarded by type of purchase 
(supplies, services, public works – by MS and at EU level 

MS and/or  Eurostat 

Source: D3 Performance Indicators Report - IDC, 2012 

Table 33 - Additional data collection needs for Performance indicators  

Type of data  

Average reserve price and average award price for all the public 
procurement contracts in the MS / across the EU 

Possible Source  

MS and/or  Eurostat 

Average number of appeals per contract for all the public procurement 
contracts in the MS / across the EU 

MS and/or  Eurostat 

Average time needed by a CA to prepare and implement a pre-award 
process with traditional procurement 

MS, independent 
studies 

Average time needed by a supplier to prepare and implement a pre-award 
process with traditional procurement 

MS, independent 
studies 

Source: D3 Performance Indicators Report - IDC, 2012 

3.5.  Assessment of Scalability, Comparability and 

Representativeness 

3.5.1. Assessment of scalability 

The  Indicator  System  is  based  on  a  hierarchy  of  indicators,  calculated 
elaborating a series of basic indicators based on data collection from a 
sample  of  e-Procurement  entities.  The  data  collected  is  structured  into 
basic indicators in order to allow comparability. The basic indicators are 
aggregated  into  synthetic  indicators  per  measurement  area  or  per 
Member State (country); country level indicators are in turn aggregated 
in top level EU indicators (Table 34).  

The  aggregation  of  the  indicators  is  based  on  the  calculation  of 
averages  or  other  algorithms,  which  may  be  weighted,  if  necessary,  to 

64 

 
take  into  account  variations  in  relevance  by  type  of  platform,  MS 
population, type of service or other factors. This is a modular approach, 
which  in  turn  guarantees  the  flexibility  and  comparability  of  the 
indicators,  since  it  allows  the  combination  and  disaggregation  of  basic 
indicators in many different ways. 

Table 34 - Hierarchy of the System of Indicators 

Source: D3 Performance Indicators Report, IDC 2012 

However,  the  solidity  of  the  system  depends  on  the  capability  of  the 
indicators  to  be  scaled  up,  that  means  aggregated  and  extrapolated 
from the measured sample of platforms to the country level and the EU 
level,  maintaining  full  comparability  at  all  levels.  The  scalability  of  the 
indicators depends on the suitability of the panel of platforms involved in 
the  measurement  system  to  provide  all  the  data  needed  and  to  be 
sufficiently representative of the EU e-Procurement landscape.  

Based  on  the  trial  results,  the  scalability  of  the  basic  indicators  is 
presently  insufficient,  because  of  the  lack  of  basic  data  for  many 
indicators  and  because  of  the  lack  of  representativeness  of  the 
platforms sample, which excluded private platforms.  

3.5.2. Main barriers to scalability and representativeness 

The  main  barriers  preventing  the  development  of  a  measurement 
capacity insuring scalability and representativeness of the indicators are 
the following:  

  Medium  technology  barrier:  the  technology  immaturity  of 
many  of  the  platforms  identified  by  the  trial  reduces  the 
availability  of  value  data,  weakening  the  representativeness  of 
the sample and the scalability of results. However, this problem 
will  improve  in  time,  as  platforms  implement  more  advanced 
technology solutions.  

  High  organizational  barrier:  There  are  at  least  two  main 
challenges to be solved, in order to improve the data collection 
activities  by  MS  and  platforms.  1)  Finding  ways  to  provide 
stronger 
to 
participate in the Measurement System, by identifying the value 
added  for  them.  2)  Promoting  better  monitoring  processes  in 
national  procurement  ecosystems  to  step  up  the  collection  of 
data.  

incentives  and  motivations 

the  platforms 

for 

  Medium  regulatory  and  policy  barrier:  the  lack  of  proactive 
transparency  and  open  data  policies  at  national  level  on  e-
Procurement  represents  an  obstacle  to  the  implementation  of 
widespread monitoring and measurement processes. From this 

65 

Basic IndicatorBasic IndicatorBasic IndicatorBasic IndicatorBasic IndicatorBasic IndicatorBasic IndicatorBasic IndicatorBasic IndicatorBasic IndicatorBasic IndicatorBasic IndicatorTop Level Indicator EUSynthetic  Indicator Country 1Synthetic  Indicator Country xSynthetic Indicator Platform 1Synthetic Indicator Platform 2Synthetic Indicator Platform xSynthetic Indicator Platform 1Synthetic Indicator Platform 2Synthetic Indicator Platform x 
 
point of view, the main challenge would probably be to motivate 
the national procurement monitoring and regulation agencies to 
pay  more  attention  to  e-Procurement  processes.  Finally,  as 
anticipated,  it  would  be  very  helpful  if  national  procurement 
agencies  authorised  formally  private  platforms  to  collect  and 
share data with the benchmarking organizations.   

3.6. Conclusions  on  the  revision  of  the  System  of 

Indicators 

Based  on  this  detailed  feasibility  assessment,  we  present  here  our 
suggestions for the revision of the system of indicators.  

For  each  indicator,  based  on  the  assessment  of  feasibility  we  indicate 
the suggested action, that is: 

  Keep indicator as is 

 

Improve indicator, explaining the action needed 

  Substitute indicator, explaining the action needed 

  Drop indicator, explaining why 

3.6.1. Revision of Take-up indicators  

Take-up indicators have  very high  value added  and it would be difficult 
to find equivalent indicators monitoring the achievement of key policies. 
Therefore,  our  recommendation  is  to  maintain  them,  even  if  they  have 
some  feasibility  (particularly  availability)  problems,  which  can  be 
managed.  

More specifically, we suggest: 

  To  improve  the  take-up  indicator  in  value  by  providing  a 
more  accurate  and  specific  definition  of  the  basic  unit  of 
the 
measurement  (value  of  awarded  contract)  clarifying 
potential  misunderstandings  which  emerged  from  the  study, 
particularly concerning framework contracts; 

  To keep the take-up indicator in volume as is; 

  To keep the take-up indicator in value and volume by type 

of purchase as is; 

  To  drop  the  indicator  on  take-up  by  type  of  procedure 
because  of  its  feasibility  problems  and  questionable  value 
added.  Data  on  procedures  should  still  be  collected,  but  used 
as supporting information. 

  To  keep  the  take-up 

indicator  above/below  the  EU 

threshold in value and volume as is; 

  To  substitute 

the 

take-up 

indicator  of  electronic 
procedures (e-Auction, DPS, e-Marketplace) by splitting it in 
3 separate indicators. E-auction can be measured with a simple 
dedicated  indicator  of  take-up  in  volume  and  value.  The  e-
Marketplace  indicator  should  be  better  defined,  possibly  by 
focusing on e-Ordering only (as each e-marketplace measures 
transactions  differently).  The  indicator  of  DPS  take-up  also 
needs to be revised to insure comparability.  

Table 35 - Revision of take-up indicators 

66 

 
Take-up indicators 

Feasibility and quality Assessment 

Availability 

Reliability 

Value 
added 

Suggested Action 

Take-up of e-Procurement in value  

Medium 

Medium 

High 

Improve Indicator  

Take-up of e-Procurement in volume  

Medium 

High 

High 

Keep indicator as is 

Take-up of e-Procurement by type of 

purchase - in value and volume 

Take-up of e-Procurement by type of 

procedure - in value and volume 

Take-up of e-Procurement above/ 

Medium 

High 

High 

Keep indicator as is 

Medium 

Medium 

Medium 

Drop indicator 

below the EU threshold - in value and 

Medium 

High 

High 

Keep indicator as is 

volume 

Take-up of e-Procurement by 

electronic procedure - in value and 

Medium 

High 

High 

Substitute indicator 

volume 

Source: IDC 2013 

3.6.1. Revision  of Take-up indicators  by  SMEs  and Cross-

border Suppliers 

The  indicators  of  take-up  by  SMEs  and  Cross-border  suppliers  have 
high reliability and value added (Table 36).  

The only problem is lack of availability because the data is not collected. 
The suggestion is: 

 

Improve  the  take-up  indicators  by  SMEs  and  cross-border 
suppliers  by  working  with  platforms  to  increase  the  availability, 
through  better  classification  of  suppliers  in  the  registration 
phase.  

67 

 
Table  36  -  Revision  of  take-up  indicators  by  SMEs  and  Cross-border 
Suppliers 

Take-up indicators 

Feasibility and quality Assessment 

Availability 

Reliability 

Value 
added 

Suggested Action 

Take-up of e-Procurement by SMEs 

in value and volume 

Low 

High 

High 

Improve Indicator 

Take-up of e-Procurement by cross-

border suppliers in value and volume 

Source: IDC 2013 

Low 

High 

High 

Improve indicator 

3.6.2. Revision of Performance indicators 

Some Performance Indicators proved difficult to implement, while others 
resulted  highly  valuable.  Based  on  the  trial  experience  we  suggest  the 
following changes, as summarised in Table 37: 

  Keep  as  top  level  EU  indicators  the  two  indicators  which 

proved highly feasible, that is:  

o  E-Submission  Ease  of  Access 

indicator:  keep 

indicator as is; 

o  E-Submission  Transparency 

improve 
indicator  by  improving  the  measurement  scale  of  the 
levels of transparency through validation and feedback. 

indicator: 

  Drop  in  the  short  term  /  improve  in  the  medium  term  the 
indicator on e-Submission price reduction. This indicator has 
low feasibility, reliability and value added.  In the short term it is 
necessary to drop it. However, since the objective (measure the 
economic  benefits  of  e-Procurement)  is  very  relevant,  it  is 
worthwhile to carry out further research to improve the indicator. 
For  example,  the  problem  of  data  collection  could  be  solved 
with  other  data  sources  (for  example 
from  Contracting 
Authorities, or contract notices sent to TED). In addition, there is 
a  need  to  evaluate  exactly  its  value  added,  possibly  on  the 
basis of case studies.  

  Drop in the short term/  Improve in the medium term  the e-
Submission reduction of litigation indicator. In this case, the 
indicator  has  high  value  added,  but  feasibility  problems  since 
the  data  must  be  collected  from  Contracting  Authorities. 
Therefore,  this  indicator  is  not  feasible  in  the  context  of  a 
measurement  based  on  a  panel  of  platforms  and  must  be 
dropped  in  the  short  term.  But  it  could  be  reintroduced  if  the 
data collection was sourced from Contracting Authorities and/or 
we found a different way to measure it. It is worthwhile to carry 
out further research.  

  Substitute 

the  e-Tendering  efficiency 

improvement 
indicator.  The  data  for  this  indicator  was  collected  by  some 
platforms,  but  the  results  were  of  difficult  interpretation.  The 
assessment of efficiency is more suited to a methodology based 
on  case  studies,  best  practices,  and/or  bench-learning 

68 

 
approaches.  It  is  not  feasible  in  the  context  of  a  measurement 
based on a panel of platforms.  It must be dropped in the short 
term, and substitute indicators should be designed.  

Table 37 - Revision of Performance Indicators 

Performance indicators 

Availability 

Reliability 

Value 
added 

Suggested Action 

E-Submission Price Reduction 
Indicator 

Low 

Low 

Low 

E-Submission Reduction of Litigation 
Indicator 

Low 

Medium 

High 

Drop short-term/ 
Improve indicator  

Drop short term / 
Improve indicator 

E-Tendering Efficiency Improvement  
indicator  

Low 

Low 

Medium 

Substitute indicator 

E-Submission Ease of access 
indicator  

E-Submission Transparency 
indicator  

Source: IDC 2013 

High 

High 

High 

Keep indicator as is 

High 

Medium 

High 

Improve indicator 

3.6.3. Update of the System of Indicators 

In  conclusion,  given  these  considerations,  we  suggest  to  refocus  the 
system  of  indicators  on  a  shorter  list  of  priorities,  focusing  primarily  on 
take-up in value and volume broken down by a few variables (Table 38).  

The  Performance  Indicators  in  the  short  term  should  be  limited  only  to 
the Ease of Access and Transparency Indicators, while further research 
(
is 

Performance 

Indicators 

needed 

other 

the 

for 

69 

 
Table 39).  

Table 38  - Updated Take-Up Indicators 

Policy Goals 

Full transition to e-
Procurement 

Policy Goals 

EU Top level Take-up indicators 

E-Procurement take-up in value (% of total public procurement)  

E-Procurement take-up in volume (% of total number of contracts) 

Other EU Take-up indicators 

Full participation of 
SMEs to e-Procurement 

E-Procurement take-up by SMEs in value and volume (over total public 
procurement)  

Enablement of cross-
border e-Procurement 

E-Procurement take-up by cross-border economic operators (in value and 
volume (over total public procurement) 

E-Procurement take-up in value and volume by type of purchase (% 
supplies, services, public works)  
E-Procurement take-up in value and volume above/below the EU threshold 
(% above/ below of all contracts processed with e-Submission)  
E-Auction take-up in value and volume (total value and number of 
contracts) 

E--Ordering take-up in value and volume (total value and number of orders) 

Full transition to e-
Procurement 

Source: IDC 2013 

70 

 
Table 39 - Updated e-Procurement Performance Indicators 

Policy Goals 

EU Top level Performance indicators 

Improve ease of access of 
public procurement 

E-Submission Ease of access indicator (based on authentication 
requirements) 

Improve transparency of 
public procurement process 

E-Submission Transparency Indicator for buyers and economic 
operators (based on the type of information and communication 
given to buyers and suppliers) 

Policy Goals 

EU Top level Performance indicators –  requiring further 
research 

Improve Effectiveness of 
public spending 

Improve efficiency of 
procurement processes 

Source: IDC 2013 

E-Submission Price Reduction Indicator (average difference 
between the asking price (reserve price) and the award price of 
contracts, as a measure of savings)  

E-Submission Reduction of Litigation indicator (average number of 
appeals for e-Procurement contracts) 

E-Tendering Efficiency Improvement  indicator for Contracting 
Authorities (based on the average time needed for the pre-award 
tendering process) 

E-Tendering Efficiency Improvement  indicator for economic 
operators (based on the average time needed for the pre-award 
tendering process) 

71 

 
 
4 .  D E V E L O P M E N T   O F   A   P A N - E U R O P E A N  

P E R F O R M A N C E   M E A S U R E M E N T  
S Y S T E M  

4.1. Overview 

This  chapter  presents  the  high-level  implementation  plan  for  the 
development  of  a  Pan-European  e-Procurement  Performance 
Measurement system, building on the results of the study.  

This  Measurement  system  was  originally  planned  as  an  EU-wide 
network  for  the  extraction,  elaboration  and  transfer/exploitation  of  high 
quality  performance  data  and  reports. The  network  was  meant  to  build 
on  a  panel  of  e-Procurement  platforms,  representative  of  the  EU  e-
the  e-
Procurement 
landscape,  delivering  data  generated 
Procurement  systems  themselves  (either  automatically,  or  with  a 
reasonable effort), through sustainable information flows.  

from 

It  is  useful  reminding  that  the  main  objectives  of  the  measurement 
system are the following: 

  To measure performance (take-up, efficiency and effectiveness) 
of  the  e-Procurement  systems,  at  present  and  in  time,  feeding 
trend analysis; 

  To  measure 

the  achievement  of 

the  main  goals  of  e-
Procurement  policies,  first  of  all  the  smooth  migration  from 
paper-based  to  electronic  procedures  across  Europe;  but  also  
the  level  of  participation  of  SMEs  and  cross-border  bidders  in 
Public Procurement; 

  To be linked with the assessment of the key success factors and 
barriers  to  e-Procurement  take-up,  able  to  provide  insights  for 
practical and policy action by main stakeholders.  

After 18 months of research and analysis of the EU27, we are now in a 
position to verify the initial assumptions of this project and assess what 
is  the  best  way  to  move  forward  towards  the  development  of  this 
measurement system. 

The main conclusion of our research is that, when writing this report and 
at this development stage of e-Procurement, the EU-wide network of e-
Procurement platforms is much more difficult to develop than expected, 
and  in  any  case  will  not  be  sufficient  to  generate  the  high  quality  data 
we are aiming for. The feasibility barriers are higher and more complex 
than expected.  

A  main  reason  is  that  the  European  level  of  development  is  less 
advanced  than  originally  expected,  but  there  are  also  other  reasons. 
More specifically: 

  The  availability  of  data,  particularly  of  value  data,  is  limited  by 
the  high  number  of  e-Procurement  platforms  with 
low 
technology maturity. In our trial sample, about half of the leading 
public platforms from 12 different EU MS were using low-level e-
Submission  solutions.  The  lack  of  availability  of  data  has 
reduced  our  ability  to  complete  the  design  and  description  of 
most indicators.  

72 

 
  Monitoring  and  data  collection  on  e-Procurement  is  under-
developed, at platform level, country level, and EU level. Private 
platforms tend to have more advanced monitoring systems than 
public platforms, but are not authorized to communicate data to 
third  parties.  Given  the  additional  effort  needed  to  collect  the 
data  we  were  asking  for,  there  is  a  need  to  provide  stronger 
motivation  for  platforms  and/or  make  it  easier  for  them  to 
respond to our requests. 

  The  still  marginal  take-up  of  e-Procurement  means  that  public 
procurement  monitoring  authorities  pay  little  attention  to  it,  or 
consider  it  only  an  alternative  service  channel  instead  of  an 
alternative  business  process.  Without  the  support  of  national 
policy, it is also difficult to motivate platforms towards organizing 
data collection as needed.  

  The  transparency  of  public  procurement  flows,  particularly  at 
regional  and  local  level,  is  quite  low  in  most  countries,  with 
Contracting  Authorities  protecting 
their  autonomy  and 
uninterested  to  share  information  and  data.  There  is  little 
systematic  monitoring,  and  very  little  good  quality,  comparable 
data  about  public  procurement  on  key  aspects  such  as  SMEs 
wins.  This  prevents  the  collection  of  complementary  data 
needed  to  benchmark  the  indicators  and  does  not  create 
incentives for CA to collaborate to the measurement system.  

  The  fragmentation  of  the  e-Procurement  landscape  by  country 
means  that  each  MS  follows  its  own  development  path,  and 
there  are  difficulties  in  finding  common  denominators,  building 
and agreeing on comparable indicators.  

important, 

the  national  ecosystems  are  multi-
Finally,  but  most 
stakeholder  systems  (see  Figure  2):  the  relationship  between  public 
buyers  and  suppliers,  intermediated  by  the  e-Procurement  entities, 
acting    as  specialised  intermediaries,  is  at  the  core  of  the  ecosystem, 
but national policy makers and vendors are also influencing the system, 
by shaping the regulatory environment and the technology solutions.  

This  leads  to  the  following  considerations  about  the  way  forward  to  a 
Pan-European measurement system: 

  While  the  panel  of  e-Procurement  platforms  remains  the  key 
building  block  of  the  measurement  system,  the  other  main 
categories of actors must be actively involved as well. There is 
an important role to play for national procurement authorities (to 
promote data transparency and to provide complementary data 
for  benchmarks)  and  there  is  a  need  to  collect  data  also  from 
the demand side that is the Contracting Authorities.  

  The measurement system should have a national dimension, to 
take  into  account  national  specificities,  the  different  level  of 
development  of  e-Procurement  in  each  country,  and  to  insure 
representativeness of data collection per country, to enable the 
scalability  of  the  indicators.  It  is  likely  that  providing  data  and 
analysis  at  the  national  level  would  go  a  long  way  towards 
contributing to motivate  platforms and contracting  authorities in 
each  country.  In  addition,  a  validation  phase  at  national  level 

73 

 
would  be  a  way  to  make  national  actors  take  responsibility  for 
the quality of data and indicators.  

In  practice,  the  Measurement  system  requires  a  further  phase  of 
development,  in  order  to  complete  the  preliminary  work  started  by  this 
project and to deal at least partially with some of the identified barriers. 
Only  after  this,  we  can  plan  to  "rollout"  fully  the  measurement  system, 
move 
full 
measurement of indicators.  

towards  automated  data  collection  and  achieve 

the 

4.2. Suggested Approach 

Based  on  these  considerations,  in  order  to  overcome  the  main 
the  Measurement  System 
availability  barriers,  we  suggest 
development strategy should be articulated as follows: 

that 

  Aim  for  measurement  of  the  updated  list  of  indicators 
selected by this study, with priority to take-up indicators in the 
short  term.  Further  work  should  be  done  on  performance 
indicators to improve their feasibility. 

  Set  up  data  collection  for  the  basic  indicators  through  a 
standardized  contract  award  notice,  formalized  by  CEN,  the 
European  Committee 
therefore 
accepted  by  all  the  MS  as  a  national  standard.  The  contract 
award notice is a mandatory requirement for all public contracts 
above the  EU threshold and  will  be updated  to be aligned  with 
the new Public Procurement Directives.  

for  Standardization,  and 

  Continue  data  collection  with  a  panel  of  e-Procurement 
platforms  willing  to  cooperate,  focusing  on  the  most  important 
national  hubs  and  CPBs,  providing  relevant  data  and  insights 
about e-Procurement take-up at national level.  

 

 

Identify  and  recruit  a  panel  of  Contracting  Authorities  to 
collaborate  for  data  collection  and  help  to  solve  feasibility 
problems, including ways to measure performance indicators.  

Implement ways to incentivise and reward the collaboration 
for  example 
of  platforms  and  Contracting  Authorities, 
providing  personalised  “dashboards”  and  statistical  reports 
based on the data collected.   

  Launch  a  web  portal,  acting  as  a  key  reference  point  of  the 
Measurement system, providing access to all the data collected, 
publishing the results of the indicators measurement, supporting 
communication  and  promotion  initiatives  of  the  Measurement 
system. This portal can be linked with the Golden Book portal to 
maximise  synergies  and  reinforce 
the  EU  community  of 
stakeholders.  

The use of a CEN standard  as the main pillar of the data collection for 
the indicators presents several advantages: 

It  will  solve  the  availability  and  scalability  problems,  since  the 
data  should  in  principle  cover  the  entire  universe  of  contracts 
awarded above the EU threshold; 

It  will  provide  common  standard  definitions  for  measuring  e-
Procurement  at  national  level,  which  may  in  the  future  be 

 

 

74 

 
extended  by  MS  to  contracts  of  value  below  the  EU  threshold. 
They  could  also  be  included  in  national  monitoring  reports  of 
public  procurement,  and  possibly  lead  to  the  collection  of 
comparable  data  to  benchmark  effectiveness  and  efficiency  of 
public spending through e-Procurement (for example savings or 
reduction of litigation).  

The main outputs of the Measurement System should be the following:  

  An  "e-Procurement  Performance  Indicators  Handbook" 
based  on  clear  and  consensus-based  definitions,  periodically 
updated; 

 

“e-Procurement  Landscape”  and  "e-Procurement  State  of 
Play" reports, periodically updated; 

  An "Indicator toolkit" available on the website portal for all the 
interested parties, to make available the methodologies used to 
measure the indicators.  

4.3. Governance of the Measurement System 

the  Measurement  System 

flexible 
In  our  plan, 
infrastructure that  will grow  year per  year and adapt to the evolution  of 
the market. As shown in the Figure 6 below, the system will include the 
following main actors:  

living  and 

is  a 

  The  European  Commission  DG  MARKT  will  oversee  the 
development  of  the  Measurement  system,  provide  strategic 
guidance,  decide  the  main  policy  objectives,  appoint  and 
delegate  the  Benchmarking  Organization,  manage  the  lifecycle 
of  the  Indicator  system  (by  deciding  when  and  how  to 
add/eliminate  indicators  or  modify  them).  The  EC  will  be 
responsible 
the  outputs  (main 
deliverables).  

to  approve  and  sign  off 

  The Benchmarking organization will support the EC and CEN 
in the definition of the standardized contract award notice. It will 
also  be  responsible  for  the  complementary  data  collection 
through  the  organization  of  Panels,  for  the  measurement  of 
indicators  drawing  on  the  data  from TED  (when  available)  and 
from  Panels,  and  for  the  production  of  deliverables.  The 
Benchmarking  organization  will  also  set  up  and  run  the  web 
portal acting as hub of the Measurement System.  

  CEN  will  develop  the  standardized  contract  award  notice,  with 
the  advice  of  the  Benchmarking  organization  based  on  the 
results of this study and the 2013 trial; 

  Panels  of  e-Procurement  Platforms  and  Contracting 
Authorities willing to collaborate with the EC for data collection, 
validation  and  interpretation  of  results,  and  the  development  of 
indicators.  

  All  the  Contracting  Authorities  in  the  27  EU  awarding 
contracts  above  the  EU  threshold,  who  are  expected  to  send 
their notices to TED, the Tenders Electronic Daily Supplement to 
the Official Journal of the EU. 

75 

 
  Universities,  research  centres,  service  providers  or  any 
other  interested third  party will be able  to access the results of 
the measurement on the Web portal. They will be able to carry 
out  additional  elaborations,  or  use  the  Indicators  Toolkit  to 
indicators  measurement 
emulate 
process, enriching the exchange of knowledge and information.   

the  data  collection  and 

  National, 

regional, 

local  governments: 

the  different 
government tiers will be able to apply the Indicator toolkit, which 
will hopefully be available on the website after 2014 and check 
it.  Their  feedback  will  be  fundamental  in  order  to  fine  tune  the 
indicators  system  taking  into  account  the  local  needs  and 
issues, where necessary. 

Figure 6 - Governance and structure of the Measurement System 

Source: IDC 2013 

76 

Panel of CAUniversities Research CentresAll interested parties....European CommissionBenchmarking OrganizationPanel of platformsWeb PortalCENMSMSMSMSMSMSMSMSMSContract Award NoticesMember States 
 
4.4. Roadmap of the Measurement System  

The  suggested  roadmap  of  the  Measurement  system  is  articulated  in 
the following way: 

  A measurement cycle (indicatively from the fall of  2013 to the 

fall of 2014) in which: 

o 

o 

 DG  MARKT  and  the  Benchmarking  Organization  will 
work  with  CEN  to  develop  the  standardized  contract 
award notice;  

In  parallel,  the  Benchmarking  organization  will  set-up 
and  organize  the  complementary  data  collection  with 
the  Panels  and  produce  the  “Landscaping”  and  “State 
of Play” reports. 

  The  roll-out  of  the  Measurement  System  (indicatively  from 
2015) when the data collection will be fully implemented through 
the TED forms. This leads to two possible scenarios: 

o  DG  MARKT  will  become  directly  responsible  for  the 
elaboration  of  indicators  and  production  of  “state  of 
play”  reports,  based  on  TED  data,  as  well  as  the 
publication of results on the web and the interaction with 
platforms  and  Contracting  Authorities.  In  this  scenario 
the  Benchmarking  Organization  could 
the  role  of 
disappear or be greatly reduced.  

o  DG MARKT may decide to continue to use a contractor 
to  manage  the  Measurement  system,  design  and  test 
new  indicators,  run  the  web  portal,  and  manage  the 
interaction  with 
the  stakeholders,  continuing  also 
complementary  data  collection  activities  with  Panels  of 
platforms  and  Contracting  Authorities.  In  this  scenario 
the  role  of  the  Benchmarking  Organization  will  remain 
central for the Measurement system.  

4.5. Roadmap of Indicators 

4.5.1. Plan for the 1st year 

Based  on  the  1st  year  plan  and  its  methodological  approach,  the 
following  Table  40  presents  the  list  of  indicators,  which  can  be 
measured. This list is prepared in view of the consideration that the data 
collection based on contract award notices will not be implemented yet, 
so the main sources will be the Panels and the MS interviews.  

This means  that  most  of  the  barriers  identified  in  the  current  study  will 
still be present.  

Therefore,  the  indicators  measurable  in  the  next  year  plan  are  the 
following: 

  The EU top level take-up indicator in  value are measureable 
with  the  same  estimate  method  as  this  year,  hopefully  with 
better evidence from the MS.  

77 

 
  The  EU  top  level  take-up  indicator  in  volume  (number  of 
contracts)  will  be  measureable  if  there  will  be  more  complete 
data from the Panels and from TED.   

  The  EU  take-up  indicator  by  SMEs  in  value  and  volume 
cannot be measured because of the lack of sufficient data; 

  The EU take-up indicator by cross-border suppliers in value 
and  volume  cannot  be  measured  because  of  the  lack  of 
sufficient data; 

  The EU take-up indicators by type of purchase (% supplies, 
services,  public  goods  will  be  measureable  if  there  will  be 
more complete data from the Panels, which is possible; 

  The  EU  take-up  indicators  in  value  and  volume  above  / 
below  the  EU  threshold  will  be  measureable  if  there  will  be 
more  complete  data  from  the  Panels  and  TED,  which  is 
possible; 

  The EU take-up indicator of e-Auctions in value and volume 
will be measureable if there will be more complete data from the 
Panels and TED, which is possible; 

  The EU take-up indicator of e-Ordering cannot be measured 
in  the  next  year  because  it  is  focused  on  post-award  and  the 
Panels are not a sufficient data source for calculation;  

  The  Performance 

indicators  on  Ease  of  Access  and 
Transparency  are  measureable,  based  on  the  data  collected 
from  the  Panels  and/or  a  web  survey  as  needed.  The  data 
needed  for  these  indicators  in  fact  is  published  on  the  web  by 
the main platforms.  

  The  Performance 

Indicators  on  Reduction  of  Prices, 
e-Tendering 
Efficiency 
Litigation, 
Reduction 
improvement are impossible to measure with this methodology, 
even though their feasibility will be explored further.    

of 

78 

 
 
Table 40 - List of e-Procurement Indicators to be measured – 1st Year Plan 

EU Top level Take-up indicators 

2013-2014  

Source of data 
collection 

Measurement 
method  

E-Procurement take-up in value (% 
of total public procurement)  

 Measureable 

 Panel + MS 
interviews 

Estimate 

E-Procurement take-up in volume (% 
of total number of contracts) 

Possibly 
measurable 

Panel + MS 
interviews + TED 

Estimate if 
data sufficient 

Other EU Take-up indicators 

2013-2014  

E-Procurement take-up by SMEs in 
value and volume  

 Not 
measurable 

E-Procurement take-up by cross-
border economic operators (in value 
and volume  

E-Procurement take-up in value and 
volume by type of purchase (% 
supplies, services, public works)  

 Not 
measurable 

Possibly 
measurable 

 Panel + MS 
interviews 

Estimate if 
data sufficient 

E-Procurement take-up in value and 
volume above/below the EU 
threshold  

Possibly 
measurable 

 Panel + MS 
interviews + TED 

Estimate if 
data sufficient 

E-Auction take-up in value and 
volume  

Possibly 
measurable 

 Panel + MS 
interviews 

Estimate if 
data sufficient 

E--Ordering take-up in value and 
volume  

 Not 
measurable 

EU Top level Performance 
indicators 

2013-2014  

E-Submission Ease of access 
indicator  

 Measureable 

E-Submission Transparency 
Indicator for buyers and economic 
operators  

 Measureable 

E-Submission price reduction 
indicator 

To be revised 

E-Submission Reduction of Litigation 
indicator 

To be revised 

E-Tendering Efficiency Improvement 
Indicator for EO and CA 

To be revised  

Source: IDC 2013 

 Panel + MS 
interviews or web 
survey 

 Panel + MS 
interviews or web 
survey 

Elaboration of 
collected data 

Elaboration of 
collected data 

4.5.2. Plan for the Roll-out of the Measurement system 

The  roll-out  of  the  Measurement  system  is  planned  when  the  new 
contract award notices will be implemented, enabling data collection for 
the  indicators  from  the  TED  forms.  Therefore,  the  list  of  measureable 
indicators changes as shown in the following Table 41: 

  All the take-up indicators become measureable, assuming of 
course that the reliability and completeness of the data collected 
through TED forms is acceptable (it may take some time before 

79 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
all the CA become used to the new forms and fill in all the fields 
in the appropriate way). 

  The  Performance 

indicators  on  Ease  of  Access  and 
Transparency remain measureable, but they will require a web 
survey  and/or  data  collection  from  the  Panels  (the  data  cannot 
be collected from TED forms). 

  The  Performance 

Indicators  on  Reduction  of  Prices, 
e-Tendering 
Efficiency 
Litigation, 
Reduction 
improvement  may  change  and  become  measureable, 
depending on the results of their revision.   

of 

Table 41 List of e-Procurement Indicators to be measured – Roll-out Plan 

EU Top level Take-up indicators 

2013-2014  

Source of data 
collection 

Measurement 
method  

E-Procurement take-up in value (% of total 
public procurement)  

 Measureable 

TED forms 

E-Procurement take-up in volume (% of total 
number of contracts) 

Measureable 

TED Forms 

Other EU Take-up indicators 

2013-2014  

E-Procurement take-up by SMEs in value and 
volume  

 Measureable 

TED forms 

E-Procurement take-up by cross-border 
economic operators (in value and volume  

 Measureable 

TED Forms 

E-Procurement take-up in value and volume by 
type of purchase (% supplies, services, public 
works)  

 Measureable 

TED Forms 

E-Procurement take-up in value and volume 
above/below the EU threshold  

 Measureable 

TED Forms 

E-Auction take-up in value and volume (total 
value and number of contracts) 

 Measureable 

TED Forms 

E--Ordering take-up in value and volume (total 
value and number of orders) 

 Measureable 

TED Forms 

EU Top level Performance indicators 

2013-2014  

 Panel + MS 
interviews or 
web survey 

 Panel + MS 
interviews or  
web survey 

E-Submission Ease of access indicator  

 Measureable 

E-Submission Transparency Indicator for 
buyers and economic operators  

 Measureable 

E-Submission price reduction indicator 

To be revised 

E-Submission Reduction of Litigation indicator 

To be revised 

E-Tendering Efficiency Improvement Indicator 
for EO and CA 

To be revised 

Elaboration on 
TED data 

Elaboration on 
TED data 

Elaboration on 
TED data 

Elaboration on 
TED data 

Elaboration on 
TED data 

Elaboration on 
TED data 

Elaboration on 
TED data 

Elaboration on 
TED data 

Elaboration of 
collected data 

Elaboration of 
collected data 

80 

 
 
 
 
 
 
 
 
 
 
 
4.6. Implementation 

Plan 

for 

the 

next 

Measurement Cycle 

4.6.1. Overview and main activities 

The next measurement cycle will indicatively run from September 2013 
to  August  2014.  It  will  be  based  on  a  revision  of  the  workplan  of  the 
contract  extension  designed  in  the  Technical  Annex  to  this  study 
contract. The main goals will be to carry out a new measurement cycle 
of  the  selected  indicators  and  contribute  to  the  development  of 
standardized  contract  forms  for  data  collection  in  view  of  the  eventual 
rollout of the system, as anticipated above.  

The  implementation  will  include  the  following  activities,  divided  in  two 
main workstreams: 

Coordination and support to standardization:  

  To  draw  an  updated  project  plan,  optimizing  timing  and 
tasks,  taking  into  account  the  lessons  learned  from  the 
current project (D0);  

  To collaborate with CEN and the EC for the development of 
updated  standardized  contract  award  notices  including  the 
data  needed  for  the  measurement  of  take-up  indicators. 
This  may  include  preparing  input  material,  participating  to 
meetings,  reviewing  draft  versions  and  providing  feedback, 
depending on the EC and CEN’s requests.  

  To  produce  an  E-Procurement  Performance  Indicators 
Handbook  (presenting  the  updated  list  of  indicators  and 
their data collection needs) to feed into the standardization 
process.  

  To  launch  and  manage  a  web  portal,  acting  as  a  key 
reference  point  of  the  Measurement  system,  providing 
access to all the data collected, publishing the results of the 
indicators  measurement,  supporting  communication  and 
promotion initiatives of the Measurement system. 

  To produce an Indicators Toolkit to be published on the web 
portal, this will present the definitions of the main indicators 
and their calculation methods to share  with  interested third 
parties.  

  To produce a final report at the end of this cycle presenting 
the results of this process, of the activities of the web portal, 
and the next steps.  

Data Collection and Measurement of Indicators 

representative  stakeholders 

  To carry out updated desk research and selected interviews 
to 
(policy  makers,  public 
authorities,  CPBs,  main  vendors)  in  order  to  update  the 
landscape of e-Procurement development and the estimate 
of the state of play of take-up;  

  To  produce  an  updated  e-Procurement  Landscape  report 
(D1).  This  report  will  present  a  short  update  of  the  policy 
and  regulatory  context  in  the  EU  28,  based  on  the  MS 

81 

 
profiles published  in  2012  and based  on the data collected 
in the D1 delivered in 2012 (adding Croatia).   

  To  contact  and  recruit  a  Panel  of  e-Procurement  platforms 
willing to collaborate in data collection, focusing on the most 
important national  hubs and CPBs, starting  with those who 
participated to this year’s trial;   

  To  identify,  contact  and  recruit  a  group  of  Contracting 
Authorities willing to join the panel for data collection and to 
collaborate in the solution of indicators’ feasibility problems, 
particularly for performance indicators; 

  To  design  and  implement  ways  to  incentivise  and  reward 
the  collaboration  of  platforms  and  Contracting  Authorities, 
for  example  providing  personalised  “dashboards”  and 
statistical reports based on the data collected; 

  To  carry  out  a  new  round  of  data  collection  with  the 
platforms and CA panels, including the following tasks: 

o  Setting  up  and  managing  a  data  collection 

tool, 
sufficiently  flexible  and  interactive,  building  on  the 
lessons learned in this year’s trial;  

o  Preparation  of  updated  guidelines  and  communication 
material for the data collection, as well as personalised 
statistical  reports  to  deliver  value  added  back  to  the 
participants; 

o  Organization  of  personalized  briefings  and  instructions 
to  the  Panel  participants  before  the  data  collection,  as 
well as on field technical/ managerial support during the 
process; 

o  To  carry  out  quality  control  of  the  data,  elaboration  of 

results and validation with the Panel participants; 

  To elaborate  the results  of data collection from the  Panels, 
from desk research and from all public sources, in order to 
measure the indicators and to update the estimate of take-
up of e-Procurement in Europe.  

  To  carry  out  a  round  of  validation  of  the  take-up  estimates 

with the national MS representatives; 

  To produce the updated e-Procurement State of Play report 
(D2).  This  report  will  present  a  summary  of  the  level  of 
development of e-Procurement in the EU (including take-up 
data);  the  results  of  the  measurement  of  the  selected 
indicators;  a  brief  qualitative  profile  of  the  state  of  play  for 
each  MS;  a  review  of  main  trends  of  development  and  of 
the challenges to be met for the successful development of 
e-Procurement. This report will have an Executive summary 
designed  for  dissemination  to  the  non-specialist  public  and 
high level policy makers.  

The indicative timing for this workplan is presented below (Table 42).  

The  coordination  and  support  to  standardization  workstream  will  have 
the following timing:  

82 

 
  Start  off  indicatively  in  September  2013  with  the  new  project 
plan  (D0)  and  a  draft  Indicators  Handbook  to  start  the 
standardization  process  (Draft  D3).  This  draft  will  provide  the 
information needed by CEN in the format they find most useful; 

  The  collaboration  with  CEN  will  start  immediately  and  continue 
for the duration of the  project, depending on  what  is requested 
by the EC and CEN itself; 

  The 

Indicators  Handbook  will  be  updated  and 
indicatively by June 2013, together with the Indicators toolkit; 

finalised 

  The  web  portal  should  be  launched  very  soon  in  the  project 
(indicatively  in  the  second  month)  an  gradually  developed  and 
enriched; 

The  workstream  on  data  collection  and  measurement  of  indicators  will 
have the following timing:  

  Start off indicatively in September 2013 with the desk research, 
leading  to  the  production  of  the  Landscape  report  (D1)  in 
January 2014; 

  The  process  of  recruiting,  communication  and  interaction  with 
the  platforms  and 
the  Contracting  authorities  will  start 
indicatively  in  the  second  month  and  continue  until  May  2013. 
This will include promotion activities for the web portal. 

  The guidelines for data collection will be prepared in the second 

month, indicatively October 2013.  

  The  implementation  of  data  collection  will  start  in  November 
2013  and  cover  indicatively  until  February  2014,  so  that  data 
about the year 2013 can be collected;  

  The  elaboration  of  results,  measurement  of  indicators  and 

calculation of estimates is planned for March-April 2014; 

  The validation of national estimates with MS is planned for April-

May 2014; 

  The production of the “State of Play report” (D2) is planned for 

June 2014. 

83 

 
 
Table 42 – Indicative workplan timing for next measurement cycle  

Source: IDC 2013 

4.6.2. Coverage of the EU 28 

team  will  strive 

the 
The  study 
representativeness of the EU MS in the panels, in order to improve the 
quality of data collection and the scalability of results. To do so the study 
team will strive to include in the panels: 

to  maintain  and 

improve 

  Representatives of the leading public platforms and CPBs from 
the  largest  MS  (UK,  Germany,  France,  Italy,  Spain,  Poland), 
given  their  relevance  for  the  dynamics  of  e-Procurement 
development.  We  hope  that  this  year’s  panel  participants  will 
agree to participate again.  

  Representatives  of 

the  small  MS  with  a  centralised  e-
Procurement  platform,  who  have  a  high  visibility  of  e-
Procurement  flows  and  proved  very  collaborative  This  list 
includes Estonia, Malta, Cyprus, Portugal, Lithuania, and could 
extend to Luxemburg who just started offering e-Submission in 
2012.  

  Representatives  of  mid-sized  countries  with  high  take-up,  this 

could include Sweden, Denmark and the Netherlands. 

84 

INDICATIVE WORKPLAN TIMINGIndicative calendar SepOctNovDecJanFebMarAprMayJunJulAugMain Tasks123456789101112Coordination and support to standardizationProduce Project PlanD0Produce Indicators Handbook and ToolkitDraft D3 D3Collaboration with CEN  Set-up, launch and management of web portal (WP)WPD4Data collection and measurement of indicatorsDesk research and selected interviews in 28 MSProduction of Landscape reportD1Organization and cooperation with e-procurement platforms panelOrganization and cooperation with Contracting Authorities Panel Preparation of guidelines for data collectionImplementation of data collection surveys with PanelsElaboration of results and estimates of take-upValidation with MSProduction of State of Play report D220132014 
 
This  does  not  mean  that  we  will  exclude  representatives  from  other 
countries  interested  to  participate  (one  example  for  all,  Austrian 
auftrag.at),  simply  that  this  list  represents  a  target  able  to  insure  the 
representativeness  of 
is 
representative  of  the  leading  EU  countries  in  terms  of  e-Procurement 
adoption  and  well  balanced  by  size  and  geographical  distribution 
(North-South, East-West).  

results.  This  selection 

the  Panels 

Concerning  the  6  largest  MS,  we  have  been  able  to  establish  fruitful 
contacts with all their national public platforms, with the exception of the 
UK.  Concerning  the  UK,  we  have  established  fruitful  contacts  with 
Bravosolution, the leading provider of e-Procurement services, and with 
the regional CPB and platform of Scotland. However, It will probably be 
necessary to plan for ad-hoc research with the help of national experts 
at  least  in  Germany  and  France,  where  there  is  a  need  of  further 
information  about  take-up  by  regional/local  administrations,  particularly 
for contracts below the EU thresholds.  

4.6.3. Approach to Contracting Authorities 

As  anticipated,  much  of  the  data  needed  for  our  key  performance 
indicators  should  be  sourced  from  Contracting  Authorities  (CA),  rather 
than service provider platforms only.  

This  creates  considerable  data  collection  problems,  but  also  problems 
for the design of a representative sample  whose results can be scaled 
up  at  the  EU  level.  The  number  of  Contracting  Authorities  is  much 
higher  than  that  of  e-Procurement  platforms  (up  to  the  hundreds  of 
thousands,  instead  of  a  few  hundreds,  as  is  the  case  for  platforms). 
Their  typology  varies  considerably  from  MS  to  MS,  so  that  Eurostat 
does not even provide reliable universe data.  

Therefore,  for  the  next  measurement  cycle  it  is  suggested  to  explore 
further the ways to collect data from Contracting Authorities.  

The  first  step  will  be  to  identify  and  recruit  a  group  of  collaborative 
Contracting  Authorities,  which  should  include  at  least  some  CPB 
(Central  Purchasing  Bodies),  to  help  in  the  assessment  of  the  data 
collection  and  measurement  method,  and  carry  out  a  pilot  data 
collection exercise.  

The panel should be asked to join in the web survey of basic indicators, 
as done with the e-Procurement platforms. A separate questionnaire will 
be prepared to take into account their different profile.  

Two alternative methods will be evaluated as well: 

  The use of Spend analysis software packages  

These  are  software  packages  from  the  Business  Intelligence  category, 
which are used to collect data on spending based on the ERP systems 
of Contracting Authorities. These packages collect data from orders and 
invoices  records.  For  example,  GPS  in  the  British  government  uses  a 
Spend  analysis  package  to  collect  and  monitor  data  from  180 
authorities.  The  software  collects  granular  data  and  re-assembles  it 
according to a series of categories and indicators decided by GPS with 
Bravosolution, the author of the package. It produces  a "dashboard"  of 
spending  categories  and  volumes,  which  is  fully  transparent  for  the 

85 

 
British government, because it allows drilling down into specific items of 
spending and/or authorities budgets.  

These  software  packages  could  be  adapted  to  collect  data  from  the 
tendering  phase  as  well.  If  the  EC  promoted  the  development  of  a 
standard EU Spend analysis package this could lead to the collection of 
comparable  and  measurable  data,  possibly  finding  ways  to  guarantee 
confidentiality to the CA (for example sharing only aggregate data).  

A strong advantage  would  be that the  package  would collect data  both 
from  pre-award  (tendering)  and  post-award  phases  (ordering  and 
invoicing)  thereby  allowing  to  measure  exactly  the  level  of  take-up  of 
pre-award  e-Procurement  on  total  spending  by  each  Contracting 
Authority. The  feasibility  and  value  added  of  this  approach  ought  to  be 
evaluated carefully.  

  An EU-wide survey of Contracting Authorities 

A  completely  different  option  would  be  for  the  EC  to  carry  out  a 
telephone  survey  of  a 
representative  sample  of  government 
organizations  across  the  EU,  with  the  goal  to  measure  the  level  of 
diffusion  of  e-Procurement  and 
the  average  value  of  contracts 
transacted annually with e-Procurement, by each CA, as a proportion of 
their  overall  spending.  The  survey  sample  could  be  segmented  by 
government  tier  (national,  regional,  local),  by  administration's  size,  and 
include national and size quotas. Concerning the estimate of take-up in 
value, such a survey would enable building a quantitative model for the 
EU,  based  on  the  extrapolation  of  average  value  of  take-up  by  type  of 
authority  to  the  universe  of  authorities.  It  could  also  help  to  measure 
opinions, drivers and barriers about the adoption of e-Procurement.  

Since  the  universe  of  public  organization  is  large,  this  type  of  surveys 
even  with  national  quotas  would  be  able  to  produce  a  representative 
sample  of  completed  questionnaires.  This  approach  is  reliable  for  the 
calculation  of  diffusion  and  general  take-up  of  e-Procurement.  It  would 
probably be unable to measure detailed indicators, such as SME wins or 
take-up above or below the threshold, or by type of purchase.  

The  main  problem  of  this  approach  is  the  cost  of  such  a  large  survey, 
however  this  could  be  done  every  few  years  (not  every  year)  and  it 
would  represent  a  very  valuable  addition  to  the  methodologies  and 
inputs to the Measurement system.  

The  pros  and  cons  of  these  methods  could  be  better  analysed  and 
discussed  with  stakeholders  in  the  course  of  the  next  measurement 
cycle.  

4.7.  Conclusions 

The high-level implementation plan designed above is a pragmatic plan, 
taking into account the challenges related with the launch of a EU-wide 
process  of  benchmarking  and  difficulties  of  data  collection.  The  choice 
to  leverage  the  mandatory  data  collection  process  of  TED  contract 
notices,  in  order  to  feed  them  into  the  benchmarking  process,  helps  to 
overcome many of these challenges.  

Nevertheless, to be successful this plan needs to take into account the 
other main measurement challenges identified in the current study, such 
as: 

86 

 
  The need to build consensus around the selected indicators, so 
that  the  data  collection  process  through  the  TED  forms  is 
accepted willingly, and not perceived as an additional burden; 

  The need to involve all the main categories  of stakeholders, at 
national and regional level, and to motivate them to collaborate.  
A way to do so is by giving back to them information and value 
added  from  the  results  of  the  EU  Measurement  system.  From 
this  point  of  view,  the  creation  of  a  Web  platform  acting  as  a 
repository and a hub is critical, given also the positive results of 
feedback received by the Golden Book website.  

  The need to continue working on the revision and testing of the 
e-Procurement  Performance  indicators  with  proven  feasibility 
problems,  which  nevertheless  address  important  measurement 
goals,  such  as  the  estimate  of  efficiency  and  effectiveness 
benefits. The TED forms approach does not appear well suited 
to  the  development  of  these  indicators,  so  alternative  solutions 
should be found.   

The suggested plan deals with these additional challenges by designing 
a  mechanism  involving  stakeholders  and  carrying  out  complementary 
data  collection,  through  the  platforms  and  Contracting  Authorities 
panels,  and  the  launch  of  the  web  portal.  The  mix  of  top-down  and 
bottom-up actions insures the flexibility and reliability of the plan.  

However,  this  report  has  strongly  underlined  that  these  are  not  simply 
practical and organizational challenges, but also policy challenges. The 
main goal of this plan is not measurement for measurement's sake, but 
the  assessment  of  progress  towards  relevant  policy  priorities,  those 
targeted by the new Public Procurement Directives, so that the MS will 
achieve fully the benefits of e-Procurement adoption.  

Therefore,  the  successful  implementation  of  an  EU  monitoring  and 
measurement  plan  would  require  also  dealing  with  larger  policy 
challenges, such as: 

  Promoting  the  adoption  by  MS  of  full  transparency  and  open 
data  policies applied to  public procurement and e-Procurement 
processes, for all government tiers; 

 

Increasing the awareness of the benefits of the monitoring and 
measurement  process  of  the  performance  of  e-Procurement, 
through communication and motivation initiatives, underlining its 
value added for the governance of public spending flows.   

87 

 
A N N E X    

Definitions of Procurement Concepts 

Availability of the platforms – definitions 

Electronic 
procedures 

Electronic  procedures  refer  to  the  electronic  execution  of  all  procedures  of  the 
procurement  process  (e-Auction,  Dynamic  Purchasing  System,  and  Electronic 
marketplace) via an online platform. 

e-Auction 

An  electronic  online  process  through  a  platform  for  processing  submitted  bids  of 
invited  economic  operators.  During  the  auction  the  ranking  or  the  lowest  price  are 
displayed to bidding parties. The goal of using this automatic evaluation method is to 
determine the lowest bidder. 

Dynamic 
Purchasing 
System (DPS) 

A completely electronic process for the purchasing of commonly used supplies, works 
and services. A DPS is limited in duration and open throughout its entire time frame to 
any  economic  operator  which  satisfies  the  selection  criteria.  Once  admitted,  the 
supplier  may  improve  his  bid  at  any  time.  Any  new  supplier  can  join  if  meeting  the 
selection requirements. This means that the market place will remain competitive and 
new providers are able to join. 

E-Procurement 
process 

The  e-Procurement  process  refers  to  the  use  of  electronic  communication    and 
transaction- processing when buying Supplies, Services or Public  works. It consists of 
two phases – the pre award phase and the post award phase.  

Pre-award 

E-Procurement  process  phases  occurring  before  the  award  of  the  contract:  e-
Notification, e-Access, e-Submission, e-Evaluation, e-Awarding. 

Post-award 

E-Procurement process phases occurring after the award of the  contract: e-Ordering, 
e-Invoicing, e-Payment.  

Electronic 
Marketplace  

Electronic  Marketplaces  are  services  providing  an  electronic  environment  for  the 
interaction between pre-qualified registered suppliers and Contracting Authorities – the 
prequalification  decision  is  mainly  based  on  the  fulfilment  of  technical  requirements. 
The  Marketplace  (environment)  presents  e-catalogues  of  the  products  and  services 
available and the Contracting Authority can select and e-order items. 

The  electronic  publication  of  standardized  notices  about  a  tender  or  a  link  to  the 
publication via the European public Procurement journal Tenders Electronic Daily TED. 

The  electronic  publication  of  standardized  notices 
documentation enabling suppliers to apply for public procurement opportunities.  

including  all  necessary 

e-Notification 
without 
documentation  

e-Notification 
with 
documentation 
and application 

88 

 
 
e-Access 

The publication of all necessary documents concerning procurement on the web. This 
relates  only  to  general  documents,  and  not  to  documents  that  a  specific  economic 
operator may require. 

Submission  of  components  through  an  interactive  process  between  the  participating 
supplier  and  the  e-Procurement  platform.  The  platform  may  provide  guidance  on  the 
preparation  and  uploading  of  the  tender  for  all  its  components.  The  components  are 
required 
technical  documentation, 
certificates, proposals of price, delivery term etc, which is all digitally uploaded and or 
entered into the platform.  We distinguish four levels: 

financial  documentation 

for  due  diligence, 

L1.) send only an attachment submitted through the platform (pdf etc.) 

L2.) bid through data entry into the platform  

L3.) export/import function to excel/access/etc. for offline data entry and re-import into 
the platform 

L4.)  advanced  complete  transaction  through  a  fully  interactive  process  with  e-
Procurement  platform  providing  guidance  on  the  preparation  and  uploading  of  the 
tender for all its components. 

Partial  (i.e.  decision  support)  or  entire  automation  of  the  assessment  of  bids  (e-
Evaluation),  followed  by  the  formalisation  and  communication  of  the  outcome  to  the 
tenders  (e-Awarding).  A  fully  automated  assessment  is  by  definition  only  possible  if 
assessment  criteria  are  entirely  quantitative  (i.e.  it  does  not  require  subjective 
appreciation)  and  clearly  defined.  The  automation  of  the  assessment  refers  to  a 
computerized  scoring  mechanism  in  the  platform,  instantly  assigning  scores  to 
answers provided in the tender. 

The  formalisation  and  communication  of  the  outcome  of  the  tenders  to  the  bidding 
participants  through  an  electronic  process.  E-awarding  may  include  the  electronic  
evaluation and ranking of the bids submitted without human interaction. 

The placement of purchase orders online, including particularly through the use of e-
Catalogues.  e-Ordering  only  occurs  in  cases  where  the  concluded  procurement 
contract  has  established  a  framework  within  which  Supplies  or  Services  can  be 
ordered. This might include the transmission of purchase order responses. 

The  automated  process  of  issuing,  sending,  receiving  and  processing  the  invoice 
through electronic means. This includes the electronic transfer of billing and payment 
information. 

Any  digital  financial  payment  involving  currency  transfer  between  a  Contracting 
Authority and a supplier for a public Procurement transaction. These transfers may be 
carried  out  by  companies  uploading  information  in  specified  formats  to  banks,  either 
through the internet or by transmitting data files directly to banks or their intermediaries 
through secured or closed networks.  

e-Submission 

e-Evaluation 

e-Awarding 

e-Ordering  

e-Invoicing 

e-Payment 

89 

 
Infrastructural 
tools 

The  tools  used  for  communicating  by  electronic  means,  as  well  as  their  technical 
characteristics.  These  must  be  non-discriminatory,  generally  available  and 
interoperable with the information and communication technology products in general 
use. 

e-Certificates/  
e-Attestations 

An e-Certificate or E-attestation is documentary evidence - provided by a third neutral 
party-  in  an  electronic  form  which  is  provided  by  the  economic  operator  .It 
demonstrates compliance with formal requirements. These documents are relevant for 
the economic operator to show its suitability to the Contracting Authority. 

e-Catalogues 

An  electronic  document,  which  allows  the  automatic  processing  of  orders  for  a 
predefined selection of products, using the customer specific prices, which have been 
negotiated in the tender process. 

Framework 
agreement  (and 
Call Offs) 

OLD DEFINITION 

A closed framework agreement between one or more Contracting Authorities and one 
or  more  economic  operators  laying  down  terms,  conditions  and  price  governing  a 
future  contract  to  be  awarded  in  a  given  period.  The  framework  agreement  is  not  a 
contractual agreement but provides agreed specifications, delivery terms, prices, and 
terms and conditions. 

NEW DEFINITION (as a result of the trial) 

initial  agreements  do  not 

The  Directive  2004/18/EC  defines  the  framework  agreement  as  an  agreement 
between one or more Contracting Authorities and one or more economic operators, the 
purpose of which is to establish the terms governing contracts to be awarded during a 
given  period,  in  particular  with  regard  to  price  and,  where  appropriate,  the  quantity 
envisaged.  The 
themselves  create  any  contractual 
obligations,  whereas  call-offs  under  those  arrangements  do.  It  is  therefore  important 
for the agreement to set out the terms and conditions of the contractual relationships 
that  will  arise  from  call-offs.   The  Call  Offs  are  the  legally  binding  contracts  between 
the  institution  and  the  framework  supplier  which  defines  the  goods/services  to  be 
provided. They incorporate a contractual commitment to purchase a particular volume 
or value of goods or services. These contracts are normally formed through the placing 
of a specific purchase order for a quantity of goods or services following the terms set 
out in the Framework Agreement, under which the call off is made. 

e-Tender 

Element  of  the  pre-award  stage  of  the  e-Procurement  process.  An  internet  based 
tendering process including at least e-Submission. 

e-Authentication  The process of establishing confidence in user identities electronically presented to an 
information  system.  Systems  can  use  the  authenticated  identity  to  determine  if  an 
individual is authorized to perform an electronic transaction.  

90 

 
 
 
 
 
Definitions on Take up of e-Procurement 

Take up of e-Procurement - definitions 

Total Number of 
contracts  

Number  of  contracts  awarded  within  the  period  of  twelve  months  in  the  period  from 
January 1st till December 31st. 

Award 

SME 

The  Contracting  Authorities  notice  to  the  bidding  party  of  the  acceptance  of  the 
submitted bid.  

Small  medium  enterprises,  enterprises  which  employ  fewer  than  250  persons  and 
which  have  an  annual  turnover  not  exceeding  50  million  Euro,  and/or  an  annual 
balance sheet total not exceeding 43 million euro. 

Cross-border 
e-Procurement 

The Contracting Authority provides the opportunity and the platform ensures technical 
access for suppliers outside the boundaries of the nation state. The legal requirements 
(such  as  electronic  signature)  also  need  to  be  accessible  for  foreign  participating 
suppliers). 

Supplies 

The term refers to the purchasing of goods and products. 

Services 

The  procurement  of  intangible  items  performed  by  an  economic  operator,  including 
advertising,  property  management,  cleaning,  management  consultancy,  financial  and 
ICT related services. 

Public works 

A  ‘work’  means  the  outcome  of  building  or  civil  engineering  works  taken  as  a  whole, 
which is sufficient of itself to fulfil an economic or technical function. 

Contract 

Written  agreement  between  one  or  more  economic  operators  and  one  or  more 
Contracting Authorities and having as their object the execution of works, the supply of 
products or the provision of services. 

91 

 
 
Total value 

For  the  total  value  in  Euro  (for  non  Euro  member-states:  with  a  fixed  exchange  rate 
based  on  the  Exchange  Rate  Mechanism  (ERM  II))  of  a  contract  the  rules  of  the 
European Union for contract valuation apply: 

Supply  contracts:  If  the  contract  is  for  a  'one-off'  purchase,  estimate  the  total  value 
including  all  component  parts,  transport  costs,  installation  and  commissioning. 
If  a  contract  is  for  a  fixed  period,  say  one  year,  but  has  an  option(s)  to  extend,  for 
example, for a further period of one or two years. Then, regardless of whether or not 
there is uncertainty about exercising the extension options, the maximum duration, ie 3 
years, is used when estimating its value. 

If  the  contract,  or  series  of  contracts,  is  for  an  indefinite  or  uncertain  period,  then  its 
total value should be calculated by multiplying the estimated monthly value by 48. 

Service contracts: If the contract is for a 'one-off' service, estimate the total value of that 
service. The  value  of  a  series  of  contracts,  or  repeat  orders,  for  the  same  or  similar 
'services'  should  be  estimated  over  12  months  (based  either  historically  or  in  the 
future). Fixed  term  contracts  for  services  should  be  estimated  for  the  duration  of  the 
contract.   

period 

Where the contract is part of a series, the whole series should be estimated over a 12-
month 
longer, 
If  a  contract  is  for  a  fixed  period  but  has  an  option(s)  to  extend,  then,  regardless  of 
whether  there  is  uncertainty  about  exercising  the  extension  options,  the  maximum 
duration is used when estimating its value. 

appropriate.  

or 

if 

If  the  contract,  or  series  of  contracts,  is  for  an  indefinite  or  uncertain  period,  then  its 
total value should be calculated by multiplying the estimated monthly value by 48.  

Works contract: Estimate the value of the whole works  contract. Where a single work 
involves  more  than  one  contract,  it  is  the  total  value  of  the  aggregated  contracts  that 
will determine if the contract falls within the scope of the legislation. 

Competitive 
Dialogue 

A  Contracting  Authority  may  make  use  of  the  competitive  dialogue  for  complex 
contracts if it is not able to define by itself the technical solutions to satisfy its needs or 
is  not  able  to  specify  the  legal  and/or  financial  make-up  of  a  project.  Large 
infrastructure projects would seem to lend themselves to this type of dialogue. 

Mini 
competition 

Mini-competition  is  the  term  used  to  describe  one  of  the  processes  for  selecting  a 
supplier on a framework agreement to place a call-off contract with. 

General invitation to tender to an economic operator responding to the profile indicated 
in  the  TOR  (Terms  of  Reference).  The  procedure  can  be  managed  through  the  pre-
award phases of the e-Procurement process. 

In a negotiated procedure, the Contracting Authority consults the economic operator of 
its choice and negotiates the terms of the contract with them. This can be applied with 
or without prior publication of a contract notice. The negotiated procedure can be used 
only in specific circumstances defined by the Directives, for example if previous tenders 
received no answers, or if the price and content of the service to be procured cannot be 
defined beforehand. 

Open 
Procedure 

Negotiated 
Procedure 

92 

 
 
  
Restricted 
Procedure 

Simplified 
Procedure 

Two-phase procedure: open invitation, to which any economic operator may request to 
participate, followed by the selection of a restricted group of economic operators based 
on  a  comparative  evaluation  of  the  tenderers  capability.  This  can  be  designed  as  a 
specific  case  of  e-Evaluation  and  e-Award  with  several  winners  instead  of  only  one 
winner.  The  short-listed  economic  operators  are  invited  to  submit  a  proposal.  The 
procedure  can  be  managed  through  the  pre-award  phases  of  the  e-Procurement 
process. 

In line with the WTO Government Procurement Agreement, the simplified procurement 
regime  applies  to  all  contracting  authorities  below  the  central  government  level,  such 
as local and regional authorities. These purchasers may use a prior information notice 
as a means of calling for competition. If they make use of this faculty, they do not have 
to  publish  a  separate  contract  notice  before  launching  the  procurement  procedure. 
They may also set certain time limits in a more flexible way by mutual agreement with 
participants. It is included in the proposed Directive on Public Procurement. 

Definitions on Performance of the platform 

Performance of the platform - definitions 

Reserve price 

The  price  fixed  and  announced  as  the  maximum  at  which  the  product  or  service  or 
work contract will be awarded. 

Award price 

The price, which the winning party in the tender is receiving for the execution of the 
Works or Services or the Supplies, delivered. The award price includes the scope and 
conditions as described in the tender. 

Appeal 

A  formal  complaint  of  a  participant  of  a  tender  process  about  a  decision  during  the 
execution of the tender.  

Contracting 
Authority 

The  State,  regional  or  local  authorities,  bodies  governed  by  public  law,  associations 
formed  by  one  or  several  of  such  authorities  or  one  or  several  of  such  bodies 
governed by public law - the buying party.  

Economic 
Operator  

The  term  economic  operator  refers  -  depending  on  the  type  of  purchase  -  to  the 
‘contractor’,  ‘supplier’  or‘  service  provider’  meaning  any  natural  or  legal  person  or 
public  entity  or  group  of  such  persons  and/or  bodies  which  offers  a  product  on  the 
market, respectively, the execution of works and/or a work, products or services. 

Purchase order  A  document  generated  by  an  authorized  person  that  internally  in  the  organization 
provides  permission  to  execute  a  purchase  transaction.  It  contains  the  desired 
quantity and the contracted price for products or services. 

Purchase order 
response 

A document sent by the supplier to the buyer (who has previously sent him a purchase 
order) to accept the order fully or partially or to reject the order 

93 

 
 
 
Registration  in 
the platform  

The  registration  includes  the  creation  of  a  user  account  with  all  relevant  company 
information. It includes all necessary levels of admission to participate in e-Tenders.  

Implementation 
of an 
 e-Tendering 
process 

Preparation 
and sending an 
electronic 
tender 

 The process of preparing all legal, technical and tender related conditions for making 
use  of  the  e-Tender  system.  This  includes  the  internal  training  of  staff,  the 
digitalization of all documents or their transfer into the right format. 

 The  preparation  of  all  tender  related  documentation  into  the  right  format  and  the 
upload  into  the  platform  measured  from  the  point  of  publication  of  the  tender  until 
digital submission of the bid. 

Definitions added during the Trial 

e-Signature 

The  electronic  equivalent  of  a  handwritten  signature.  E-Signatures  are  defined  in 
Directive 1999/93/EC (the e-Signatures Directive) as data in electronic form which are 
attached  to  or  logically  associated  with  other  electronic  data  and  which  serve  as  a 
method of authentication. 

National ID

Identification password provided by a government organization. 

Simplified e-
ID 

Qualified 
Digital 
Signature 

Simple username/password schemes based on online registration via web forms. 

A  digital  signature  based  on  a  qualified  certificate,  i.e.  a  certificate  issued  by  a 
Certificate Authority that has a national accreditation for providing those. 

94 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
95 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
K
M
-
0
1
-
1
3
-
0
4
4
-
E
N
N

-

DOI : 10.2780/94897 

ISBN : 978-92-79-29926-1