COMMISSION STAFF WORKING DOCUMENT Monitoring programme for Regulation (EU) 2021/784 on addressing the dissemination of terrorist content online Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL on the implementation of Regulation (EU) 2021/784 on addressing the dissemination of terrorist content online
Date: 2024-02-14
Institutions: Directorate-General for Migration and Home Affairs, European Commission
Subjects: Internet, application of EU law, cross-border cooperation, digital content, digital single market, dissemination of information, fight against crime, freedom of expression, provision of services, terrorism

EN 

  EN 

   EUROPEAN COMMISSION  Brussels, 14.2.2024  SWD(2024) 36 final  COMMISSION STAFF WORKING DOCUMENT Monitoring programme for Regulation (EU) 2021/784 on addressing the dissemination of terrorist content online  Accompanying the document REPORT FROM THE COMMISSION TO THE EUROPEAN PARLIAMENT AND THE COUNCIL on the implementation of Regulation (EU) 2021/784 on addressing the dissemination of terrorist content online   {COM(2024) 64 final}  
 
 
1.  Context 

In line with the inter-institutional Agreement on Better Law-Making1, Union legislation should 
be  comprehensible  and  clear,  allow  citizens,  administrations  and  businesses  to  easily 
understand  their  rights  and  obligations,  include  appropriate  reporting,  monitoring  and 
evaluation  requirements,  avoid  overregulation  and  administrative  burdens,  in  particular  on 
Member States. 

Monitoring is a continuous and systematic process of data collection about an intervention. It 
generates factual information for future evaluations and helps identify actual problems in the 
application of legislation. Monitoring is necessary to allow policy makers and stakeholders to 
check if policy implementation is ‘on track’ and to generate information that can be used to 
evaluate whether it has achieved its objectives. While monitoring looks at “what” changes have 
occurred since the entry into force of a policy intervention, evaluation looks at "whether" the 
intervention has been effective in reaching its objectives, and whether the objectives have been 
met  efficiently  (i.e.  at  least  cost),  as  well  as  the  reasons  for  the  success  or  otherwise  of  an 
intervention. 

In  that  context  and  pursuant  to  Article  21(2)  of  Regulation  (EU)  2021/784  of  the  European 
Parliament and of the Council of 29 April 2021 on addressing the dissemination of terrorist 
content online, hereafter ‘the Regulation’, which entered into force on 7 June 2021 and applied 
from  7 June 2022,  the Commission  shall establish a detailed programme  for monitoring the 
outputs, results and impact of the Regulation. 

2.  Objective of the Regulation 

The Regulation pursues a general objective, namely to establish uniform rules to prevent the 
misuse of hosting services (HSPs) for the dissemination to the public of terrorist content online 
in order to guarantee the smooth functioning of the digital single market (Article 1(1) and recital 
50 of the Regulation). It also seeks to contribute to public security, reinforcing legal certainty 
for  HSPs  and  users’  trust  in  the  online  environment  and  strengthening  fundamental  rights 
safeguards (recital 1 of the Regulation).  

3.  Aim and purpose of the monitoring programme 

The  monitoring  programme  established  by  the  Commission  foresees  a  continuous  and 
systematic process of data collection on the outputs, results and impact of the Regulation. It 
aims to help generate information for the future evaluation of the Regulation pursuant to Article 
23 of the Regulation and to help identify possible problems in its application. It will allow to 
check if the implementation of the Regulation is ‘on track’ and to generate information that can 
be used to evaluate whether it has achieved its objectives.  

In  accordance  with  Article  21(2)  of  the  Regulation,  the  monitoring  programme  sets  out  the 
means by which, and at which intervals, data and other necessary evidence are to be collected 
to monitor the outputs, results and impacts of the Regulation and specifies the actions to be 
taken by the Commission and by the Member States in collecting, sharing and analysing those 

1 OJ 2016 L 123/1 of 12.5.2016. 

1 

 
 
 
 
 
 
 
 
 
 
data  and  other  evidence.  To  this  end,  it  recalls  the  objective  of  the  Regulation,  monitoring 
indicators as well as sources of data and collection methods to measure the achievement of that 
objective and the various elements that the objective consists of. It devises a strategy to collect 
reliable data and evidence on the indicators at regular intervals.  

Subject to monitoring are the implementation of the Regulation, its objective, and effects. This 
monitoring programme should enable the Commission to assess whether and to what extent the 
outputs, results and impacts of the Regulation correspond to its objectives.  

The  monitoring  programme,  whilst  not  legally  binding,  complements  obligations  of  HSPs, 
national  competent  authorities  and  Member  States  under  Articles  7,  8  and  21(1)  of  the 
Regulation,  respectively,  to  make  annual  transparency  reports  on  their  activities  under  the 
Regulation publicly available and to collect and send to the Commission information about the 
actions taken under the Regulation.  

Data gathered and analysed under this monitoring programme will feed into the evaluation of 
the Regulation. Pursuant to Article 23 of the Regulation,  the Commission shall carry out an 
evaluation of the Regulation and submit a report to the European Parliament and to the Council 
on its application including: 
(a)  the functioning and effectiveness of the safeguard mechanisms, in particular those provided 

for in Article 4(4), Article 6(3) and Articles 7 to 11; 

(b) the  impact  of  the  application  of  this  Regulation  on  fundamental  rights,  in  particular  the 
freedom  of expression and information,  the respect  for private life  and the protection of 
personal data; and 

(c)  the contribution of this Regulation to the protection of public security. 

4.  Indicators  

An indicator is a quantitative or qualitative measure of how close one is to achieving a set goal 
(e.g. policy outcome). They help to analyse and compare performance and can be useful for 
determining policy priorities. Indicators should only give one perspective of the performance 
of a policy intervention, which is highly dependent on the type of indicator selected, data, other 
influences, etc. It is important, therefore, to use other complementary approaches to monitoring, 
such as qualitative analysis or surveys.  

Pursuant  to  Article  21(2)  of  the  Regulation,  the  Commission  shall  establish  a  detailed 
programme for monitoring the outputs, results and impacts of this Regulation. Therefore, 
three types of indicators corresponding to this task were developed accordingly, as follows: 

•  Outputs indicators: indicators that measure the immediate activities  stemming from 
the implementation of the Regulation. Most of those indicators reflect the obligations 
from  the  Regulation,  including  the  data  that  must  be  reported  by  the  relevant 
stakeholders,  i.e.,  HSPs,  Member  States  and  national  competent  authorities  (NCAs). 
Those indicators are therefore aligned with the obligations of the Regulation, and more 
specifically with the ones presented in Articles 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 18 and 21. 
•  Results indicators: these indicators are not currently required to be collected under the 
Regulation  and  will  be  collected  by  other  means,  such  as  stakeholder  consultations. 
Results indicators result from the application of the Regulation and seek to measure the 
functioning  and  effectiveness  of  the  safeguard  mechanisms  once  they  have  been 
implemented. 

2 

 
 
 
 
 
 
 
 
•  Impacts indicators: the impacts indicators help measure the long-term impact of the 
Regulation  on  specific  stakeholders  (e.g.,  HSPs)  or  areas  (e.g.,  fundamental  rights, 
overall trust in online information and impact on safety and security).  

The indicators include the following: 

-  Preservation of terrorist content online (TCO) by the HSP for a period of six months 
after the removal or disabling (duration, criteria for when the data should be preserved, 
implementation of appropriate technical and organisational safeguards) (Article 6) 
Information about the removal or disabling is made accessible to the content provider 
by the HSP when TCO is removed (Article 11(1)) 

- 

-  The number of complaints received by the HSP from content providers (Article 21(1)(d) 

and Article 7(3)(d)) 

-  Establishment  of  clear  rules  by  each  Member  State  on  penalties  applicable  to 
infringements of the Regulation by HSPs and implementation of measures necessary to 
ensure that they are implemented (Article 18(3)) 

-  The  number  of  cases  in  which  the  HSP  was  required  to  reinstate  content  or  access 
thereto as a result of administrative or judicial review proceedings (Article 7(3)(f)) 
-  Number of removal orders that were effectively proceeded within one hour of receipt of 

the removal order by the HSP (Article 3(3)) 

The full list of indicators is presented in Section  7. For each indicator, the type of indicator 
(impact, output or result) is indicated, also mentioning the level of reporting obligation, the type 
and source of data.  

5.  Data collection 

In listing this set of data for collection (Section 7) and by whom, account has been taken of the 
need  to  minimize  administrative  burden  and  costs  on  all  involved  actors,  whilst  ensuring 
meeting the requirements of Article 21(2) of the Regulation. The programme therefore aims to 
collects  only  what  is  absolutely  necessary  in  order  to  allow  for  a  proper  monitoring  and 
evaluation of the Regulation, and makes maximum use of existing data to save time and increase 
coherence of results. 

To avoid putting any additional administrative burden on Member States or the private sector, 
the indicators mainly  rely  on the existing  data sources. Additionally,  it is presumed that  the 
Commission  and  Member  States  collect,  share  and  analyse  the  data  and  other  evidence  as 
required under the Regulation. 

5.1 Existing data sources 

5.1.1  HSPs’ annual transparency reports under Article 7(2) and (3) of the Regulation  

Pursuant  to  Article  7(2)  of  the  Regulation,  a  HSP  that  has  taken  action  to  address  the 
dissemination of terrorist content or has been required to take action pursuant to the Regulation 
in a given calendar year, shall make publicly available a transparency report on those actions 
for that year. This report shall be published before 1 March of the following year.  

3 

 
 
 
 
 
 
 
 
 
 
Article 7(3) refers to the minimum information that these transparency reports shall include, 
such as measures taken to identify and disable access to terrorist content, the number of items 
removed and/or reinstated and the number and outcome of complaints. 

5.1.2  National competent authorities’ annual transparency reports under Article 8 

of the Regulation  

According  to  Article  8  of  the  Regulation,  Member  States’  designated  competent  authorities 
shall publish annual transparency reports on their activities under the Regulation. Article 8(1) 
refers  to  the  minimum  information  that  these  reports  shall  include,  such  as  the  number  of 
removal orders issued under Article 3, the number of removal orders scrutinised under Article 
4 and the number of decisions imposing penalties pursuant to Article 18. 

5.1.3  Member States’ reporting obligation to the Commission under Article 21(1) of 

the Regulation  

According to Article 21(1), Member States shall collect from their competent authorities and 
the  HSPs  under  their  jurisdiction  and  send  to  the  Commission  by  31  March  every  year, 
information  about  the actions that they have taken in  accordance with  the Regulation  in  the 
previous calendar year.  

Article 21(1) refers to the type of information that Member States shall collect on measures 
taken  to  comply  with  the  Regulation,  such  as  the  number  of  removal  orders  issued  and  the 
number of items  of terrorist  content which have  been removed or  access  to  which has been 
disabled, the specific measures taken pursuant to Article 5 and the number of access requests 
to content preserved to allow investigations. 

5.2 New data sources 

5.2.1  Qualitative evidence to be collected from HSPs through an online survey and/or 

interviews  

Result and impact indicators consist of more qualitative types of indicators, which are a short- 
and long-term  consequence of the application of the Regulation.  These indicators should be 
collected by means of an online survey and/or interviews with HSPs.  

5.2.2  Qualitative  evidence  to  be  collected  from  national  competent  authorities 

through an online survey and/or interviews 

Result and impact indicators consist of more qualitative types of indicators, which are a short- 
and long-term  consequence of the application of the Regulation.  These indicators should be 
collected by means of an online survey and/or interviews with national competent authorities.  

4 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6.  Benchmark and targets 

The benchmark against which progress will be measured is the baseline situation following the 
entry into application of the Regulation. For existing data sources, the baseline will be defined 
based on the most recently collected data from  the entry into  application  of the Regulation; 
whereas, for new data sources, the first collected data will serve as baseline. 

With  regard  to  targets  (a  proxy  for  success  criteria),  given  the  transnational  nature  of  the 
dissemination of terrorist content online and the different data collection methods and sources, 
it is considered more effective to measure aggregated progress across Member States against 
an average baseline at Union level. When possible, progress will be measured for each Member 
State against its own baseline. 

5 

 
 
 
 
 
 
 
 
 
7.  Table: monitoring framework for the Regulation 

7.1 Data to be collected from Member States’ competent authorities 

ID 
KPI 

KPI 
1 

KPI 
2 

KPI 
3 

KPIs2 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

Establishment of clear rules 
by each Member State on 
penalties applicable to 
infringements of this TCO 
Regulation by HSPs and 
implementation of measures 
necessary to ensure that they 
are implemented (Art. 18.3) 

Member States shall ensure 
that the NCA(s), when 
deciding whether to impose 
a penalty and when 
determining the type and 
level of penalty, take into 
account all relevant 
circumstances (list the 
elements) (Art. 18.2) 
Member States to ensure 
that a systematic or 
persistent failure to comply 
with obligations pursuant to 
Article 3(3) (HSP to remove 
TCO or disable access as 
soon as possible, within 1 
hour) is subject to financial 
penalties of up to 4 % of the 
HSP’s global turnover of the 
preceding business year. 

Output 

Output 

Safeguard 
mechanisms 
and 
fundamental 
rights 
(freedom to 
conduct a 
business) 

Safeguard 
mechanisms 
and 
fundamental 
rights; 
freedom to 
conduct a 
business 

Output 

Safeguard 
mechanisms 

Penalties and 
administrative 
or judicial 
review 
proceedings 

Penalties and 
administrative 
or judicial 
review 
proceedings 

Penalties and 
administrative 
or judicial 
review 
proceedings 

Level 1 

Primary 

NCA 
consultation 
/ survey 

0-1 dichotomous 
scale  

(Yes or No) 

Level 1 

Primary 

NCA 
consultation 
/ survey 

Yes/no and 0-7 
discrete scale 

Level 1 

Primary 

NCA 
consultation 
/ survey 

0-1 dichotomous 
scale  

(Yes or No) 

- 0 means that the Member State did not establish clear rules on penalties 
applicable to infringements of this TCO Regulation for HSPs and did not 
implement measures necessary to ensure that such rules are implemented 
by HSPs 
- 1 means that the Member State established clear rules on penalties 
applicable to infringements of this TCO Regulation for HSPs and 
implemented measures necessary to ensure that such rules are 
implemented by HSPs 

Yes/No (have/have not taken into account all circumstances) 

- 0 means that the NCA, when deciding whether to impose a penalty and 
when determining the type and level of penalty, does not consider the 
relevant circumstances (none of the elements listed in Art. 18.2) 
- 7 means that the NCA, when deciding whether to impose a penalty and 
when determining the type and level of penalty, takes into account all 
relevant circumstances (all the elements listed in Art. 18.2) 

- 0 means that the Member State does not ensure that a systematic or 
persistent failure to comply with obligations pursuant to Article 3(3) (HSP 
to remove TCO or disable access as soon as possible, within 1 hour) is 
subject to financial penalties of up to 4 % of the HSP’s global turnover of 
the preceding business year. 
- 1 means that the Member State ensures that a systematic or persistent 
failure to comply with obligations pursuant to Article 3(3) (HSP to remove 
TCO or disable access as soon as possible, within 1 hour) is subject to 
financial penalties of up to 4 % of the HSP’s global turnover of the 
preceding business year. 

2 For the sake of comprehensiveness, a large number of indicators is included.    

6 

 
 
 
 
 
 
 
 
ID 
KPI 

KPI 
4 

HSP reporting to the NCA 
within 3 months of receipt 
of the decision and on an 
annual basis thereafter on 
the specific measures that it 
has taken and that it intends 
to take in order to comply 
with paragraphs 2 and 3 of 
article 5 (Art. 5.5) 

KPI 
5 

NCAs' activity(ies) or 
programme(s) to raise 
awareness among HSPs of 
the importance and risk of 
TCO (exchange of good 
practices and challenges) 

KPI 
6 

Consideration by NCAs and 
MS of the proportionality of 
the risk and resources of the 
HSP in the assessment of 
proactive measures taken by 
HSPs to moderate TCO (by 
performing risk assessment 
for instance) 

Output 

KPI 
7 

The number of effective 
procedures implemented by 
the Member States to enable 
the HSP and content 
providers to have the right 
to an effective remedy (Art. 
9.3)  

Result 

KPIs2 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

Output 

Public 
security 

Output 

Public 
security 

Procedures for 
removal orders 
including 
cross-border 

Transparency 
and awareness 
from EC, 
NCAs and 
HSPs 

Level 1 

Secondary 

NCA 
Transparenc
y Report 

0-1 dichotomous 
scale  

(Yes or No) 

- 0 means that the HSP does not always report to the NCA within 3 months 
of receipt of the decision and on an annual basis thereafter on the specific 
measures that it has taken and that it intends to take in order to comply 
with paragraphs 2 and 3 of Article 5 
- 1 means that the HSP always reports to the NCA within 3 months of 
receipt of the decision and on an annual basis thereafter on the specific 
measures that it has taken and that it intends to take in order to comply 
with paragraphs 2 and 3 of Article 5 

Level 3 

Primary 

NCA 
consultation 
/ survey 

0-1 dichotomous 
scale  

(Yes or No) 

- 0 means that the no awareness-raising activity or programme was put in 
place by the NCA regarding the importance and risks of terrorist content 
online; 
- 1 means that at least one awareness-raising activity or programme was 
put in place by the NCA regarding the importance and risks of terrorist 
content online. 

Competitivene
ss and effect 
on SMEs 

Level 3 

Primary 

NCA 
consultation 
/ survey 

1-5 Likert scale 

- 1 means strongly disagree; 
- 5 means strongly agree. 

Remedies 
mechanism 

Level 2 

Primary 

NCA 
consultation 
/ survey 

(2) 0-N discrete 
scale 

(absolute number 
of procedures for 
an effective 
remedy) 

- 0 means that the Member States did not implement any effective 
procedures to enable HSPs and content providers to have the right to an 
effective remedy. 

- N means that the Member States implemented N effective procedures to 
enable HSPs and content providers to have the right to an effective remedy 

Quartile computation of the indicator. The quartiles will have to be 
computed once the data will be collected. 

Fundamental 
rights 
(freedom to 
conduct a 
business) 

Safeguard 
mechanisms 
and 
fundamental 
rights 
(freedom of 
expression 
and 
information, 
the respect 
for private 
life, the 
protection of 
personal 
data) 

7 

 
 
 
 
 
 
 
 
KPIs2 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

ID 
KPI 

KPI 
8 

The number of cases in 
which removal orders and 
decisions taken were subject 
to administrative or judicial 
review proceedings and 
information on the outcome 
of the relevant proceedings 
(Art. 8.1 (c)) (by NCA) 

Result 

Safeguard 
mechanisms 

KPI 
9 

The number of decisions 
imposing penalties by NCA 
and a description of the type 
of penalty imposed (Art. 8.1 
(d)) 

Result 

Safeguard 
mechanisms 
and 
fundamental 
rights 
(freedom to 
conduct a 
business) 

Penalties and 
administrative 
or judicial 
review 
proceedings 

Penalties and 
administrative 
or judicial 
review 
proceedings 

Level 1 

Secondary 

NCA 
Transparenc
y Report 

Level 1 

Secondary 

NCA 
Transparenc
y Report 

Result 

Public 
security 

Procedures for 
removal orders 
including 
cross-border 

Level 1 

Secondary 

HSP & NCA 
Transparenc
y Report 

Result 

Public 
security 

Procedures for 
removal orders 
including 
cross-border 

Level 1 

Secondary 

NCA 
Transparenc
y Report 

KPI 
10 

Number of procedures for 
removal orders and referrals 
sent to/ received by the HSP 
as well as cross-border ones 
(Art. 7.3(c)) 

Number of cases in which 
the content which had been 
subject to a removal order 
was reinstated by the HSP 
following a reasoned 
decision finding that the 
concerned content was not 
considered as terrorist (Art. 
4.2) 
Frequency at which the HSP 
informed the NCA of the 
removal of terrorist content 
or the disabling of access to 
the terrorist content 
(indicating the specific time 
of the removal) (Art. 3.6) 

KPI 
11 

KPI
12 

(7) 0-N discrete 
scale 

(absolute number 
of times the 
removal orders 
were subject to 
admin or judicial 
proceedings) 

(8) 0-N discrete 
scale 

(absolute number 
of decisions 
taken to impose a 
penalty by NCA) 

(9) 0-N discrete 
scale 

(absolute number 
of removal 
orders / referrals 
received by 
HSPs) 
(14) 0-N discrete 
scale 

(absolute number 
of times content 
was reinstated on 
platform 
following a 
decision) 

- 0 means that none of the removal orders and decisions taken were subject 
to administrative or judicial review proceedings 
- N means that N times, the removal orders and decisions taken were 
subject to administrative or judicial review proceedings  

- 0 means that no decision was taken to impose a penalty by NCA and thus 
no description of the type of penalty imposed was provided. 
- N means that N decisions were taken to impose a penalty by NCA and 
thus a description of the type of penalty imposed was provided. 

- 0 means that no procedure for removal orders and referrals was received 
by the HSP including cross-border ones 
- N means that N procedure(s) for removal orders and referrals were 
received by the HSP including cross-border ones 

- 0 means that a HSP did not reinstate a content on the platform which had 
been subject to a removal order (following a reasoned decision finding that 
the concerned content was not considered as terrorist) 
- N means that a HSP reinstated N times a content on the platform which 
had been subject to a removal order (following a reasoned decision finding 
that the concerned content was not considered as terrorist) 

Result 

Public 
security 

Procedures for 
removal orders 
including 
cross-border 

Level 1 

Secondary 

NCA 
Transparenc
y Report 

0-1 dichotomous 
scale  

(Yes or No) 

- 0 means that the HSP does not always inform the NCA of the removal of 
terrorist content or the disabling of access to the terrorist content)   
- 1 means that the HSP always informs the NCA of the removal of terrorist 
content or the disabling of access to the terrorist content)   

8 

 
 
 
 
 
 
 
 
KPIs2 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

ID 
KPI 

KPI 
13 

KPI 
14 

KPI 
15 

Frequency at which the HSP 
informed the NCA of its 
incapacity (due to force 
majeure or de facto 
impossibility not 
attributable to the HSP) to 
comply with the removal 
orders (Art 3.7) 
Number of Member States 
connected to Europol for 
channelling removal orders 
and referrals 
Number of HSPs that the 
NCA (established in the 
same country as the HSP) 
considers to be exposed to 
TCO (based on number of 
removal orders issued by the 
HSP, at least 2 removal 
orders should have been 
issued by the HSP to be 
considered as exposed to 
terrorist content) (Art. 5.4 
(a)) 

Result 

Public 
security 

Result 

Public 
security 

Result 

Public 
security 

KPI 
16 

Number of NCAs 
publishing annual 
transparency reports on 
TCO 

Result 

Public 
security 

KPI 
17 

Additional costs due to 
additional resources needed 
related to the introduction of 
removal orders for NCAs 
and HSPs 

Impact 

HSPs 

Procedures for 
removal orders 
including 
cross-border 

Procedures for 
removal orders 
including 
cross-border 

HSPs policy 
for moderating 
terrorist 
content and its 
functioning, 
i.e., specific 
measures 

Transparency 
and awareness 
from EC, 
NCAs and 
HSPs 

Administrative 
burden and 
costs on public 
authorities 
related to the 
TCO 
Regulation 

Level 1 

Secondary 

NCA 
Transparenc
y Report 

0-1 dichotomous 
scale  

(Yes or No) 

- 0 means that the HSP does not always inform the NCA of its incapacity 
(due to force majeure or de facto impossibility not attributable to the HSP) 
to comply with the removal orders 
- 1 means that the HSP always inform the NCA of its incapacity (due to 
force majeure or de facto impossibility not attributable to the HSP) to 
comply with the removal orders 

Level 3 

Primary 

NCA 
consultation 
/ survey 

0-1 dichotomous 
scale  

(Yes or No) 

- 0 means that the MS is not connected to Europol for channelling removal 
orders and referrals; 
- 1 means that the MS is connected to Europol for channelling removal 
orders and referrals. 

Level 2 

Primary 

NCA 
consultation 
/ survey 

Level 2 

Secondary 

NCA 
Transparenc
y Report 
available 
online 

(16) 0-N discrete 
scale 

(Absolute 
number of 
HSP(s) 
considered as 
exposed to TCO) 

(18) 0-N discrete 
scale 

(absolute number 
of annual 
transparency 
reports published 
by NCAs) 

-0 means that no HSP is considered to be exposed to TCO; 
- N means that the NCA considers there are N HSPs exposed to TCO. 

-0 means that no annual transparency reports were published; 
- N annual transparency reports were published by NCAs 

Level 3 

Primary 

NCA 
consultation 
/ survey 

Yes/No 

0-3 discrete scale 

- 0 means that no additional costs were incurred by the NCA due to 
additional resources needed related to the introduction of removal orders; 
- 3 means that several costs were incurred by the NCA due to additional 
resources needed related to the introduction of removal orders. 

Yes/No 

9 

 
 
 
 
 
 
 
 
 
ID 
KPI 

KPIs2 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

KPI 
18 

Additional costs due to the 
reporting obligations of the 
TCO Regulation for NCAs 

Impact 

NCAs 

KPI 
19 

Costs related to the risk 
assessment done by NCA 

Impact 

NCAs 

Administrative 
burden and 
costs on public 
authorities 
related to the 
TCO 
Regulation 

Competitivene
ss and effect 
on SMEs 

Yes/No 

Level 3 

Primary 

NCA 
consultation 
/ survey 

Yes/No 

0-3 discrete scale 

- 0 means that no additional costs were incurred by the NCA related to the 
reporting obligations of the Regulation; 
- 3 means that several costs were incurred by the NCA related to the 
reporting obligations of the Regulation. 

Level 3 

Primary 

NCA 
consultation 
/ survey 

(20) 0-N discrete 
scale 

- 0 means that no costs were incurred by the NCA related to the risk 
assessment; 
- N relates to the amount of money incurred by the NCA for the risk 
assessment. 

ID 
KPI 

KPI 
20 

KPI 
21 

7.2 Data to be collected from HSPs 

KPIs3 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

Preservation of TCO by the 
HSP for a period of six 
months after the removal or 
disabling (duration, criteria 
for when the data should be 
preserved, implementation 
of appropriate technical and 
organisational safeguards) 
(Art. 6) 

TCO and related data that is 
preserved by the HSP is 
subject to appropriate 
technical and organisational 
safeguards, that such data is 
only accessed and processed 
for the purpose of the 
removal order by the HSP to 
ensure a high level of 

Output 

Output 

Safeguard 
mechanisms 
and 
fundamental 
rights 
(freedom of 
expression 
and 
information) 

Safeguard 
mechanisms 
and 
fundamental 
rights 
(freedom of 
expression 
and 
information, 
the respect 

Preservation of 
content and 
related data  

Level 2 

Primary 

HSP 
consultation 
/ Survey 

Yes/no and 1-5 
Likert scale 

Preservation of 
content and 
related data  

Level 2 

Primary 

HSP 
consultation 
/ Survey 

Yes/no and 1-5 
Likert scale 

Yes/No 

- 1 means that the HSP never preserved TCO for a period of six months 
from the removal or disabling of content (duration, criteria for when the 
data should be preserved, implementation of appropriate technical and 
organisational safeguards) 
- 5 means that the HSP always preserved TCO for a period of six months 
from the removal or disabling of content (duration, criteria for when the 
data should be preserved, implementation of appropriate technical and 
organisational safeguards) 
Yes/No 

- 1 means that when TCO and related data was preserved by the HSP, the 
HSP never implemented appropriate technical and organisational 
safeguards, in particular to ensure that such data is only accessed and 
processed for the purpose of the removal order 
- 5 means that when TCO and related data was preserved by the HSP, the 
HSP always implemented appropriate technical and organisational 
safeguards, in particular to ensure that such data is only accessed and 
processed for the purpose of the removal order 

3 For the sake of comprehensiveness, a large number of indicators is included.    

10 

 
 
 
 
 
 
 
 
 
 
ID 
KPI 

KPIs3 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

security of the personal data 
concerned (Art. 6.3) 

KPI 
22 

Information about the 
removal or disabling is 
accessible to the content 
provider by the HSP when 
TCO is removed (Art. 11.1) 

Output 

KPI 
23 

HSP informs complainant(s) 
within 2 weeks about the 
actions taken following a 
complaint received (Art. 
10.2) 

Output 

for private 
life, the 
protection of 
personal 
data) 

Safeguard 
mechanisms 
and 
fundamental 
rights 
(freedom of 
expression 
and 
information, 
the respect 
for private 
life, the 
protection of 
personal 
data) 
Safeguard 
mechanisms 
and 
fundamental 
rights 
(freedom of 
expression 
and 
information, 
the respect 
for private 
life, the 
protection of 
personal 
data) 

Remedies 
mechanism 

Level 2 

Primary 

HSP 
consultation 
/ Survey 

Yes/No and 1-5 
Likert scale 

Yes/No 

- 1 means that the HSP never makes information about the removal or 
disabling of content accessible to the content provider when it removes or 
disables access to terrorist content 
- 5 means that the HSP always makes information about the removal or 
disabling of content accessible to the content provider when it removes or 
disables access to terrorist content 

Complaints 
mechanism 

Level 2 

Primary 

HSP 
consultation 
/ Survey 

Yes/No and 1-5 
Likert scale 

Yes/No 

- 1 means that the HSP never informs complainant(s) within 2 weeks about 
the actions taken following a complaint received 
- 5 means that the HSP always informs complainant(s) within 2 weeks 
about the actions taken following a complaint received 

11 

 
 
 
 
 
ID 
KPI 

KPIs3 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

KPI 
24 

Consideration of the rights 
and legitimate interest of the 
users in the design and 
implementation of specific 
measures by HSPs to 
counter terrorist content 
online, in particular users’ 
fundamental rights 
concerning freedom of 
expression and information, 
respect for private life and 
protection of personal data 
(Art 5.3 (c)) 

KPI 
25 

KPI 
26 

Application of diligence and 
non-discrimination in the 
design and implementation 
of specific measures 
(safeguard mechanisms) by 
the HSP (Art 5.3 (d)) 

Existence of specific 
measures implemented by 
the HSP in relation to the 
identification and removal 
of or disabling of access to 
terrorist content (Art 7.3(a)) 

KPI 
27 

Existence of specific 
measures implemented by 
the HSP to mitigate the level 
of exposure of its services to 
terrorist content (Art 5.3 (a)) 

Output 

Safeguard 
mechanisms 
and 
fundamental 
rights 
(freedom of 
expression 
and 
information, 
respect for 
private life 
and 
protection of 
personal 
data) 

Output 

Safeguard 
mechanisms 

Output 

Public 
security 

Output 

Public 
security 

Consideration 
of users’ rights 
in the 
development 
of such 
safeguard’s 
mechanisms 

Consideration 
of users’ rights 
in the 
development 
of such 
safeguard’s 
mechanisms 
HSPs policy 
for moderating 
terrorist 
content and its 
functioning, 
i.e., specific 
measures 
HSPs policy 
for moderating 
terrorist 
content and its 
functioning, 
i.e., specific 
measures 

Level 1 

Primary 

HSP 
consultation 
/ Survey 

0-1 dichotomous 
scale  

(Yes or No) 

- 0 means that the HSP does not consider the rights and legitimate interest 
of the users in the design and implementation of specific measures to 
counter terrorist content online, in particular users’ fundamental rights 
concerning freedom of expression and information, respect for private life 
and protection of personal data  
- 1 means that the HSP considers the rights and legitimate interest of the 
users in the design and implementation of specific measures to counter 
terrorist content online, in particular users’ fundamental rights concerning 
freedom of expression and information, respect for private life and 
protection of personal data  

Level 1 

Primary 

HSP 
consultation 
/ Survey 

0-1 dichotomous 
scale  

(Yes or No) 

- 0 means that the HSP does not apply diligence and non-discrimination in 
the design and implementation of specific measures (safeguard 
mechanisms)  
- 1 means that the HSP applies diligence and non-discrimination in the 
design and implementation of specific measures (safeguard mechanisms)  

Level 1 

Secondary 

HSP 
Transparenc
y Report 

0-1 dichotomous 
scale  

(Yes or No) 

- 0 means that the HSP did not implement specific measures in relation to 
the identification and removal of or disabling of access to terrorist content 
- 1 means that the HSP implemented specific measures in relation to the 
identification and removal of or disabling of access to terrorist content 

Level 2 

Primary 

HSP 
consultation 
/ Survey 

0-1 dichotomous 
scale 

- 0 means that the HSP has not put in place nor planned by 2024 any 
specific measures in relation to the identification and removal of or 
disabling of access to TCO; 
- 1 means that the HSP has put in place specific measures in relation to the 
identification and removal of or disabling of access to TCO. 

12 

 
 
 
 
 
 
ID 
KPI 

KPI 
28 

KPI 
29 

KPI 
30 

KPI 
31 

KPIs3 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

Existence of specific 
measures implemented by 
the HSP to address the 
reappearance online of 
material which has 
previously been removed or 
to which access has been 
disabled because it was 
considered to be terrorist 
content. (Art. 7.3 (b)) 

Use of open-source 
technologies (specialised 
hash matching technology) 
to improve the detection of 
TCO across the HSPs 
community in the EU 

The extent to which the HSP 
makes use of the tools 
established by Europol or 
other temporary/ad interim 
ones for the effective 
implementation of the 
Regulation (Art. 14.4) 

Output 

Public 
security 

Output (as 
optional) 

Public 
security 

Output (as 
optional) 

Public 
security 

HSPs policy 
for moderating 
terrorist 
content and its 
functioning, 
i.e., specific 
measures 

Collaboration 
between HSPs, 
Member States 
& other 
relevant 
stakeholders 
for TCO 

Collaboration 
between HSPs, 
Member States 
& other 
relevant 
stakeholders 
for TCO 

Level 1 

Secondary 

HSP 
Transparenc
y Report 

0-1 dichotomous 
scale  

(Yes or No) 

- 0 means that the HSP did not implement specific measures to address the 
reappearance online of material which has previously been removed or to 
which access has been disabled because it was considered to be terrorist 
content 
- 1 means that the HSP implemented specific measures to address the 
reappearance online of material which has previously been removed or to 
which access has been disabled because it was considered to be terrorist 
content 

Level 3 

Primary 

HSP 
consultation 
/ survey 

0-1 discrete scale 
(Yes or No) 

- 0 means the HSP never makes use of open-source technologies to 
improve the detection of TCO;  
- 1 means the HSP makes use of open-source technologies to improve the 
detection of TCO. 

Level 3 

Primary 

HSP 
consultation 
/ Survey 

0-1 discrete scale 
(Yes or No) 

- 0 means that the HSP never uses the tools established by Europol or other 
temporary/ad interim ones;  
- 1 means that the HSP uses the tools established by Europol or other 
temporary/ad interim ones. 

The number of cases in 
which the HSP, after 
receiving a removal order or 
a decision pursuant to 
Article 4.4 or Article 5.4, 6 
or 7, challenged such a 
decision (Art. 9.1) 

Result 

Safeguard 
mechanisms 
and 
fundamental 
rights 

Remedies 
mechanism 

Level 2 

Primary 

HSP 
consultation 
/ Survey 

(1) 0-N discrete 
scale 

(absolute number 
of challenged 
decisions 
following a 
removal order) 

- 0 means that the HSP did not challenge the decision after receiving a 
removal orders or decisions pursuant to Article 4.4 or 5.4, 6 or 7 
- N is the number of challenged decision(s) by HSPs after receiving a 
removal orders or decisions pursuant to Article 4.4 or 5.4, 6 or 7 

Quartile computation of the indicator. The quartiles will have to be 
computed once the data will be collected. 

13 

 
 
 
 
 
 
ID 
KPI 

KPIs3 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

KPI 
32 

The number of complaints 
received by the HSP (Art 
21.1 (d) and Art 7.3 (d)) 
(from content providers) 

Result 

Safeguard 
mechanisms 

Complaints 
mechanism 

Level 1 

Secondary 

HSP 
Transparenc
y Report 

(3) 0-N discrete 
scale 

(absolute number 
of complaints 
received by a 
country) 

- 0 means that a HSP did not receive complaints from content providers 
- N means that a country received N complaints from content providers 

Quartile computation of the indicator. The quartiles will have to be 
computed once the data will be collected. 

KPI 
33 

Establishment of an 
effective and accessible 
mechanism by the HSP 
allowing for content 
providers to submit a 
complaint concerning 
removal or disabling of 
terrorist content (Art. 10.1) 

Result 

Safeguard 
mechanisms 
and 
fundamental 
rights 
(freedom of 
expression 
and 
information, 
the respect 
for private 
life, the 
protection of 
personal 
data) 

Complaints 
mechanism 

Level 2 

Primary 

HSP 
consultation 
/ Survey 

Yes/no and 0-3 
discrete scale 

- Yes/no 

- 0 means that the HSP has not established nor planned by 2024 an 
effective and accessible mechanism for content providers to submit a 
complaint concerning removal or disabling terrorist content online 
- 3 means that the HSP established an effective and accessible mechanism 
for content providers to submit a complaint concerning removal or 
disabling terrorist content online 

KPI 
34 

The number of times 
content was reinstated by 
the HSP following a 
complaint received from a 
content provider (ratio: 
complaint received / valid 
complaint = content restored 
to the platform) (Art 21.1 
(d) and Art 7.3 (d)) 

Result 

Safeguard 
mechanisms 

Complaints 
mechanism 

Level 1 

Secondary 

HSP 
Transparenc
y Report 

KPI 
35 

The number of 
administrative or judicial 
review proceedings brought 
by the HSP (Art. 7.3 (e)) 

Result 

Safeguard 
mechanisms 

Penalties and 
administrative 
or judicial 
review 
proceedings 

Level 1 

Secondary 

HSP 
Transparenc
y Report 

(4) 0-N discrete 
scale 

(absolute number 
of times content 
has been 
reinstated on 
platform 
following a 
complaint) 
(5) 0-N discrete 
scale 

(absolute number 
of administrative 
or judicial appeal 
proceedings by 
HSPs) 

- 0 means that a HSP did not reinstate a content on the platform following 
a complaint received from a content provider 
- N means that a HSP reinstated N times a content on the platform 
following a complaint received from a content provider 

Quartile computation of the indicator. The quartiles will have to be 
computed once the data will be collected. 

-0 means that no administrative or judicial review proceedings was 
brought by the HSP 
-N means that N administrative or judicial appeals proceedings were 
brought by the HSP 

Quartile computation of the indicator. The quartiles will have to be 
computed once the data will be collected. 

14 

 
 
 
 
 
 
 
 
 
 
ID 
KPI 

KPI 
36 

KPIs3 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

The outcome of 
administrative or judicial 
review proceedings brought 
by the HSP (Art. 7.3 (e))  

Result 

Safeguard 
mechanisms 

Penalties and 
administrative 
or judicial 
review 
proceedings 

Level 1 

Secondary 

HSP 
Transparenc
y Report 

0-1 dichotomous 
scale 

(Yes or No) 

- 0 means that the HSP did not report on the outcome of administrative or 
judicial review proceedings 
- 1 means that the HSP report on the outcome of administrative or judicial 
review proceedings. 

KPI 
37 

The number of cases in 
which the HSP was required 
to reinstate content or access 
thereto as a result of 
administrative or judicial 
review proceedings (Art. 7.3 
(f)) 

Result 

Safeguard 
mechanisms 
and 
fundamental 
right 
(freedom of 
expression 
and 
information, 
the respect 
for private 
life, the 
protection of 
personal 
data) 

Penalties and 
administrative 
or judicial 
review 
proceedings 

Level 1 

Secondary 

HSP 
Transparenc
y Report 

KPI 
38 

Number of procedures for 
removal orders and referrals 
sent to/ received by the HSP 
as well as cross-border ones 
(Art. 7.3(c)) 

Result 

Public 
security 

KPI 
39 

Number of removal orders 
scrutinised (Art. 7.3(c))  

Result 

Public 
security 

KPI 
40 

Number of removal orders 
that were rejected (Art. 
7.3(c)) 

Result 

Public 
security 

Procedures for 
removal orders 
including 
cross-border 

Procedures for 
removal orders 
including 
cross-border 

Procedures for 
removal orders 
including 
cross-border 

Level 1 

Secondary 

HSP & NCA 
Transparenc
y Report 

Level 1 

Secondary 

Level 1 

Secondary 

HSP 
Transparenc
y Report 

HSP 
Transparenc
y Report 

(6) 0-N discrete 
scale 

(absolute number 
of times a 
content was 
reinstated on 
platform 
following admin 
or judicial review 
proceedings) 

(9) 0-N discrete 
scale 

(absolute number 
of removal 
orders / referrals 
received by 
HSPs) 
(10) 0-N discrete 
scale 

(absolute number 
of scrutinised 
removal orders) 
(11) 0-N discrete 
scale 

(absolute number 
of rejected 
removal orders) 

-0 means that a HSP did not reinstate a content on the platform as a result 
of administrative or judicial review proceedings 
-N means that a HSP reinstated N times a content on the platform as a 
result of administrative or judicial review proceedings 

Quartile computation of the indicator. The quartiles will have to be 
computed once the data will be collected. 

- 0 means that no procedure for removal orders and referrals was received 
by the HSP including cross-border ones 
- N means that N procedure(s) for removal orders and referrals were 
received by the HSP including cross-border ones 

- 0 means that no removal order was scrutinised  
- N means that N removal orders were scrutinised  

- 0 means that no removal orders were rejected  
- N means that N removal orders were rejected 

15 

 
 
 
 
 
 
 
 
 
ID 
KPI 

KPIs3 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

KPI 
41 

Number of decisions to 
effectively remove terrorist 
content online (following 
removal order) (Art. 7.3(c)) 

Result 

Public 
security 

Procedures for 
removal orders 
including 
cross-border 

Level 1 

Secondary 

HSP 
Transparenc
y Report 

KPI 
42 

Number of removal orders 
that were effectively 
proceeded within one hour 
of receipt of the removal 
order by the HSP (Art. 3.3) 

Result 

Public 
security 

Procedures for 
removal orders 
including 
cross-border 

Level 2 

Primary 

HSP 
consultation 
/ Survey 

KPI 
43 

Number of HSPs putting 
proactive measures in place 
to identify, remove or 
disable the access to TCO 

Result 

Public 
security 

KPI 
44 

Number of HSPs publishing 
transparency reports on 
TCO 

Result 

Public 
security 

HSPs policy 
for moderating 
terrorist 
content and its 
functioning, 
i.e., specific 
measures 

Transparency 
and awareness 
from EC, 
NCAs and 
HSPs 

Level 2 

Secondary 

HSP 
Transparenc
y Report 

Level 2 

Secondary 

HSP 
Transparenc
y Reports 
available 
online 

(12) 0-N discrete 
scale 

(absolute number 
of decisions 
taken to 
effectively 
remove terrorist 
content online) 
(13) 0-N discrete 
scale 

(absolute number 
of removal 
orders that were 
effectively 
processed within 
1 hour) 
(15) 0-N discrete 
scale 

(absolute number 
of HSP(s) putting 
in place 
proactive 
measures to 
identify, remove 
or disable the 
access to TCO) 
(17) 0-N discrete 
scale 

(absolute number 
of annual 
transparency 
reports published 
by HSPs) 

- 0 means that no decision was taken to effectively for removal orders and 
referrals was received by the HSP including cross-border ones 
- N means the number of decisions taken to effectively remove terrorist 
content online (following removal order) 

- 0 means that no removal orders were effectively proceeded within one 
hour of receipt of the removal order by the HSP  
- N means that N removal orders were effectively proceeded within one 
hour of receipt of the removal order by the HSP 

- 0 means that no HSP has put proactive measures in place to identify, 
remove or disable the access to TCO 
- N is the number of HSP(s) that have put proactive measures in place to 
identify, remove or disable the access to TCO  

-0 means that no annual transparency reports were published; 
- N annual transparency reports were published by HSPs. 

16 

 
 
 
 
 
 
 
ID 
KPI 

KPIs3 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

Result 

Fundamental 
rights 
(freedom to 
conduct a 
business) 

Competitivene
ss and effect 
on SMEs 

Level 2 

Primary 

HSP 
consultation 
/ survey 

1-5 Likert scale 

-1 means strongly disagree; 
- 5 means strongly agree. 

The specific measures 
implemented by the HSP are 
targeted and proportionate, 
taking into account, in 
particular, the seriousness of 
the level of exposure of the 
services of the hosting 
service provider to terrorist 
content as well as the 
technical and operational 
capabilities, financial 
strength, the number of 
users of the services of the 
hosting service provider and 
the amount of content they 
provide. (Art 5.3) 

Number of HSPs using 
digital tools (automated 
tools) for the moderation of 
TCO (to detect/identify 
TCO) 

KPI 
45 

KPI 
46 

Impact 

HSPs 

KPI 
47 

Additional costs due to the 
reporting obligations of the 
TCO Regulation for HSPs 

Impact 

HSPs 

KPI 
48 

Number of HSPs 
contracting third parties 
(vendors) for TCO 
moderation  

Impact 

HSPs 

Administrative 
burden and 
costs on HSPs 
related to TCO 
moderation 

Administrative 
burden and 
costs on HSPs 
related to TCO 
moderation 

Administrative 
burden and 
costs on HSPs 
related to TCO 
moderation 

Level 3 

Primary 

HSP 
consultation 
/ Survey 

0-1 dichotomous 
scale (Yes or No) 

- No means that the HSP does not use digital tools for the moderation of 
TCO; 
- Yes means that the HSP uses digital tools for the moderation of TCO. 

Level 3 

Primary 

HSP 
consultation 
/ Survey 

Yes/No 

0-3 discrete scale 

- 0 means that no additional costs were incurred by the HSP related to the 
reporting obligations of the Regulation; 
- 3 means that several costs were incurred by the HSP related to the 
reporting obligations of the Regulation ("to a great extent") 

Yes/No 

Level 3 

Primary 

HSP 
consultation 
/ Survey 

0-1 dichotomous 
scale  

- 0 means that the HSP does not contract third parties for TCO moderation 
- 1 means that the HSP contracts third parties for TCO moderation. 

17 

 
 
 
 
 
ID 
KPI 

KPIs3 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

KPI 
49 

Costs related to the 
application of the 1-hour 
deadline for removal orders 

Impact 

HSPs 

KPI 
50 

Costs related to the content 
moderation or filtering 
technologies for HSPs (for 
TCO only) 

Impact 

HSPs 

KPI 
51 

Number of new staff to be 
hired for this activity of 
content moderation/filtering 
technologies (for TCO only) 

Impact 

HSPs 

KPI 
52 

Costs related to the trainings 
of staff for this activity 

Impact 

HSPs 

Administrative 
burden and 
costs on HSPs 
related to TCO 
moderation 

Administrative 
burden and 
costs on HSPs 
related to TCO 
moderation 

Administrative 
burden and 
costs on HSPs 
related to TCO 
moderation 

Administrative 
burden and 
costs on HSPs 
related to TCO 
moderation 

7.3 Data to be collected from other sources 

Level 3 

Primary 

HSP 
consultation 
/ Survey 

Yes/No 

0-3 discrete scale 

- 0 means that no costs were incurred by the HSP related to the application 
of the 1-hour deadline for removal orders; 
- 3 means that several costs were incurred by the HSP related to the 
application of the 1-hour deadline for removal orders. 

Yes/No  

Level 3 

Primary 

HSP 
consultation 
/ Survey 

Yes/No 

0-3 discrete scale 

- 0 means that no costs were incurred by the HSP related to the content 
moderation or filtering technologies; 
- 3 means that several costs were incurred by the HSP related to the 
content moderation or filtering technologies. 

Yes/No 

Level 3 

Primary 

HSP 
consultation 
/ Survey 

0-N discrete 
scale 

- N is the discrete number of new staff hired for the activity of content 
moderation/filtering technologies 

Level 3 

Primary 

HSP 
consultation 
/ Survey 

Yes/No 

0-3 discrete scale 

- 0 means that no costs were incurred by the HSP related to the training of 
new staff; 
- 3 means that several costs were incurred by the HSP related to the 
training of new staff. 

Yes/No 

18 

 
 
 
 
 
 
 
 
 
 
KPIs 

Type of 
indicator  

Monitoring 
areas 

Sub-
monitoring 
area 

Level of 
reporting 
obligation  

Type of 
data 

Data 
sources 

Units of 
measurement 

Units of measurement - description 

ID 
KPI 

KPI 
53 

European Commission 
awareness-raising 
activity(ies) or awareness 
programme among Member 
States and HSPs of the 
importance and risk of TCO 
and to support related 
activities 

Output 

Public 
security 

KPI 
54 

Number of EU 
institutions/organisations 
publishing annual 
transparency reports on 
TCO 

Result 

Public 
security 

Transparency 
and awareness 
from EC, 
NCAs and 
HSPs 

Transparency 
and awareness 
from EC, 
NCAs and 
HSPs 

Level 3 

Secondary 

Level 3 

Secondary 

European 
Commission 
and desk 
research  

0-1 dichotomous 
scale  

(Yes or No) 

EU 
institutions / 
orga. 
Transparenc
y report 
available 
online 

(19) 0-N discrete 
scale 

(absolute number 
of annual 
transparency 
reports published 
by EU 
institutions/organ
isations) 

- 0 means that no awareness-raising activity or programme among MS and 
HSP was put in place by the EC regarding the importance and risks of 
terrorist content online; 
- 1 means that at least one awareness-raising activity or programme among 
MS and HSP was put in place by the EC regarding the importance and 
risks of terrorist content online. 

-0 means that no annual transparency reports were published; 
- N means that the EU institutions/organisation has published N annual 
transparency reports. 

19