EUROPEAN COMMISSION Brussels, 12.9.2018 SWD(2018) 408 final COMMISSION STAFF WORKING DOCUMENT



IMPACT ASSESSMENT

Accompanying the document Proposal for a Regulation of the European Parliament and of the Council on preventing the dissemination of terrorist content online



{COM(2018) 640 final}

{SEC(2018) 397 final}

{SWD(2018) 409 final}



Table of contents 1.Introduction 2.Problem statement 3.Why should the EU act? 4.Objectives: What is to be achieved? 5.What are the available policy options? 6.What are the impacts of the policy options? 7.How do the options compare? 8.Preferred option 9.How will actual impacts be monitored and evaluated?



1.Introduction 1.1.Political and legal context The availability and proliferation of illegal content online represents a very important public policy and security concern in the EU. This impact assessment report focuses in particular on the availability and spread of terrorist content on online hosting services, and analyses a series of legislative options.

The Commission has been working on the issues of terrorist content online, but also on other types of illegal activities online, such as illegal hate speech, child sexual abuse material (CSAM), and illegal commercial practices such as infringements of intellectual property rights, selling of illicit drugs, counterfeits or other illicit goods create significant harm in society and reduce trust in the digital economy. Illegal content is shared online in particular through online services that allow upload of third party content, also known as hosting service providers. At EU level, the general legal framework for illegal content removal by hosting service providers is provided by the E-Commerce Directive , which balances the need for effective action for illegal content removal, safeguards against over-removal of legal content, and promotes growth and innovation in the Digital Single Market. Building on this general legal framework, the Commission has reinforced and strengthened the fight against illegal content online with a series of regulatory and non-regulatory initiatives. The regulatory initiatives include the Directives combatting the child sexual abuse material 2011/93/EU, combatting terrorism (EU) 2017/541, as well as the recently agreed revision of the Directive on audio-visual media services 2016/0151 (COD), and the proposal for a Directive on Copyright 2016/0280 (COD). In addition, the Commission's Recommendation on measures to effectively tackle illegal content online C(2018)1177, and the Communication on tackling illegal content online COM 2017/555 cover all types of illegal content and offer guidance to hosting service providers and authorities. These initiatives have been complemented by a series of voluntary initiatives, such as the Code of Conduct on combatting Hate Speech adopted in May 2016, the EU Internet Forum on terrorist content online launched in December 2015, as well as the June 2016 Memorandum of Understanding on counterfeit goods sold online. Despite these initiatives, there remains a particular concern about the availability of terrorist material online. In 2017, there were a total of 205 foiled, failed and completed terrorist attacks in the EU, which killed over 68 and injured over 800 . The continued high level of terrorist threat in the EU is accompanied by continued concern about the role of the internet in aiding terrorist organisations to pursue and fulfil their objectives to radicalise and to recruit, to facilitate and direct terrorist activity. In June 2018, the European Council called upon the Commission 'to present a legislative proposal to improve the detection and removal of content that incites hatred and to commit terrorist acts'. This followed previous calls from the European Council of 22-23 June 2017 for industry to ‘develop new technology and tools to improve the automatic detection and removal of content that incites to terrorist acts. This should be complemented by the relevant legislative measures at EU level, if necessary’. Similarly, the European Parliament, in its resolution on Online Platforms of June 2017, urged the platforms concerned "to strengthen measures to tackle illegal and harmful content", and called on the Commission to present proposals to address these issues. These calls were echoed by statements issued by the leaders of the G7 and G20 in 2017 in the shared fight against terrorism. Tackling terrorist content online is part of a broader comprehensive approach to counter terrorism, which includes the criminalisation of terrorist offences and prosecution of perpetrators, as foreseen under the Terrorism Directive (EU) 2017/541. Whilst ideally perpetrators will always be brought to justice, some of the material is produced in war zones and ungoverned territories, outside of the EU, making it difficult for EU law enforcement to successfully investigate and prosecute. Whilst the prosecution of terrorist perpetrators remains a priority, of equal importance is to reduce the accessibility to online terrorist content online so as to protect online users from such harmful content and to make it harder for terrorists to operate online. Therefore the main focus of this Impact Assessment is to assess problems related to terrorist content online, its impact on the Digital Single Market and options to further reduce its availability. The impact of these options on, as well as the complementarity to, criminal investigation will be part of the assessment. 2.Problem statement 2.1.Scope and further context Internet access continues to expand across the globe, while at the same time it becomes ever easier and cheaper to create and disseminate content online, allowing users to connect and share information with billions of individuals across the globe. Almost 70% of Europeans are regular users of social media; a similar share watch videos or live-streaming, or listen to music online regularly, 47% use file storage or sharing services regularly, 51% engage on review and rating platforms and a similar share read blogs, comment on articles or news websites. However, the ability to reach such a large audience with minimum cost also attracts individuals who want to use the Internet for illegal purposes. The presence of illegal content is a concern for users and citizens at large, public authorities as well as online hosting services providers. These services now store unprecedented amounts of third-party content and information. A rich and diverse ecosystem of such hosting services has emerged across the globe, consisting of some very large service providers, but also many small and specialised ones. Services in scope of this analysis cover a range of business models and a mix of hosting services such as content storage and distribution services (e.g. web hosting, file sharing, online media sharing), networking, collaborative production and matchmaking (e.g. social networks, marketplaces, online games), and selection and referencing services (e.g. online search tools, rating platforms) . Each company is free to conduct its business under their preferred business model, and to set out and apply its terms of service to define the content it is willing to host. Hosting service providers have a complex set of incentives relating to their own policies towards illegal content online, depending on their size, profitability, reputation, and business model, which includes incentives to maintain trust in its services (in particular by avoiding the proliferation of illegal content on its services). The scale and speed necessary for effective content moderation has spurred companies’ action to protect their users as well as their own reputation, taking into account that they are often best placed to swiftly act against any illegal or harmful content that they host. Service providers may also have incentives not to take any action against such content being disseminated on its platforms taking into account their users preferences. The dissemination of illegal content online continues to be of strong and urgent societal and political concern. To tackle this issue, the relevant legal framework includes the general provisions set out in the E-Commerce Directive, and targeted instruments, some of which have been recently adopted or under negotiations, and which should produce their effects in the near future. This includes the Directive on Combatting Terrorism, proposed measures in the context of the revision of the Audiovisual Media Services Directive (AVMSD), and the proposal for a Directive on Copyright in the Digital Single Market. Other relevant legal acts include the Directive against Child Sexual Abuse, or the Directive on the Enforcement of Intellectual Property Rights. The Commission’s 2017 Communication and 2018 Recommendation have started to bear fruit, but have not yet produced their full effect across all types of illegal content. Besides these recent non-binding initiatives, the Commission continues to support and to monitor targeted voluntary cooperation with industry: On illegal hate speech, reporting on the implementation of the Code of Conduct shows that the voluntary collaboration can yield very positive results in a relatively short time. A key indicator is the progress made to achieve the takedown of 70% of the content reported as hate speech. Considering the careful balance which needs to be struck in the takedown of illegal hate speech and the high risks of misidentification on lawful speech as illegal, this indicator shows good progress since the beginning of the cooperation. The strategic priority in this area is to extend the cooperation to a broader number of hosting services, expected to further improve the situation. With regards to counterfeit products the recent guidelines released by the Commission should give further clarity to hosting services and improve the results of the specific measures. In addition, the Memorandum of Understanding is bringing results which show a positive evolution and are expected to further improve. Other voluntary actions are tackling the problem from different angles, most recently through a Memorandum of Understanding on online advertising, signed on the 25th June and aiming at further incentivising website owners to take vigilant action against the sale of counterfeit products on their platforms. On CSAM, the work of the WePROTECT Global Alliance to end child sexual exploitation online, and the long-standing EU support to the INHOPE network of hotlines on child sexual abuse content continue to make progress. The analysis of the current situation, including the effects of the implementation of Directive 2011/93/EU, is ongoing and is expected to be finalised by the end of 2020. Therefore, legislating at this point in the area of child sexual abuse material would be premature as the necessary evidence is still being collected. On other types of illegal content, legal proposals (e.g. Copyright Directive, New Deal for Consumers) or emerging cooperation (e.g. Consumer Protection Joint Action) cover further action and are expected to yield positive results. Similarly, in the case of terrorist content online, ongoing efforts under the EU Internet Forum are bringing positive results. However, while the Commission continues its efforts and monitors closely the effectiveness of measures concerning other types of illegal content, there is a particular sense of urgency for effective policy intervention against terrorist content spreading online, as highlighted by the Commission in its Recommendation and as called upon by the European Council. The changes in operations of terrorist organisations lead to the unpredictable and unprecedented use of hosting services for disseminating terrorist propaganda. Content is often hosted by various small service providers, then shared through mainstream social media. The voluntary efforts and collaboration are bound by their natural limitations of only gathering a relatively small number of hosting service providers, and fail to spread good practice and gather sufficient impact and transparency from the hosting services most affected at any given time by the spread of terrorist content. Furthermore, speed is of the essence to remove terrorist content, therefore further efforts are needed to detect content as fast as possible, as well as removing flagged content quickly to avoid further dissemination across platforms and subsequent harm. In addition, the fragmentation of procedural rules across Member States limits the effectiveness and efficiency in the cooperation between authorities and hosting service providers. This impact assessment analyses the problems arising from the use of hosting service providers for spreading terrorist content online, and proposes and analyses the potential impacts of further legislative measures. Stakeholders' views on the scope of a possible EU intervention

Stakeholders' views have changed over time, in light of the emerging sector-specific legal initiatives and results of the voluntary cooperation. In past consultations starting with 2010, 2012, and even 2016, the majority of hosting service providers was pointing to the need for further legal certainty and harmonisation in respect to the notice-and-action procedures. In the 2018-run consultations, service providers were rather supporting continuing targeted voluntary actions or very targeted, content-specific interventions, focusing on terrorist content. This follows the new steps taken with the adoption of soft instruments by the Commission – i.e. the Communication, and the Recommendation on measures to tackle illegal content online. An important number of Member States, especially the majority of those most directly affected by terrorism, have called for legislation specifically in the area of terrorism. The European Council in its conclusions in June 2018 has called upon the Commission to take legal initiative to tackle the dissemination of terrorist content online. 2.2.Problem tree 2.3.What are the problems? The dissemination of terrorist content online is an overall problem which is affecting private and public interests, hampering trust, innovation and growth in the Digital Single Market, and affecting public security in the EU. 2.3.1.Hosting service providers are abused for the dissemination of terrorist content online affecting the business models and users' trust in the digital single market While the Internet plays a crucial role enabling instant access to information, content and knowledge across borders, online hosting services are in certain cases abused by third parties to carry out illegal activities online, notably the dissemination of material related to terrorism. Hosting illegal content creates important reputational damage on companies; it can affect a company’s business model, affecting relations with users consuming or providing content, or advertisers, and other interconnected services. Social networks and media sharing platforms, whose business model is often based on advertising, have faced intense public criticism when terrorist or other illegal content was reported on their networks. This has in some cases triggered direct revenue loss, following a major backlash from certain advertisers, and led to user distrust . Similar incidents were also reported by other business models such as web hosting services. Estimates point to a basis of over 10.500 hosting service providers established in Europe, and almost 20 000 HSPs established both in Europe and in the US and Canada. Over 90% of European hosting services are SMEs, amounting to roughly 9700 companies , over 45% of which are micro-enterprises, and 40% medium-sized businesses . SMEs offering hosting services are particularly vulnerable to exploitation for illegal activities, not least since they tend to have limited capacity to deploy state-of-the-art content moderation tools or specialized staff. Whilst the percentage of referrals sent to micro, small and unregistered companies only amounted to 7% in 2015 (when the EU IRU was first set up), they amount to 68% today. Europol reports that over 150 companies were identified as hosting terrorist content, a large part of them being established outside of Europe and offering their services across the Single Market. Data on location of users accessing the services identifies only two likely to operate in a single Member State. More than half of these companies offer file storage and sharing services. Most of these are SMEs, and reach a very limited audience (median of 20 000 views per month , with a first quartile including below 1000 views). Around 20% are online media sharing services (including audio-visual media sharing platforms), half of which have a limited audience in Europe (average number of monthly views below 250.000), the rest reaching an audience of over 1 million users per month, and the large multinational companies reaching hundreds of million views. Just under 10% offer web-hosting services. Just under 10% are social networking and discussion forums, half of which have less than 150 000 views, with the major social media platforms as outliers with a very large user base. A very small number are online market places, collaborative production services, or selection and referencing services. Consumers using hosting services also incur the negative consequences. 61% of respondents to the dedicated Eurobarometer claim to have seen online some type of illegal content, the vast majority point to some sort of scams, frauds or illegal commercial practices. 6% of respondents claimed to have seen terrorist content online, reported exposure is however 11% for young people aged 15-24. Stakeholders’ views In the various consultations carried out in the context of this impact assessment, hosting service providers underlined the challenge raised by the volume of illegal online content and acknowledged their negative to very negative impact on their users and on their reputation. This was in particular confirmed by companies reporting, in the context of the EU Internet Forum, on the presence of terrorist content on their service, affecting relations with users, content creators, advertisers, but also payment processors. On the other hand, 65% of the respondents to a Eurobarometer on illegal content carried out in June 2018 did not consider the Internet to be safe for its users. In the public consultation, on the other hand, 75% of the respondents considered the internet to be safe. This divergence could be explained by the fact that the phone-run Eurobarometer collected the views of a representative, randomly built sample of 33 000 EU citizens, both users and non-users of online hosting services, whereas the internet based public consultation received 8749 replies from individuals, self-reported frequent Internet users, and self-selected in the sample by voluntarily replying to the consultation. 2.3.2.Terrorist content is accessible online, reappears, and spreads across service providers posing a security challenge Given its illegal nature but also the lack of systematic reporting on terrorist content identified online, comprehensive and fully reliable numbers of illegal content available online cannot be established. However, reports on the amount of content removed by certain companies, the number of pieces of content referred by public authorities or figures produced by research can give an indication of the size of the problem .

For example, Europol’s IRU has made over 50,000 decisions for referrals to service providers about terrorist content in their platforms since July 2015 and the UK’s Internet Referral Unit (CTIRU) identified 300,000 pieces of terrorist content between 2010 and 2018 . However, for those companies that are taking proactive measures to identify terrorist content themselves, in general, referrals only account for a small proportion of the total content removed. For example, Twitter has suspended over 1.2 million accounts for violations of their Terms of Service related to the promotion of terrorism between August 2015 and December 2017 and Facebook took action on 1.9 million pieces of Daesh and al-Qaeda content in the first quarter of 2018 . The terrorist group Daesh, which since its beginning has put in place an aggressive communication strategy, has been producing an average of ~1200 new propaganda items every month in 2015-2017, according to research surveys . This material spreads online across various service providers through link-sharing and reposting, as illustrated in Figure 2. While in January 2018 the number of new items of official Daesh propaganda in January 2018 had gone down to 700, it also represents a recent increase from 300 items in December 2017. According to Europol , by 2017 over 150 social media platforms were identified as being abused by terrorists for propaganda dissemination. Other analyses point to a number of 400 online platforms being misused by Daesh supporters for spreading terrorist material in 2017, of which 145 service providers were identified for the first time in the second half of the year as being abused . According to Europol, whilst industry and law enforcement action have resulted in a reduction of the terrorist abuse of larger mainstream providers, similar progress has yet to be made with start-up social media and companies with limited resources. Figure 2 Shares of links to Daesh propaganda hosted on the top twenty service providers in Q1 2018. While the four most affected providers have reduced the amount of Daesh links they host by approximately 80% since Q2 2017, the total amount hosted in the other 16 most affected platforms has barely decreased in absolute numbers. These 16 companies hosted around 10% of Daesh links out of the 20 most affected companies in Q1, a share that increased to around half in Q4 2017 and one third in Q1 2018 Another important indicator is how quickly terrorist content spreads across providers. Terrorist content is considered most harmful in the first hours of its appearance because of the speed at which it is disseminated and therefore multiplied. Research has found that one third of all links to Daesh propaganda is disseminated within 1 hour of release, and three quarters of all links are shared within four hours of release . In summary, terrorist content is not always detected at the point of uploading, is not sufficiently quickly removed when flagged to companies, and continues to move easily from one platform to another, allowing terrorists to operate online to intimidate, to groom and recruit, to facilitate and direct terrorist activity, and to glorify in their atrocities. Stakeholders' views: During consultations with Member States two Member States reported an increase or substantial increase of the amount of terrorist content online, four indicated that the situation remained the same and twelve that terrorist content online has decreased. Decrease in volume was mostly attributed to losses incurred by Daesh and to the voluntary efforts undertaken in particular by the large internet companies as well as Internet Referral Units. However, Member States underlined the dispersal of terrorist content across a greater number of smaller service providers as an emerging trend posing a persistent security challenge. 2.4.What are the problem drivers? 2.4.1.Hosting service providers face legal fragmentation and uncertainty The E-Commerce Directive establishes in its Article 3 the country of origin principle, following which providers of information society services have to comply with the law of the Member State where they are established. This principle, pursuant to which Member States are not allowed to restrict the provision of services from another Member State, represents the cornerstone of the Digital Single Market. Restrictive measures against a given information society service provider established in a different Member State can only be allowed if they are necessary and proportionate to protect objectives of public policy, public security, public health or protection of consumers. The EU presents a patchy legal framework as regards the possibility for national competent authorities and courts to request the takedown of illegal terrorist content, or the blocking of websites publishing such content. In the majority of the cases, the order has to stem from a jurisdictional body, but some legislations allow for an administrative order – subject to court appeal. These laws cohabit with non-regulatory practices based on the voluntary cooperation by online services providers, based on the implementation of their own terms of service. This is also reflected in the transposition of the Directive on Combatting Terrorism. Article 21 calls on Member States to put in place measures to ensure swift removal of content inciting to the commission of terrorist acts. Fifteen Member States have informed the Commission that they have already transposed it. The information about transposition measures refers to existing or amended legislation allowing a prosecutor or a court to order companies the removal of content or the blocking of content or a website. In some cases, there are time limits of 24 hour or 48 hours for companies to take action. In most cases, however, these powers can only be exercised within a criminal procedure. Member States also have in place measures in-line with the transposition of the E-Commerce Directive’s Article 14 (3) for establishing procedures governing the removal or disabling of access to information. Annex 7 presents in detail the different provisions in Member States’ law, governing procedures for notification by national authorities related to terrorist content in particular . These measures have a different scope in terms of offences covered, time limits for removal and consequence of non-compliance. In some cases notice-and-action procedures are limited to manifestly illegal content (Austria, France, Portugal), which includes in some instances terrorist content. Only recently, Germany included, under the NetzDG, some offences related to terrorist content such as disseminating propaganda material by unconstitutional organisations (including terrorist organisations), forming a terrorist organisation or encouragement the commission of a serious violent offence endangering the state. Another source of potential legal uncertainty is that, while terrorist offences are defined in the Terrorism Directive, there is at EU level no explicit definition of terrorist propaganda, contrary to provisions that exist under national law. Some MS have legislation in place obliging hosting service providers to report to authorities when they identify illegal content (proactively or following a notice from a third party) and others are imposing the need for a legal representative to be established on their respective jurisdiction. While so far limited to the field of copyright, national courts have followed different interpretations of the scope of duties of care ranging from a limited duty to take down (upon notice), to more extended duties to prevent also future infringements of the same kind under specific circumstances . At the same time, Member States are increasingly imposing transparency obligations on online hosting services, regardless of their place of establishment, with the purpose of increasing the public oversight on the way these intermediaries’ implement their content policies and their reactions to notices . In addition, in a few Member States law enforcement authorities can issue removal orders and an important number of Member States have established mechanisms for law enforcement authorities to refer content for the voluntary removal of companies, allowing them to achieve faster removal outside a criminal procedure. Many Member States have established some sort of voluntary framework to facilitate cooperation with large HSPs. The fragmentation across the administrative procedures entails significant costs for small and medium size hosting service providers offering their services cross-border. Companies face a legal complexity which makes it difficult and costly to comply with the different procedures and timeframes for removal of content and reporting obligations, and, importantly, fragmentation of definitions of what constitutes terrorist propaganda and the need to have a legal representative in the Member State’s jurisdiction. There is a risk that legal fragmentation would further increase, and, on ground of national security, request exemptions from the country of origin principle. While this is in principle possible within EU law, the heavy costs of legal fragmentation could lead to either non-compliance or non-enforceability of the respective laws, or to insurmountable obstacles in particular to small European hosting service providers and blockages in the Digital Single Market.

Stakeholders' views: Based on a specific questionnaire sent to companies on terrorist content, all responding companies acknowledged a serious concern posed by the risk of diverging legislation in different countries to address terrorist content online, and that smaller companies would suffer more from 28 different sets of rules, which could end up being used as a tool for competitive advantage by larger companies . 2.4.2.Member States face obstacles in intervening against terrorist content online With regards to terrorist content online, the Commission has been engaging with over 20 companies, including many of the most affected and largest platforms, as part of the EU Internet Forum to support the implementation of an Action Plant to Combat Terrorist Content Online and the terrorism-specific measures foreseen under the Recommendation on illegal content. Whilst important progress has been achieved in terms of increased co-operation, improving responses to referrals, deployment of proactive measures and co-operation between the industry (including for example the Database of Hashes of content identified and removed by companies) as well as increased transparency, the Commission has struggled to establish contact with all companies affected by terrorist content. To better measure progress under the Recommendation, the Forum developed a set of indicators to enhance collective understanding of the level of effort being applied to reduce accessibility to terrorist content online. Of the 33 companies for whom the Commission had a point of contact, only 14 responded to the two requests for reporting on their efforts. A more detailed summary is set out below in Table 1 The difficulty in establishing contact with all the companies which are affected by terrorist content is due to a number of reasons. The Commission and indeed Europol and Member States do not have contact details for all the platforms affected. Whilst outreach efforts have been made with many, in some cases, it may be that the companies are completely unaware of EU initiatives and for some, it might be that they are aware but choose not to engage. This obstacle makes it difficult to process referrals or to have any meaningful discussion or co-operation with regards to terrorist content hosted on their sites. Finally, another challenge arising from removal of terrorist content online is the risk of companies’ removals interfering with investigations. All Member States responding to the terrorism-specific questionnaire highlighted this risk, since the online activities of a suspect can be crucial for law enforcement and the removal of his or her content can impair an investigation and reduce the chances of disrupting criminal activity and obtaining the necessary evidence for prosecution purposes. Furthermore, the content that terrorist groups disseminate online gives competent authorities key insights on the groups’ messages, strategy and methods. Companies' removal of such content without retaining it or informing EU law enforcement or Europol, potentially risks hampering an essential source of intelligence and/or evidence for competent authorities to be able to effectively prosecute. Member States therefore consider it necessary to adopt measures to limit any interference with investigations and to ensure removed content is not lost but able to support operational and analytical requirements. Stakeholders' views: During stakeholders' consultation, Member States highlighted the progress achieved under the voluntary process. However, some also noted that regardless, the number of companies engaged remained limited. As exemplified by the reporting exercise under the Recommendation in which only half of the companies who were asked to provide data responded. 2.4.3.Measures to detect, remove and prevent the dissemination of terrorist content are not effectively or evenly implemented by hosting services The Commission’s Communication on tackling illegal content online , as well as the subsequent Recommendation set out a number of measures governing the detection, removal and prevention of dissemination of illegal content online. The Communication identifies different sources of information about the presence of illegal content and alleged illegal content on hosting services, from court orders or administrative decisions and notices from competent authorities (e.g. law enforcement bodies), specialised ‘trusted flaggers’, intellectual property rights holders to ordinary users. Hosting service providers have implemented specific procedures to various degrees for the different types of notices. Practices include trusted flaggers schemes for specific types of content, including, in some cases, fast removal of terrorist content referred by law enforcement. The Recommendation encourages hosting service providers to put in place notice and action procedures and to take specific proactive measures, including where appropriate automated detection tools (subject to the necessary safeguards). The Recommendation builds on and consolidates efforts under voluntary processes currently in place. With respect to terrorism, initial insights into follow-up actions by hosting service providers show that the recommended measures are not being applied consistently by all. Those companies who have engaged under the EU Internet Forum have undertaken a number of actions and progress can visibly be seen. The table below summarises results, as reported by companies to the Commission. Hosting services PARTICIPATING Proactive takedown (% of the total removed content, compared to content removed following notices) NUMBER OF referrals received % REMOVALS (out of content notified SPEED Reached out to 20 platforms, including. Facebook, Youtube, Microsoft, Twitter, Internet archive, Justpaste.it, Wordpress, snap, Soundcloud After Recommendation: Baaz, Dropbox, Mega, Userscloud, Telegram Varies across companies: e.g. 44% (Q2 2018) to 83% (Q4 2017) reported by one SME; 99% by Facebook in Q1 2018 Database of hashes used by 13 companies For EU IRU: Q4 2017: 8,103 referrals Q1 2018: 5,708 referrals For EU IRU: Q4 2017: 89% Q1 2018: 61% (But between 96 and 100% for “big four”: FB, YouTube, Microsoft and Twitter) Proactive measures: 5 companies reported to remove content within 1h, out of which 3 companies could do it within 1 minute, using proactive measures. For referrals: majority of companies not removing within one hour; nevertheless some have jumped from 0% to 66% removals within one hour or 8% to 52%. With regards to referrals, Europol established an EU Internet Referral Unit (EU IRU) in 2015 to actively scan the internet for terrorist content and then refer it to the hosting service providers, accompanied by an assessment. This voluntary arrangement has resulted in over 50,000 decisions for referrals across over 80 services providers in more than 10 languages. The EU IRU works closely with its Member States' counterparts. The number of national IRUs has steadily risen to 6 and an additional number of Member States are active in this area but do not have IRUs as such. However, companies do not always respond in a sufficiently swift manner, despite efforts to reduce the life-span of such content online, and some companies do not respond at all to referrals. The majority of companies (whether large or small businesses) have not managed to remove content referred within one hour. Whilst referrals are an important element of the response, it has become clear that referrals alone are not able to achieve the necessary impact with regards to volume and speed of removal. For example, it might be a matter of hours, or days, before IRUs identify a certain piece of content on a platform. By contrast, some companies using automated detection claim that content can be identified within a matter of minutes or less. Furthermore, where companies are using proactive measures, referrals only seem to amount to a very small percentage of the total amount of terrorist content removed. Nevertheless, the survival time of terrorist content online varies considerably across platforms, ranging from minutes to days, depending on whether proactive or reactive measures are employed. Generally speaking, the longer the content is able to survive online, the more views it may receive, and the more harm it may cause. This is also linked to the fact that terrorist content is spread quickly and easily across platforms. Often terrorists or their supporters set up accounts in advance of publications in order to facilitate the swift dissemination of new terrorist propaganda items. Even where content is removed from one site, it can be uploaded onto other services, or even onto the same site from which it was previously removed. To address this challenge 13 companies are using a shared database of hashes including content identified as terrorist according to the companies’ terms and conditions. The database is collectively used to prevent the re-uploading and dissemination of removed content across platforms. According to reporting by companies, the database includes over 80,000 distinct hashes of terrorist videos and images. Reporting suggests varying but increasing degrees of use as well as differences in the frequency of use by the participating companies. The feedback on the impact of the Database of Hashes in terms of content removed is still limited and is not systematic across platforms. Other tools to detect terrorist content, include the E-Glyph tool , or Twitter’s URL database. No data is available for estimating costs of setting up, maintaining or using such a database. Costs can vary, from the use of simple (and not particularly reliable) hashing software, to 100 million EUR reported by one major video sharing platform for developing such advanced tools for filtering copyright protected materials.

Recently, two major online platforms have reported to be using machine-learning classifiers to flag content as possible terrorist or violent extremist material. Companies claim high levels of accuracy in flagging content for human review, but media reports have recently highlighted cases where the systems have misidentified content from conflict areas as being terrorist content. The UK Home Office has also reported the development of a tool by ASI Data Science to detect terrorist materials online. For text detection, natural language processing is considered by and large immature for accurately identifying illegal hate speech or other violent speech , and anecdotal reports point to misidentification and takedown of lawful speech which show that automated detection of hateful content still needs to be combined with human verification. While the cost of such tools varies depending on the type of content, they involve, besides highly skilled data science staff for development, access to an important volume of content to train the classifier, as well as substantial human verification to accompany and correct the training The Commission’s Communication on tackling illegal content called for such automated tools to be deployed along with human verification before the takedown of content. This serves a double purpose: to train the classifiers, and to limit the erroneous identification by still immature tools, leading to removal of lawful content and limiting the possibility for users to express themselves freely. Despite the efforts by some service providers, not all companies implement such tools, and the effectiveness or success rates of content taken down varies. Resource constraints (financial, technical and human review capabilities) may be of consideration, as well as proportionality of the efforts necessary, considering the risk, frequency and impact of illegal content companies might store at any given time. For some hosting service providers, the core value proposition of the service offered is the encrypted storage of data ; in these cases, the hosting service cannot access the content by design and, implicitly, does not run software to detect illegal content, but depends on external notices. Micro, medium and small enterprises are particularly vulnerable facing illegal content which might be uploaded by their users. Discussions within the EU Internet Forum highlighted how small companies can be suddenly exploited for storing and disseminating terrorist propaganda. The EU Internet Forum has also offered a space for collaboration and support, allowing SMEs to benefit from shared tools they can adapt to their content policy. However, not all companies are equally affected by the problem. For example, a small company headquartered within the EU emphasised during the public consultation that it had tried to set up a content moderation system with an investment as high as half the staff, but, with no incidence of illegal content, it had to repurpose scarce resources. At the same time, there is a recurrent criticism of the existing systems implemented by the hosting services providers regarding the lack of sufficient safeguards and the limitation to users’ rights through erroneous removal of legal content or suspension of accounts. Both the Communication and the Recommendation raise these issues and call on companies to set up reliable safeguards for over-removal of lawful content. Calls have been made not only for the establishment of complaint procedures (counter-notices) and/or increased clarity around these, but also for future reports to quantify the removals and the outcomes of complaints. Academics submitting replies to the open public consultation have pointed to the lack of real incentives (apart from sporadic media attention) companies might have to deal with counter-notices, whereas non-treatment of notices can more easily lead to legal and financial consequences. They also underlined that existing counter-notice procedures are by and large underused, with the few companies who do report on counter-notices listing on a yearly basis only one-to-two digits numbers. In addition, a rich literature is emerging on the need for specific safeguards for algorithmic decision-making, where biases and inherent errors and discrimination can lead to erroneous decisions. This was particularly flagged by academics in the feedback on the Inception Impact Assessment, pointing to the plausible impacts on vulnerable groups. Stakeholders' views Regarding notice and action procedures, the 2018 Eurobarometer on illegal content reported that : ·33% of the individual respondents have seen allegedly illegal content and have reported it to the hosting service; ·over 83% of them found the procedure easy to follow; ·30% of the respondents whose content was in their view wrongly removed had issued a counter-notice; ·64% of the respondents whose content was removed found both the content removal process, and the process to dispute removal not to be transparent; ·8% of the respondents said their content had been wrongly removed (self-reported) by a hosting service, including on false grounds of terrorist content or child sexual abuse, but mostly for claims of IPR infringements; ·27% claim to have contacted the hosting service and contested the decision. 2.4.4.Hosting services providers’ policies to detect, assess and remove content are not transparent for users and public authorities to monitor companies' action against terrorist content. The Communication on illegal content identified a general lack of transparency on online platforms’ content and removal policies, as well as a lack of detailed data on their removal decisions. The Commission’s Recommendation called for further measures to inform both users as well as public authorities. Transparency is needed to be able to assess the full breadth of the problem and the impact of efforts, to provide the necessary public assurance that sufficient mitigating measures are being taken to protect citizens against illegal online activity, as well as to capture the necessary evidence for investigative and prosecutorial purposes. The voluntary dialogues for the different types of illegal content have significantly improved the reporting on the advances made by hosting services providers and notice providers, not least though the alignment of reporting indicators and some comparable frequency of measurements . However, this is necessarily limited to the service providers joining the dialogues.

Furthermore within those dialogues, reporting has not always been followed-up with consistently. For instance on terrorist content online, the EU Internet Forum developed a set of indicators in order to improve transparency in this area and better track progress. Whilst first rounds of reporting ensured greater transparency in relation to how those who responded are addressing this problem, only approximately 14 out of the 33 companies that were invited to provide data, responded. This figure accounts for just a fraction of the overall number of platforms which are reportedly being exploited for terrorist purposes. Beyond the voluntary dialogues, only a relatively limited number of intermediaries publish transparency reports . While they offer some level of understanding on the company’s actions, they are rarely granular enough to inform policy-makers or law enforcement on the evolution of illegal content online and they are not comparable. The reports generally present limited data, sometimes in absolute values, on the notices received, content taken down and, in even fewer instances, the counter-notices and shares of content reinstated after takedown. Importantly, they rarely distinguish with precision the types of illegal content beyond copyright infringements and content signaled through ‘government notices’. Recent developments include YouTube’s report on the enforcement of its community guidelines, presenting also data on proactive measures to take down terrorism content or child abuse; Telegram publishes an (API accessible) daily report on the number of ISIS channels and bots taken down. In the open public consultation, only one third of the users who notified content received information about the course of action taken by the hosting service. For organisations flagging content, the solution adopted is generally a monitoring system including regular checks of the URL until the content is taken down. However, while they provide an external check in the process, such systems are not resource-efficient neither for the flaggers (cost of setting up the system), nor for the hosting service (generating unnecessary traffic and network congestion), and run the risk of blocking for suspicious network activity. Stakeholders' views According to the open public consultation, nearly half of the individuals who had notified content did not receive any information from the service regarding the notice, and one third reported to have been informed about the follow-up given to the notice. For one fifth, the content was reported to be taken down within hours. One fifth of the respondents who had their content removed from hosting services reported not to have been informed about the grounds for removal. In terms of removals, out of those who claimed in the open public consultation that they reported illegal content, 7% say the content was kept online, while 22% say it was taken down. Out of those who responded in the Eurobarometer data collection, 45% says the reported data was removed, while 20% says it was kept online. Further 7% says that the content was kept online but with more restricted access, while 8% said that it was kept online but was slightly modified. 2.5.What are the consequences of the problem? Availability of terrorist content online can accelerate the radicalisation process, recruit terrorist supporters, and galvanise, facilitate and instruct terrorist activity. Organisations such as Daesh, Al Qaida and others have used the internet to promote their ideology, groom and recruit, and more recently, to incite, train and provide instructions for carrying out terrorist attacks through the use of detailed online magazines and manuals. According to research conducted by VOX-Pol on a set of UK-based convicted terrorists, “a third (32%) prepared for their attacks by using online resources”. “More than half (54%) of all actors used the Internet to learn about some aspect of their intended activity”, a figure which has increased over time up to 76%, which could be an indication of increased use of the internet by would-be terrorists or of increased availability of terrorist material online. In the aftermath of terrorist attacks, terrorists use the internet to glorify in their atrocities and urge others to follow suit. Some anecdotal cases also point to examples where it appears that perpetrators of terrorist attacks have been radicalised solely through the internet. For instance, in Denmark in 2017, a teenage girl was found guilty of attempted terrorism for having tried to make bombs to be used in terrorist attacks against her own former local school and against a Jewish school in Copenhagen. The girl became radicalised via the internet and chat contacts within just a few months . The nature of content might be different but terrorist content poses a serious harm to victims and the society more generally. Images of murder and torture of terrorists' victims, such as the immolation of the encaged Jordanian pilot in 2015 by Daesh, are For the general public who happen to fall upon this material, and for the victims and their relatives, this material can be extremely distressing. However, not all terrorist content is necessarily violent. Often, particularly in the early grooming stages, terrorists' will use a softer approach. Since early 2017, Daesh has even resorted to using cartoons to glorify and promote their ideology to young children with the intention of indoctrination. For the curious user, such material has the potential to groom and recruit. And for those who have the desire to inflict harm and commit terrorist acts, terrorist material can be used to galvanise them into action (as was the case with the attack on the Thalys train in August 2015 ), and equip them with the capability and know-how to convert that intent into devastating action. With regards to consequences of response measures, flaws in content removal procedures can lead to unintended removal of legal content, also to minimise exposure to legal or financial consequences, as such, any regulatory incentives need to be appropriately balanced with safeguards to ensure freedom of speech is protected. Another consequence of the problems is the inability of public authority to monitor the evolution of the problem, as insufficient data prevents public authorities to assess the effectiveness of current procedures to detect and remove illegal content, as well as their impact on public and private interests. Finally, the overall consequence of the problems collectively is a lack of trust in the online environment, which limits the growth and innovation capacity of hosting services providers in the Digital Single Market. 2.6.How will the problem evolve? Without action, the problem of terrorist content online would persist to different degrees, depending on the effectiveness of the different measures already in place to curb its dissemination. While terrorist content originating from organisations such as Daesh might be forced increasingly to smaller platforms as a result of reinforced content moderation policies, it is likely that terrorist organisations will continue to seek innovative way to exploit the reach of large platforms, in parallel with disseminating content on smaller ones. Therefore, hosting services would continue to be abused for the dissemination of illegal content, in particular those who do not participate in the voluntary dialogues, or those who do not comply with the Recommendation on illegal content. Although the voluntary process would expand, it is unrealistic to expect that all platforms which host illegal content would join the current voluntary process, or indeed take heed of the Recommendation. The spreading and reappearance of content across platforms would continue. The uneven application of procedures to detect and remove illegal content would likely continue, given the non-binding nature of the Recommendation. The ability of authorities and policy-makers to monitor the evolution of the problem, and for users to understand content moderation principles would remain patchy – with those platforms participating in the voluntary dialogues providing more information than those who do not. At the same time, a series of recent and forthcoming legal instruments are expected to bring further clarity and give legal incentives to some types of services providers to act against terrorist content they host. In particular, the revised Audio-Visual Media Services Directive would require video-sharing platforms whose principal purpose or essential functionality is the provision of audiovisual content to take a series of measures to protect the general public from terrorist content, as well as from illegal hate speech, and CSAM, and to protect minors from harmful content. Measures expected to be put in place by services include prohibiting such content through their Terms of Service, flagging mechanisms available to users and feedback to users on the follow-up of the notices, as well as complaint and redress mechanisms for users in case of disputes over the application of the envisaged measures (i.e. counter-notices). Even though not directly imposing obligations on hosting service providers, the measures implemented by Member States under Article 21 of the Terrorism Directive will ensure that material that amounts to public provocation to commit a terrorist offence will be removed (e.g. in the context of a criminal investigation). However, action by Member States alone to refer or block material, will always fall short considering it is the companies who are best placed to know which technology can be applied and how in order to ensure a safe online environment for their users. Nor would this approach achieve an effective EU-wide approach with regards to securing the removal of terrorist material for preventative efforts. 3.Why should the EU act? 3.1.Legal basis

As far as EU action may take the form of a legislative proposal in the area of terrorist content, the legal basis depends on the primary objective and scope of the proposal. Given the problems that this Impact assessment is addressing and the solutions proposed, Article 114 TFEU is the most appropriate legal basis for an EU intervention. In general, in the field of terrorist content online, the legal basis could be found mainly either in Article 114 TFEU (approximation of laws for the improvement of the internal market) or in Article 83 TFEU (judicial cooperation in criminal matters for the definition of criminal offences). According to Article 114 TFEU, the European Parliament and the Council shall adopt the measures for the approximation of the provisions laid down by law, regulation or administrative action in Member States which have as their object the establishment and functioning of the internal market. Following well-established case-law of the CJEU , Article 114 is the appropriate legal basis to address differences between Member State provisions which are such as to obstruct the fundamental freedoms and thus have a direct effect on the functioning of the internal market, and a possible legal basis for measures to prevent the emergence of future obstacles to trade resulting from differences in the way national laws have developed. As the Court has said, the EU ‘legislature cannot be prevented from relying on that legal basis on the ground that the protection of other policy objectives is a decisive factor in the choices to be made.’ Recourse to Article 114 TFEU as a legal basis does not presuppose the existence of an actual link with free movement between the Member States in every situation covered by the measure founded on that basis. To justify recourse to Article 114 TFEU as legal basis what matters is that the measure adopted on that basis must actually be intended to improve the conditions for the establishment and functioning of the internal market . Article 114 TFEU can also be used to impose obligations on services providers established outside the territory of the EU where their service provision affects the internal market, when this is necessary for the desired internal market goal pursued. For instance, the Regulation on Geoblocking or the Proposal for a Regulation on promoting fairness and transparency for business users of online intermediation services consider that the ultimate effect of the instrument would be undermined if the geographic scope was limited to services providers established in the EU. Finally, Article 114 TFEU can also serve as a legal basis to impose an obligation to third country companies to appoint a representative within the territory of the Union, insofar as this is merely incidental with regard to the main purpose or component of the act. This is the case, for instance, for the NIS Directive , exclusively based on Article 114 TFEU. According to Article 83(1) TFEU the European Parliament and the Council may establish minimum rules concerning the definition of criminal offences and sanctions in the areas of particularly serious crime with a cross-border dimension such as terrorism. Article 83(2) TFEU provides that if the approximation of criminal laws and regulations of the Member States proves essential to ensure the effective implementation of a Union policy in an area which has been subject to harmonisation measures, directives may establish minimum rules with regard to the definition of criminal offences and sanctions in the area concerned. In view of the problems to be addressed, Article 83 seems to be a less appropriate legal basis for an EU intervention, as the analysis does not conclude on the need for ‘establish[ing] minimum rules concerning the definition of criminal offences and sanctions’, but rather for harmonization of conditions for hosting service providers to provide services cross-border in the Digital Single Market. The intervention would not seek to harmonise criminal content, as the Terrorism Directive, nor obligations under criminal proceedings. 3.2.Subsidiarity: necessity of EU action According to the principle of subsidiarity laid down in Article 5(3) of the Treaty, action at EU level should be taken only when the aims envisaged cannot be achieved sufficiently by Member States alone and can therefore, by reason of the scale or effects of the proposed action, be better achieved by the EU. Several Member States have legislated in the field of removal of illegal content online in relation to aspects such as notice and action or transparency. However, the internet is by its nature cross-border, and content hosted in one Member State can normally be acceded from any other Member State. The need to ensure public security at national level has to be balanced with the fundamental freedom of provision of services and freedom of establishment in the Internal Market, which justifies the existence of derogations to that freedom for reasons of public security. Public security has a particular importance in the area of terrorism, given the gravity of the risks involved. In principle, the place of establishment should not be a barrier to provide services across the EU (subject to derogations, as long as justified, proportionate and necessary). As a result of this, a patchy framework of national rules is appearing and risks to increase, which would jeopardise an effective exercise of the freedom of establishment and the freedom to provider services in the EU (see 2.4.1). Intervention at national level cannot solve this problem. This justifies the need for EU action, as echoed by the Council Conclusions of June 2018 inviting the Commission to present a legislative proposal in this area. 3.3.Subsidiarity: added value of EU action The different and diverging legal regimes applicable to online intermediaries increase compliance costs while also being the source of legal uncertainty on the qualification of the content as illegal content and on the scope of responsibilities and obligations for service providers, raising a serious concern among operators. As representatives of the industry have stated during consultations, diverging requirements in different Member States impose compliance with diverging regulations, or there is a risk that new regulations would do so. In addition, the effects of any action taken under national law would be limited to a single Member State. EU action reducing compliance costs, allowing their predictability and enhancing legal certainty would ensure that hosting service providers’ actions against terrorist content online can be streamlined and scaled up increasing its effectiveness. This would allow more companies to take action strengthening the integrity of the digital single market. EU action in this area would ensure a coherent approach applicable to online platforms operating in all Member States. Action at EU level would be only partially effective if it was limited to hosting services providers established in the EU. This would create a competitive disadvantage vis-à-vis companies established in third countries, which would not be subject to any compliance costs in this regard. Furthermore, the effect in the availability of terrorist content online would be only limitedly achieved. Due to the interest of companies outside the EU to continue providing its services within the Digital Single Market, the EU can act as a standard-setter for measures to combat terrorist content online globally. 4.Objectives: What is to be achieved? 4.1.General objective The general objective is to establish uniform rules to prevent the misuse of hosting services for the dissemination of terrorist content online in order to guarantee the smooth functioning of the Digital Single market, with high levels of trust and security.

4.2.Specific objectives 4.2.1.Facilitate the provision of online services across the DSM by limiting further legal fragmentation The first specific objective is to ensure a predictable, harmonised legal framework on the obligations related to handling terrorist content required from hosting service providers when providing their services cross-border. This would bring legal certainty and allow companies to streamline processes, limiting compliance costs with fragmented rules. 4.2.2. Increase the effectiveness of measures to detect, identify and remove terrorist content The second specific objective is to ensure greater effectiveness of the mechanisms to reduce availability of terrorist content in the EU, in particular by ensuring that hosting service providers and national authorities take appropriate, effective and proportionate actions against the spread of terrorist content. 4.2.3.Increase transparency and accountability of hosting services for measures taken to detect, identify and remove terrorist content The third specific objective is to ensure that both citizens, and in particular Internet users, and public authorities have sufficient information to appraise the actions taken by hosting service providers to detect, identify and remove illegal content. 4.2.4.Improve the ability of relevant authorities to intervene against terrorist content online and to combat crime A fourth specific objective relates to enabling public authorities to exercise effective action against the dissemination of illegal content online and take appropriate action including investigations into the underlying criminal offences. 4.2.5.Safeguard against the risk of erroneous removal of legal content and ensure protection of fundamental rights A fifth specific objective is to provide for appropriate measures to safeguard against erroneous removal of legal content and ensure the effective exercise of fundamental rights online. 5.What are the available policy options? 5.1.What is the baseline from which options are assessed? In the baseline scenario, reactive measures will continue to be carried out in the framework set out by the E-commerce Directive and the liability regime set out therein, under which hosting service providers are under certain conditions exempted from liability for the illegal content they store. The Directive on Combating Terrorism should be transposed into national laws by September 2018, and obliges Member States to "take the necessary measures to ensure the prompt removal of online content constituting a public provocation to commit a terrorist offence". As clarified in recital 21, the Directive leaves Member States a wide margin of discretion as to the form of removal measures, stipulating that these may take the form of legislative, non-legislative or judicial action and that it is without prejudice to voluntary action taken by the internet industry. Information transmitted to the Commission so far indicates divergence between the transposition of the measures between Member States. The transposition is a mix of notice and takedown actions under the general framework of Directive 2000/31/EC and measures based on criminal law, often constituting general competence of seizure of illegal content. Although such measures seem to meet the requirements of the Directive, they are unlikely to result in a significant reduction of the volume of terrorist content online. Furthermore, the Audio-visual Media Directive's transposition into national law would similarly be completed by end 2020, obliging Member States "to ensure that video-sharing platform providers under their jurisdiction take appropriate measures to protect […] the general public from programmes, user-generated videos, and audiovisual commercial communication [containing] a public provocation to commit a terrorist offence, [as well as] child pornography, […] racism and xenophobia." The Recommendation on measures to effectively tackle illegal content online can also be expected to bring further results, not least in what concerns detecting, notifying, and removing illegal content online, as well as detailed transparency reporting requirements. However, given the non-binding nature of the Recommendation, the adoption of the measures relies on the voluntary action of a limited number of companies and is unlikely to effectively tackle and keep up with the changing online operations from terrorist organisations. Under the baseline, the implementation of current non-regulatory measures would continue and evolve. The cooperation between industry, Member States, Europol and the Commission under the EU Internet Forum would be pursued and would be reinforced. Efforts would focus both on expanding the number of companies participating, and on following up and supporting further commitments from companies and sharing of tools and best practices. In particular, some companies, mainly major ones, would continue to develop and use proactive measures, such as human review combined with automated detection tools. Gradual expansion of the use of the filtering database developed by industry currently used by 13 companies up from 4 last year can also be expected; and company initiatives such as the Global Internet Forum to Counter Terrorism would also continue to reach out to new companies. Further R&D&I support would reinforce the financing of research projects and networks such as VoxPol, which help better understand the evolving threat of terrorist content online. Such research has been very valuable in the past in designing targeted regulatory and non-regulatory responses and has helped hosting service providers and public authorities to keep track of the nature of the problem. National laws in the area of illegal content can be expected to continue to evolve. The German NetzDG, which covers hate speech and some elements related to terrorist content, has entered into force on 1 January 2018, and first transparency reports are expected by July 2018. The French government has similarly been reported to consider a dedicated law on hate speech.

Stakeholders’ views Member States representatives in the E-Commerce Expert Group supported the ongoing collaboration under European and national-level cooperation with hosting service providers and considered that the measures to implement the Recommendation are ongoing. Horizontal instruments at EU level would hence not be needed yet. Furthermore whether under the Open Public Consultation, or the high-level meeting with a wide range of companies in January 2018, stakeholders raised the preference for targeted intervention on specific issues of particular public value. On terrorist content, Member States representatives noted that the EU Internet Forum has generated important progress and good results within a short period of time. However, they also highlighted shortcomings and that results (in terms of effective and swift removal and proactive measures by companies) did not match the challenge at hand. A number of Member States called for legally binding obligations to speed up progress, ensure sustainable results and compliance by a larger number of companies. Other Member States highlighted that more time was required to fully assess progress before deciding on legislation. Hosting services, as reported in the public consultation and targeted consultations, do not consider that additional regulatory intervention would be conducive to better tackling illegal content online, highlighting the general cooperation and good results in the actions taken through the sector specific voluntary dialogues. Civil society organisation advocating digital rights are also in favour of maintaining the baseline option. Associations representing large numbers of hosting service providers stated that, if legal instruments are envisaged at EU level, they should in any case be problem-specific and targeted whilst broadly supporting further harmonisation of notification information. Different companies highlight the different available capabilities across businesses, as well as the different incentives and technical possibilities depending on their value propositions. Caution was expressed as to proactive measures and the feasibility of establishing strict time limits on takedown of content from upload, pointing to burdens for SMEs, and to incentives for over-removal. Companies were also generally open to cooperation especially with government agencies or law enforcement flagging illegal content. In the Eurobarometer, the overwhelming majority of respondents agreed that arrangements need to be put in place to limit the spread of illegal content online, and that, at the same time, freedom of expression needs to be protected. In the open public consultation, respondents were asked to scale the importance of having such measures in place, and to protect freedom of expression online, respectively; they considered overall that the protection of freedom of expression as a more important measure. In the Open Public Consultation, 30% of respondents considered that the current legal framework for tackling each of the different types of illegal content was effective and nearly 40% found that actions currently taken by hosting service providers are effective. At the same time, 60% expressed that different types of illegal content should be dealt with in different legal frameworks, to take into account specificities, while 12% disagreed with this. 5.2.Options discarded at an early stage A number of options were considered early in the analysis process and discarded against effectiveness, coherence and proportionality criteria: 1.Transformation of the Recommendation on measures to effectively tackle illegal content online into binding rules across all types of illegal content In this option, the provisions of the Recommendation would be made legally binding. However, preliminary assessment against the set objectives points to an unnecessary and disproportionate intervention, and raises concerns in particular as regards the necessity to establish generalised requirements for proactive measures to detect and remove all types of allegedly illegal content, amounting to a general monitoring of the content uploaded by users of hosting services. Moreover, the specificities of the different types of content need to be acknowledged and treated accordingly, including regarding the effectiveness and evolution of the current voluntary measures. Hence, to date, such policy option is not suited for a ‘one size fits all intervention’ and does not appear necessary and proportionate and would not justify a potential interference with fundamental rights, in particular, with freedom of expression and information, data protection and freedom to conduct a business and compliance with the existing legal framework, in particular the E-Commerce Directive. 2.Harmonised rules across all types of illegal content clarifying notice-and-action procedures A harmonised legal approach to all categories of illegal content, including terrorist content, CSAM, illegal hate speech, content infringing intellectual property rights, but also content rendered illegal in national law, could cover notice-and-action procedures, as clarified in the Recommendation. This could include an obligation for hosting service providers to put in place an easy to access mechanisms for submissions of notices by electronic means, as well as easy to access complaint mechanisms for content providers supported by the obligation to inform content providers when their content is taken down. The option would also include minimum quality requirements for submitted notices, and transparency obligations through annual reports, as well as sanctions for systematic non-compliance with the obligations. This instrument would be applicable to hosting service providers who offer/direct their services to users residing in the Union, irrespective of their place of establishment. While the option would be proportionate and would have limited interference with fundamental rights, it would have only a limited impact in addressing the objective to increase the effectiveness of measures to detect, identify and remove terrorist content and to improve the ability of the authorities to intervene against terrorist content, compared to the evolving baseline. The limited effect is specific to terrorist content, not least due to the specific behavioural patterns in the dissemination of the content, as well as the expected transposition and enforcement of the revised AVMS Directive, already establishing grounds for such measures in the case of video-sharing platforms. In addition, the Recommendation gives guidance on such horizontal measures, and should bring further certainty and relief to hosting service providers. The option is therefore discarded at the current stage, as it does not sufficiently address the policy objectives, without precluding potential benefits such a measure could bring in the future, depending on the evolution of the baseline for all types of illegal content. 3.Enhanced voluntary approaches There have been several Member States and stakeholders advocating continued or enhanced efforts under the voluntary arrangements within the EUIF. The current voluntary arrangements within the EU Internet Forum could be further strengthened by developing a Memorandum of Understanding or a code of conduct together with companies and Member States setting out more specific commitments from companies in terms of e.g. specific timeframes for removal of content as well as proactive measures to avoid the dissemination of terrorist content in particular across different platforms. However, the reliance on commitments from companies, the limited outreach (only a small fraction of affected service providers have engaged in the EUIF), the limited level of progress when compared to the scale of the problem, developments and the need to significantly reduce accessibility to terrorist content as a matter of urgency reveal the limits of the voluntary approach. A further continuation or strengthening of the voluntary approach was therefore discarded as not being sufficient for tackling terrorist content.

4.Revising the E-Commerce Directive: the revision of the framework conditions set in the E-Commerce Directive for the obligations on information society services in the removal of illegal content could create further legal incentives for hosting service providers to remove terrorist content, but would not address the objectives of the intervention, nor would this address convincingly and proportionately the drivers of the problem. The balance achieved by the current provisions in protecting fundamental rights, in particular the freedom to conduct a business and freedom of expression, would also be jeopardised. In addition, the measure would not be aligned with the Digital Single Market’s Strategy concerning online platforms, as presented in the Communication on online platforms , and would be disproportionate as to the small share of hosting services abused by terrorist content. 5.Amending the Terrorism Directive: a targeted amendment of Article 21 to further specify obligations on Member States in relation to the removal of terrorist content online could lay down more specific rules on procedures to request removal of terrorist content including possibly an extension of the type of activities/material that should be covered by such obligations (i.e. extension from currently only incitement to terrorism to other offences). However, the focus on criminalisation of terrorist offences as opposed to purely preventative measures, the geographical limitations and limitations in terms of safeguards and other flanking measures (which would not have been possible under this legal basis) would have meant that the initiative would have had limited impact on the objectives of preventing the dissemination of terrorist content under harmonised rules for the cross-border provision of services in the Internal Market and thereby increasing trust in the Digital Single Market. An amendment of the Terrorism Directive was therefore discarded given the limited impact on the overall objective. 5.3.Intervention logic and options retained The abuse of hosting service providers for spreading terrorist content online follows a specific pattern, outlined in Section 2.3, involving a range of hosting service providers of different size, audience and reach. Although voluntary processes under the EU Internet Forum have led to demonstrable progress amongst those companies who participate, it continues to be easy for terrorists to spread content online. Only 14 companies out of 33 contacted in the context of the EU Internet Forum cooperated with the reporting following the Recommendation, while Europol has identified more than 150 hosting service providers as being particularly exposed to terrorist content. The scale and pace of progress among hosting service providers as a whole is not sufficient to adequately address the rapidly evolving problem. At the same time, the likelihood of new attacks in the EU by jihadist terrorists remains high , and online propaganda and networking via social media continue to be essential means by which to recruit and radicalise terrorists. In almost every attack that has occurred, the perpetrators have either used the internet by way of preparation or even in the delivery of the attack, and supporters have glorified the acts in the aftermath and urged others to follow suit. In light of the urgency to act against the spread of terrorist content online , as well as the limits of the voluntary processes, the retained options are all of a legally binding nature. They build on the evolving baseline and additional supporting measures, and include a series of building blocks – with different intensities of intervention – to tackle the problem drivers and pursue the specific objectives. The intervention logic presents how the building blocks included in the options address the specific objectives and the problem drivers is based on: Provisions to harmonise the procedures for removal or disabling access to terrorist content following a removal order from a national authority (giving effect to Article 14 (3) of the E-Commerce Directive). To enable the procedures, harmonisation further includes a common definition of terrorist content online, as well as clarity concerning judicial redress available to hosting service providers and content providers against removal orders. These provisions would limit further legal fragmentation and reduce costs of operations for hosting service providers, not only in line of a standardised process they are exposed to when receiving removal orders, but also in relying on accurate and clear orders, allowing for swift removal. The quality of removal orders and the smooth process would also increase the overall effectiveness in the removal of terrorist content. Provisions to ensure transparent processes and reporting to authorities and the Commission, would increase the accountability and trust in the content moderation process, and would support policy-makers and national authorities in combatting terrorist content and would allow users to better understand how hosting service providers apply their content management policies. Cooperation across national authorities and Europol would improve their ability to act collectively against terrorist content, avoiding duplication, and would reduce complexity and costs on hosting service providers in interacting with national authorities when offering their services cross-border. In addition, provisions to ensure that, in those cases where companies are exposed to terrorist content, the hosting service providers put in place appropriate and proportionate measures to proactively detect terrorist content. Safeguards and provisions to ensure that measures taken to detect and remove terrorist content do not lead to erroneous over-removal of legal content and comply with fundamental rights. Provisions to ensure that measures are enforceable, including the establishment of legal representatives for non-EU companies, establishing points of contact and ensuring Member States have a coherent set of sanctions in place. The figure below summarises the intervention logic. 5.4.Description of the policy options The report presents and assesses three policy options. The common elements, as well as the main differences are presented below. Table 2 presents in a comparative format the respective measures under each option

Main measures under all options All options have in common the creation of a new, harmonised system of removal orders for terrorist content online from national authorities, upon which hosting services must act within one hour. This system is based on a clear legal definition or what constitutes terrorist content. Such a definition would be aligned as closely as possible to the relevant provisions of the Directive on Combatting Terrorism. The assessment of the content would need to account for factors such as the nature and wording of the statements, the context in which the statements were made including whether it is disseminated for educational, journalistic or research purposes and the potential to lead to harmful consequences. These orders should not require any assessment on the part of the hosting service providers, but would be subject to judicial redress. Safeguards, notably information to content providers, complaint procedures and judicial redress as well as other provisions to ensure that measures taken to detect and remove terrorist content do not lead to erroneous over-removal of legal content to ensure compliance with fundamental rights are also common to all options, including a general due diligence requirement to avoid erroneous removal of legal material. Further, all options have in common a number of reporting obligations in the form of publishing transparency reports, reporting on the measures taken to Member States and the Commission, as well as notifying authorities of suspected criminal offences. In addition, cooperation obligations between national authorities, hosting service providers, and where relevant Europol are also foreseen. The scope of these obligations in all three options would focus on all hosting service providers (personal scope) established in the EU and in third countries - insofar as they offer their services in the Union (geographic scope) - based on the above-mentioned definition of what constitutes terrorist content (material scope). Given the nature of the problem and the need to avoid the abuse of smaller platforms, no exemptions are foreseen for SMEs under any of the options. All options would foresee sanctions as well as requirements to establish a legal representative in the EU for companies established outside the EU. Such a legal representative would ensure enforceability of EU rules for hosting services established outside the EU. Main differences The main differences between the three options are as follows: Material scope -Option 1 would limit the material scope to content disseminated to directly incite to commit a terrorist act, following a narrow definition. -Options 2 and 3 would cover material disseminated to incite, recruit and train for terrorism closely linked to the terrorist offences as stipulated in the Terrorism Directive. Harmonisation of procedures governing referrals Option 1 would not regulate Referrals in a legal text, while in options 2 would introduce obligations for hosting service providers to have procedures in place for treating referrals received from Europol and option 3 would additionally include those of Member States. Furthermore feedback on action following referrals would need to be provided to Europol (option 2 and 3) and MS (option 3). This would not entail a requirement for take down. Requirements to Member States: -Options 1 and 2 require Member States to designate competent authorities -Option 3 requires Member States to establish competent authorities with capacity to detect and notify terrorist content -To ensure coordination and avoid duplication of removal orders, in options 2 and 3 Member States would be obliged to inform, coordinate and cooperate with each other with regards to removal orders (option 2 and 3), but also referrals (option 3), and would have the possibility to channel removal orders and referrals via Europol. Proactive measures Under all three options, provisions related to proactive measures would only apply to a subset of hosting service providers, i.e. those exposed to terrorist content, based on objective criteria. Estimates situate the number of hosting services concerned in between 150 and 400 . The scope of specific proactive measures ranges across the three options, as follows:

-Option 1, where a risk assessment is required, as a basis for potential voluntary measures; -Option 2, where hosting service providers are required to draw up an action plan to tackle terrorist content, proportionate to the level of risk, which may include the deployment of automated tools for the prevention of re-upload of already removed and identified material; -Option 3 includes more comprehensive proactive measures requiring service providers to detect also new material, in all instances the scope being proportionate to the level of exposure to terrorist material as well as the economic capacities of the service provider, where appropriate. Data preservation requirements In addition to the reporting obligations for suspected criminal offences, included in all options, option 3 would require service providers to ensure data preservation as a safeguard in cases of erroneous removal as well as to ensure the existence of evidence for any potential criminal investigations, with the appropriate safeguards to comply with the obligations under the General Data Protection Regulation Supporting measures In addition to the legal provisions, either one of the three legislative options would be accompanied by a series of supporting measures, in particular to facilitate cooperation across national authorities and Europol, as well as the collaboration with hosting service providers, and R&D&I support for development and take-up of technological solutions. In particular, near-real-time information transmission across authorities on removal orders and referrals would bring further efficiency for the authorities and limit burden on HSPs in dealing with such notices. A cooperation platform could be further developed and managed by Europol, in line with its mandate. Further functionalities could lead to the creation of a publicly run, high-accountability automatic system to further enhance the expeditious takedown of terrorist content and curb its dissemination across platforms. This electronic infrastructure would be accessible to companies on a voluntary basis. Additional awareness-raising instruments for SMEs could be deployed following the adoption of the legal instrument, as well as specific training and information from Europol, in collaboration with the national authorities. Stakeholder views: In a targeted questionnaire addressed to the Member States, to which the Commission received 19 responses, the Member States considered the following measures as necessary or very necessary: access to content removed by companies for law enforcement purposes, reporting by companies, a common definition of terrorist content, a sanction mechanism, points of contact established by companies, tools for companies to detect content already identified as terrorist material, requirements for action upon referral and for proactive measures by companies, safeguards, transparency reports, requirement for Member States to establish referral capabilities. At the same time, they highlighted risks that need to be taken into account: burden of creating oversight and dispute resolution mechanisms, the need to adapt measures to changing modus operandi of terrorists and evolution of technology, the need to address companies on a global level, need to balance fundamental rights concerns. Timeframes for removal were seen as important during the targeted consultations with Member States, in which some noted that a one hour removal time was seen essential, especially on content which is referred by a competent authority. Additionally, in replies to the open public consultation non-for-profit organisations identifying and reporting allegedly illegal content online noted that the setting of time limits to processing referrals and notifications from trusted flaggers would be useful to support cooperation between hosting services providers and trusted flaggers. While many stakeholders in the OPC favoured the continuation of the baseline, they pointed the need of ensuring any intervention of a legislative nature should take into account the specificities of different types of content. Some of the consulted companies highlighted that they see value in an unambiguous definition of terrorist content across the EU. In line with answers to the OPC, many companies were not in favour of further obligations, however one noted that it would be positive, as long as it is proportionate to the size of the platform. Concerns were voiced over strict timelines should the content be assessed and reviewed prior to removal. On proactive measures a number of concerns were expressed as to the feasibility of these measures for smaller companies and noted the additional burden of safeguards which need to be put in place. Nomination of point of contact, transparency and reporting were overall seen as non-problematic albeit some noted the need for flexibility. For a more detailed overview of the content of the three options, these are presented in more detail in Table 2 below. Table 2 Terrorism -specific legislative options: overview of alternative scenarios Action Option 1 Option 2 Option 3 Scope of Application Definition of terrorist content Narrow definition material disseminated with purpose to directly incite a terrorist act produced by an EU listed terrorist organisations Comprehensive definition material disseminated with purpose to incite, recruit and train for terrorism, including in particular material produced by EU listed terrorist organisations Comprehensive definition material disseminated with purpose to incite, recruit and train for terrorism, including in particular material produced by EU listed terrorist organisations Material scope: Hosting Service Providers ü ü ü Geographic scope: companies directing/offering services to EU citizens ü ü ü Removal of terrorist content Action upon removal order Requirement to have measures, procedures and tools in place that enable the HSP to remove within 1h Designation of competent authorities but no obligation to establish such entities Feedback from companies on action taken Requirement to have measures, procedures and tools in place that enable the HSP to remove within 1h Designation of competent authorities but no obligation to establish such entities Feedback from companies on action taken Requirement to have measures, procedures and tools in place that enable the HSP to remove within 1h MS to establish competent authorities with capacity to detect and notify terrorist content Feedback from companies on action taken Action upon referrals Requirement to have procedures in place to make an assessment on referrals from Europol Such an obligation would be combined with more specific requirements to ensure the quality of referrals Feedback from companies on action taken Requirement to have procedures in place to make an assessment on referrals from Europol and Member States. Such an obligation would be combined with more specific requirements to ensure the quality of referrals Feedback from companies on action taken Proactive measures (applicable to companies exposed to terrorist content) Requirement to carry out a standardised risk assessment, in cooperation with Member States and where appropriate Europol. Risk assessment would include (1) threat of terrorist activity, known intent and perceived attractiveness of the HSP (users/viewers, business model) (2) vulnerability (i.e. effectiveness of measures already in place) (3) identification of the level of risk. Establishment of a remedial action plan is not mandatory but can be established with the involvement of the authorities. Obligation to present a remedial action plan which should contain appropriate measures, proportionate to the level of risk and the specific circumstances of the company, to prevent the dissemination of terrorist content on their services. Such measures may include, where appropriate automated means to detect, identify and expeditiously remove terrorist content which has already been removed following a removal order or referrals, as well as to immediately prevent content providers from re-submitting such content, in cooperation with competent authorities and Europol. Obligation to identify and put in place measures, proportionate to the level of risk and the specific circumstances of the company to prevent the dissemination of terrorist content on their services. Such measures shall include, where appropriate automated means to detect, identify and expeditiously remove terrorist content: 1) which has already been removed following a removal order or referrals, as well as to immediately prevent content providers from re-submitting such content and, as appropriate 2) reliable technical means, such as automatic detection technologies, to detect and prevent the appearance of new terrorist content, in cooperation with competent authorities and Europol. Role of Europol and cooperation with law enforcement Cooperation between national authorities (and HSPs) and Europol Points of contact in HSPs for the purpose of receiving legal orders Points of contact in HSPs for the purpose of receiving legal orders and Europol referrals Requirement for MS to inform, cooperate and coordinate with each other. MS may inform Europol of the removal orders and action taken by HSPs Points of contact in HSPs for the purpose of receiving legal orders and Europol referrals Requirement for MS to inform, cooperate and coordinate with each other. MS may channel removal orders through Europol HSPs may channel their feedback on removal orders through Europol Requirement to maintain accessibility of terrorist content for law enforcement purposes Reporting obligation for HSPs on suspected criminal offences Reporting obligation for HSPs on suspected criminal offences Retention of terrorist content for law enforcement purposes Reporting obligation for HSPs on suspected criminal offences Safeguards, transparency and accountability Due diligence requirements for HSPs to avoid erroneous removal of legal content ü ü ü Complaint procedures and judicial redress for HSPs, content providers ü ü ü Transparency reports of HSPs ü ü ü Reporting to the Commission Reporting requirements on implementation of the obligations, including information on proactive measures taken voluntarily by the company Member States to report to the Commission on number of legal orders and referrals sent Reporting requirements on the implementation of the obligations, including information on proactive measures taken by the company as part of the remedial action plan Member States to report to the Commission on number of legal orders and referrals sent Reporting requirements on the implementation of the obligations, including information on proactive measures by the company Member States to report to the Commission on number of legal orders and referrals sent Enforcement Sanctions Member States to establish regimes under national law including designation of relevant authority Determination of rules on jurisdiction ü ü ü Requirement to establish a legal representative (for companies established outside the EU) ü ü ü 6.What are the impacts of the policy options?

All the retained options have been assessed against the following economic and societal impacts, in addition to other specific impacts on fundamental rights, presented in the next section. Table 3 summarises the assessment. There are no significant environmental impacts to be assessed in the context of this initiative. Impacts Options 1 2 3 Economic impacts Functioning of the Internal Market ++ + + + + Impact on competition - - - Competitiveness and effect on SMEs (costs / support) -/ + - / + - - / + Technological development and the digital market (costs / incentive) -/ + -/ + -/ + Public authorities (costs / effectiveness) - / + -/+ -/++ Social impacts Crime, terrorism and security + ++ +++ Safety of Internet users + ++ +++ Governance and good administration + + ++ Impacts on i