EUROPEAN COMMISSION Brussels, 21.4.2021 SWD(2021) 84 final COMMISSION STAFF WORKING DOCUMENT



IMPACT ASSESSMENT

Accompanying the Proposal for a Regulation of the European Parliament and of the Council LAYING DOWN HARMONISED RULES ON ARTIFICIAL INTELLIGENCE (ARTIFICIAL INTELLIGENCE ACT) AND AMENDING CERTAIN UNION LEGISLATIVE ACTS























{COM(2021) 206 final} - {SEC(2021) 167 final} - {SWD(2021) 85 final}



Table of contents 1.Introduction: technological, socio-economic, legal and political context 1.1. Technological context 1.2. Socio-economic context 1.3. Legal context 1.3.1. Relevant fundamental rights legislation 1.3.2. Relevant product safety legislation 1.3.3. Relevant liability legislation 1.4. Political context 1.5. Scope of the impact assessment 2.Problem definition 2.1. What are the problems? 2.2. What are the main problem drivers? 2.3. How will the problem evolve? 3.WHY SHOULD THE EU ACT? 3.1. Legal basis 3.2. Subsidiarity: Necessity of EU action 3.3. Subsidiarity: Added value of EU action 4.Objectives: What is to be achieved?

4.1.General objectives 4.2.Specific objectives 4.3. Objectives tree/intervention logic. 5. What are the available policy options? 5.1. What is the baseline from which options are assessed? 5.2. Option 1: EU legislative instrument setting up a voluntary labelling scheme 5.3. Option 2: A sectoral, ‘ad-hoc’ approach 5.4. Option 3: Horizontal EU legislative instrument establishing mandatory requirements for high-risk AI applications 5.5. Option 3+: Horizontal EU legislative instrument establishing mandatory requirements for high-risk AI applications + co-regulation through codes of conduct for non-high risk applications 5.6. Option 4: Horizontal EU legislative instrument establishing mandatory requirements for all AI applications, irrespective of the risk they pose 5.7. Options discarded at an early stage 6.What are the impacts of the policy options? 6.1. Economic impacts 6.1.1.Functioning of the internal market 6.1.2.Impact on uptake of AI 6.1.3.Costs and administrative burdens 6.1.4.SME test 6.1.5.Competitiveness and innovation 6.2. Costs for public authorities 6.3. Social impact

6.4. Impacts on safety 6.5. Impacts on fundamental rights 6.6. Environmental impacts 7.How do the options compare? 7.1. Criteria for comparison 7.2. Achievement of specific objectives 7.2.1. First specific objective: Ensure that AI systems placed on the market and used are safe and respect fundamental rights and Union values 7.2.2.Second specific objective: Ensure legal certainty to facilitate investment and innovation 7.2.3.Third specific objective: Enhance governance and effective enforcement of fundamental rights and safety requirements applicable to AI systems 7.2.4.Fourth specific objective: Facilitate the development of a single market for lawful, safe and trustworthy AI applications and prevent market fragmentation 7.3. Efficiency 7.4. Coherence 7.5.Proportionality 8.Preferred option 9.How will actual impacts be monitored and evaluated? 1.Introduction: technological, socio-economic, legal and political context As part of the Commission’s overarching agenda of making Europe ready for the digital age, the Commission is undertaking considerable work on Artificial Intelligence (AI). The overall EU strategy proposed in the White Paper on AI proposes an ecosystem of excellence and trust for AI. The concept of an ecosystem of excellence in Europe refers to measures which support research, foster collaboration between Member States and increase investment into AI development and deployment. The ecosystem of trust is based on EU values and fundamental rights, and foresees robust requirements that would give citizens the confidence to embrace AI-based solutions, while encouraging businesses to develop them. The European approach for AI aims to promote Europe’s innovation capacity in the area of AI, while supporting the development and uptake of ethical and trustworthy AI across the EU economy. AI should work for people and be a force for good in society. The development of an ecosystem of trust is intended as a comprehensive package of measures to address problems posed by the introduction and use of AI. In accordance with the White Paper and the Commission Work Programme, the EU plans to adopt a set of three inter-related initiatives related to AI: (1)European legal framework for AI to address fundamental rights and safety risks specific to the AI systems (Q2 2021);

(2)EU rules to address liability issues related to new technologies, including AI systems (Q4 2021-Q1 2022); (3)Revision of sectoral safety legislation (e.g. Machinery directive, Q1 2021, General Product Safety Directive, Q2 2021). These three initiatives would be complementary and their adoption will proceed in stages. Firstly, as entrusted by the European Council, requested by the European Parliament and supported by the results of the public consultation on the White Paper on AI, the European Commission will adopt European legal framework for AI. This legal framework should set the ground for other forthcoming initiatives by providing: (1) a definition of an AI system; (2) a definition of ‘high risk’ AI system, and (3) common rules to ensure that AI systems placed or put into service in the Union market are trustworthy. The introduction of the European legal framework for AI will be supplemented with revisions of the sectoral safety legislation and changes to the liability rules. This staged, step-by-step and complementary approach to regulate AI aims to ensure regulatory coherence throughout the Union, therefore contributing to legal certainty for developers and users of AI systems and citizens. More details on the scope of the existing safety and liability legislation are discussed in section 1.3. (legal context) and the interaction between the three initiatives are presented in section 8 (preferred option). This impact assessment focuses on the first AI initiative, the European legal framework for AI. The purpose of this document is to assess the case for action, the objectives, and the impact of different policy options for a European framework for AI, as envisaged by the 2020 Commission work programme. The Proposal for a European legal framework for AI and this impact assessment build on two years of analysis of evidence and involvement of stakeholders, including academics, businesses, non-governmental organisations, Member States and citizens. The preparatory work started in 2018 with the setting up of a High-Level Expert Group on AI (HLEG) which had an inclusive and broad composition of 52 well-known experts tasked to advise the Commission on the implementation of the Commission’s Strategy on Artificial Intelligence. In April 2019, the Commissioned welcomed the key requirements set out in the HLEG ethics guidelines for Trustworthy AI, which had been revised to take into account more than 500 submissions from stakeholders. The Assessment List for Trustworthy Artificial Intelligence (ALTAI) made these requirements operational in a piloting process with over 350 organisations. The White Paper on Artificial Intelligence further developed this approach, inciting comments from more than 1250 stakeholders. As a result, the Commission published an Inception Impact Assessment that in turn attracted more than 130 comments. Additional stakeholder workshops and events were also organised, the results of which support the analysis and the proposals made in this impact assessment. 1.1.Technological context Today, AI is one of the most vibrant domains in scientific research and innovation investment around the world. Approaches and techniques differ according to fields, but overall AI is best defined as an emerging general-purpose technology: a very powerful family of computer programming techniques that can be deployed for desirable uses, as well as more harmful ones. The precise definition of AI is highly contested. In 2019, the Organisation for Economic Co-operation and Development (OECD) adopted the following definition of an AI system: ‘An AI system is a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy.’ The OECD Report on Artificial Intelligence in Society provides a further explanation on what an AI system is. An AI system, also referred to as ‘intelligent agent’, “consists of three main elements: sensors, operational logic and actuators. Sensors collect raw data from the environment, while actuators act to change the state of the environment. Sensors and actuators are either machines or humans. The key power of an AI system resides in its operational logic. For a given set of objectives and based on input data from sensors, the operational logic provides output for the actuators. These take the form of recommendations, predictions or decisions that can influence the state of the environment.” Figure 1: A high-level conceptual view of an AI system Source: OECD, Report – Artificial Intelligence in Society, p.23. AI systems are typically software-based, but often also embedded in hardware-software systems. Traditionally AI systems have focused on ‘rule-based algorithms’ able to perform complex tasks by automatically executing rules encoded by their programmers. However, recent developments of AI technologies have increasingly been on so called ‘learning algorithms’. In order to successfully ‘learn’, many machine learning systems require substantial computational power and availability of large datasets (‘big data’). This is why, among other reasons, despite the development of ‘machine learning’ (ML), AI scientists continue to combine traditional rule-based algorithms and ‘new’ learning based AI techniques. As a result, the AI systems currently in use often include both rule-based and learning-based algorithms. 1.2. Socio-economic context The use of AI systems leads to important breakthroughs in a number of domains. By improving prediction, optimising operations and resource allocation, and personalizing service delivery, the use of AI can support socially and environmentally beneficial outcomes and provide key competitive advantages to companies. The use of AI systems in healthcare, farming, education, infrastructure management, energy, transport and logistics, public services, security, and climate change mitigation, can help solve complex problems for the public good. Combined with robotics and the Internet of Things (IoT), AI systems are increasingly acquiring the potential to carry out complex tasks that go far beyond human capacity. A recent Nature article found that AI systems could enable the accomplishment of 134 targets across all the Sustainable Development Goals, including finding solutions to global climate problems, reducing poverty, improving health and the quality and access to education, and making our cities safer and greener. In the ongoing Covid-19 pandemic, AI systems are being used, for example, in the quest for vaccines, in disease detection via pattern recognition using medical imagery, in calculating probabilities of infection, or in emergency response with robots replacing humans for high-exposure tasks in hospitals. This example alone already indicates the breadth of possible benefits of AI systems. Other practical applications further show how citizens can reap a lot of benefits whern accessing improved services such as personalised telemedicine care, personalised tutoring tailored to each student, or enhanced security through applications that ensure more efficient protection against cybersecurity risks. The successful uptake of AI technologies also has the potential to accelerate Europe’s economic growth and global competitiveness. McKinsey Global Institute estimated that by 2030 AI technologies could contribute to about 16% higher cumulative global gross domestic product (GDP) compared with 2018, or about 1.2% additional GDP growth per year. AI systems and the new business models they enable are progressively developing to at-scale deployment. Accordingly, those AI systems will increasingly impact all sectors of the economy. The International Data Corporation AI market development forecast suggests that global revenues for the AI market are expected to double and surpass USD 300 billion by as early as 2024. Many businesses in various sectors of the EU economy are already seizing these opportunities. In addition to the ICT sector, the sectors using AI most intensively are education, health, social work and manufacturing. However, Europe is home to only 3 of the top 25 AI clusters worldwide and has only a third as many AI companies per million employees as the US.

Table 1: AI technologies adopted in European businesses AI Technologies Currently use it Plan to use it Process or equipment optimisation 13% 11% Anomaly detection 13% 7% Process automation 12% 11% Forecasting, price-optimisation and decision-making 10% 10% Natural language processing 10% 8% Autonomous machines 9% 7% Computer vision 9% 7% Recommendation/personalisation engines n/a 7% Creative and experimentation activities 7% 4% Sentiment analysis 3% 3% Source: Ipsos Survey, 2020 The same elements and techniques that power socio-economic benefits of AI systems can also bring about risks or negative consequences for individuals or for society as a whole. For example, deployment of AI systems may be intentionally used by a developer or an operator to deceive or manipulate human choices, or altogether disable human agency, control and intermediation. This possible use of AI could have strong negative consequences for the protection of fundamental rights and for human safety. In the world of work, AI systems could also undermine the effective enforcement of labour and social rights. In light of the speed of technological change and possible challenges, the EU is committed to strive for a balanced approach. European Commission President von der Leyen stated: ‘In order to release that potential we have to find our European way, balancing the flow and wide use of data while preserving high privacy, security, safety and ethical standards.’ 1.3. Legal context European Union law does not have a specific legal framework for AI. Thus, as it currently stands, EU law does not provide for a definition of an AI system, nor for horizontal rules related to the classification of risks related to AI technologies. The development and uptake of AI systems more broadly, as outlined in this section, takes place in the context of the existing body of EU law that provides non-AI specific principles and rules on protection of fundamental rights, product safety, services or liability issues. 1.3.1. Relevant fundamental rights legislation The Union is founded on the values of human dignity and respect of human rights that are further specified in the EU Charter of Fundamental Rights (the Charter). The provisions of the Charter are addressed to the institutions and bodies of the Union and to the Member States only when they are implementing Union law. Some fundamental rights obligations are further provided for in EU secondary legislation, including in the field of data protection, non-discrimination and consumer protection. This body of EU secondary legislation is applicable to both public and private actors whenever they are using AI technology. In this context, the EU acquis on data protection is particularly relevant. The General Data Protection Regulation and the Law Enforcement Directive aim to protect the fundamental rights and freedoms of natural persons, and in particular their right to the protection of personal data, whenever their personal data are processed. This covers the processing of personal data through ‘partially or solely automated means’, including any AI system. Users that determine the purpose and means of the AI processing (‘data controllers’) have to comply with a number of data processing principles such as lawfulness, transparency, fairness, accuracy, data minimization, purpose and storage limitation, confidentiality and accountability. On the other hand, natural persons, whose personal data are processed, have a number of rights, for instance, the right to access, correction, not to be subject to solely automated decision-making with legal or similarly significant effects unless specific conditions apply. Stricter conditions also apply for the processing of sensitive data, including biometric data for identification purposes, while processing that poses high risk to natural persons’ rights and freedoms requires a data protection impact assessment. Users of AI systems are also bound by existing equality directives. The EU equality acquis prohibits discrimination based on a number of protected grounds (such as racial and ethnic origin, religion, sex, age, disability and sexual orientation) and in specific context and sectors (for example, employment, education, social protection, access to goods and services). This existing acquis has been complemented with the new EU Accessibility Act setting requirements for the accessibility of goods and services, to become applicable as of 2025. Consumer protection law and obligations to abstain from any unfair commercial practices listed in the Unfair Commercial Practice Directive are also highly relevant for businesses using AI systems. Furthermore, EU secondary law in the area areas of asylum, migration, judicial cooperation in criminal matters, financial services and online platforms is also relevant from a fundamental rights perspective when AI is developed and used in these specific contexts. 1.3.2. Relevant product safety legislation In addition, there is a solid body of EU secondary law on product safety. The EU safety legislation aims to ensure that only safe products are placed on the Union market. The overall EU architecture on safety is based on the combination of horizontal and sectoral rules. This includes the General Product Safety Directive (GPSD) applicable to consumer products, insofar there are not more specific provisions in harmonised sector-specific safety legislation, as for example, the Machinery Directive (MD), the Medical Device Regulation (MDR) and the EU framework on the approval and market surveillance of motor vehicles (in particular Vehicle Safety Regulation). Reviews of both the MD and the GPSD are currently under way. Those reviews aim to respond, among other things, to the challenges of new technologies, such as IoT, robotics and AI. In addition, delegated acts are expected to be soon adopted by the Commission under the Radio Equipment Directive to enact certain new requirements on data protection and privacy, cybersecurity and harm to the network. Moreover, in the automotive sector new rules on automated vehicles, cybersecurity and software updates of vehicles will become applicable as part of the vehicle type approval and market surveillance legislation from 7 July 2022. While the European identifies the review of the General Product Safety Directive, the Machinery Directive and the Radio Equipment Directive as priorities, other pieces of product legislation may well be updated in the future in order to address existing gaps linked to new technologies. The product safety legislation is technology-neutral and focuses on the safety of the final product as a whole. The revisions of the product safety legislation do not have the objective to regulate AI as such, but aim primarily at ensuring that the integration of AI systems into the overall product will not render a product unsafe and the compliance with the sectoral rules will not be affected. 1.3.3. Relevant liability legislation

Safety legislation sets rules to ensure that products are safe and safety risks are addressed, nevertheless, damages can still occur. For that purpose, the liability rules at national and EU level complement the safety legislation and determine which party is liable for harm, and under which conditions a victim can be compensated. A longstanding approach within the EU with regard to product legislation is based on a combination of both safety and liability rules. In practice, while being driven by different regulatory rationales and objectives, safety and liability initiatives are essential and complementary in nature. At EU level, the Product Liability Directive (PLD) is currently the only EU legal framework that harmonizes part of national liability law, introducing a system of ‘strict’ liability without fault of the producer for physical or material damage caused by a defect in products placed on the Union market. While the PLD provides legal certainty and uniform consumer protection as a safety net applicable to all products, its rules also face increasing challenges posed by emerging technologies, including AI. As part of the upcoming revision, the Commission will be exploring several options to adapt the EU product liability rules to the digital world, for instance by adapting the definition of product and producer, or by extending the application of the strict liability regime to certain other types of damages (e.g. to privacy or personal data). The Commission will also explore options on how to address the current imbalance between consumers and producers by reversal or alleviation of the burden of proof (with access to information and presumption of defectiveness under certain circumstances), and explore the abolishment of existing timelines and threshold (€500). At national level, non-harmonised civil liability frameworks complement these Union rules by ensuring compensation for damages from products and services and by addressing different liable persons. National liability systems usually include fault-based and strict liability regimes. In order to ensure that victims who suffer damage to their life, health or property as a result of AI technologies have access to the same compensation as victims of other technologies, the Commission has announced possible revision of rules on liability. The main objective of the revision is to ensure that damages caused by AI systems are covered. In order to achieve this objective, together with the update of the PLD, the Commission is also considering possible new AI-specific rules harmonising certain aspects of national civil liability frameworks with regard to the liability for certain AI systems. In particular, options which are currently under evaluation include the possible setting of strict liability for AI operators, possibly combined with mandatory insurance for AI applications with a specific risk profile as well as adaptation of burden of proof concerning causation and fault for all other AI applications. In addition to the general review of liability rules, the Commission is examining liability challenges which are specific to certain sectors, such as health-care, and which may deserve specific considerations. The relationship between the proposed European legal framework for AI analysed in this impact assessment and the forthcoming new rules on liability is further discussed under the preferred option in section 8. In terms of timing for the adoption of the new rules on liability, the Commission decided for a staged approach. First, the Commission will propose in Q2 2021 the AI horizontal framework (the current initiative) and then, the EU rules to address liability issues related to new technologies, including AI systems (expected Q4 2021 – Q1 2022). The future changes to the liability rules will take into account the elements of the horizontal framework with a view to designing the most effective and proportionate solutions with regard to liability. Moreover, compliance with the requirements of the AI horizontal framework will be taken into account for assessing liability of actors under future liability rules. With regard to intermediary liability, for example when sellers place faulty products through online marketplaces, the E-Commerce Directive regulates the liability exemptions for online intermediaries. This framework is currently updated in the Commission’s proposal for a Digital Services Act. 1.3.4. Relevant legislation on services The EU also has a comprehensive legal framework on services that is applicable whenever AI software is provided as a stand-alone service or integrated into other services, including in particular digital services, audio-visual media services, financial services, transport services, professional services and others. In the context of information society services, the e-Commerce Directive provides for an applicable regulatory framework laying down horizontal rules for provisions of such services in the Union. The proposed Digital Services Act includes rules on liability exemption for providers of intermediary services (i.e. mere conduit; caching; hosting), which are to date contained in the e-Commerce Directive that remains largely unchanged and fully applicable. At the same time, the Digital Services Act introduces due diligence obligations for providers of intermediary services so as to keep users safe from illegal goods, content or services and to protect their fundamental rights online. These due diligence obligations are adapted to the type and nature of the intermediary service concerned and transparency and accountability rules will apply for algorithmic systems, including those based on AI, used by online platforms. In the field of financial services, the risk governance requirements under the existing legislation provide a strong regulatory and supervisory framework for assessment and management of risks. Specific rules additionally apply in relation to trading algorithms. With respect to creditworthiness assessment, European Banking Authority guidelines have been recently adopted to improve regulated financial institutions’ practices and associated governance arrangements, processes and mechanisms in relation to credit granting, management and monitoring, including when using automated models in the creditworthiness assessment and credit decision-making processes. However, no sector-specific EU guidance or rules currently apply to non-regulated entities when assessing the creditworthiness of consumers. 1.4. Political context To address the opportunities and challenges of AI, in April 2018 the European Commission put forward a European approach to AI in its Communication “Artificial Intelligence for Europe.” In June 2018, the Commission appointed the High-Level Expert Group on Artificial Intelligence, which produced two deliverables: the Ethics guidelines for trustworthy AI and the Policy and investment recommendations for trustworthy AI. In December 2018, the Commission presented a Coordinated Plan on AI with Member States to foster the development and use of AI. In June 2019, in its Communication “Building Trust in Human-Centric Artificial Intelligence”, the Commission endorsed the seven key requirements for Trustworthy AI identified by the HLEG. After extensive consultation, on 17 July 2020, the HLEG published an Assessment List for Trustworthy Artificial Intelligence for self-assessment (ALTAI) which was tested by over 350 organisations. In February 2020, the European Commission published a White Paper on AI setting out policy options for a regulatory and investment oriented approach. It was accompanied by a Commission Report on the safety and liability implications of AI. The White Paper opened a wide public consultation where more than 1 215 contributions were received from a wide variety of stakeholders, including representatives from industry, academia, public authorities, international organisations, standardisation bodies, civil society organisations and citizens. This clearly showed the great interest from stakeholders around the globe in shaping the future EU regulatory approach to AI, as assessed in this impact assessment. The European Council and the European Parliament (EP) also repeatedly called for the Commission to take legislative action to ensure a well-functioning internal market for AI systems where both benefits and risks are adequately addressed at EU level. In 2017, the European Council called for a ‘sense of urgency to address emerging trends’ including ‘issues such as artificial intelligence …, while at the same time ensuring a high level of data protection, digital rights and ethical standards’. In its 2019 Conclusions on the Coordinated Plan on the development and use of artificial intelligence Made in Europe, the Council further highlighted the importance of ensuring that European citizen's rights are fully respected and called for a review of the existing relevant legislation to make it fit for purpose for the new opportunities and challenges raised by AI. The European Council has also called for a clear determination of what should be considered as high-risk AI applications. The most recent Conclusions from 21 October 2020 further called for addressing the opacity, complexity, bias, a certain degree of unpredictability and partially autonomous behaviour of certain AI systems, to ensure their compatibility with fundamental rights and to facilitate the enforcement of legal rules. In 2017, the European Parliament (EP) adopted a Resolution on Civil Law Rules on Robotics urging the Commission to analyse the impact of use of AI technologies in the main areas of EU legislative concern, including ethics, liability, standardisation and institutional coordination and oversight, and to adopt legislation where necessary. In 2019, the EP adopted a Resolution on a Comprehensive European Industrial Policy on Artificial Intelligence and Robotics. In June 2020, the EP also set up a Special Committee on Artificial Intelligence in a Digital Age (AIDA) tasked to analyse the future impact of AI systems in the digital age on the EU economy and orient future EU priorities. In October 2020, the EP adopted a number of resolutions related to AI, including on ethics, liability and copyright. EP resolutions on AI in criminal matters and AI in education, culture and the audio-visual sector are forthcoming. The EP Resolution on a Framework of Ethical Aspects of Artificial Intelligence, Robotics and Related Technologies, specifically recommends to the Commission to propose a legislative action to harness the opportunities and benefits of AI, but also to ensure protection of ethical principles. The EP resolution on internal market aspects of the Digital Services Act presents the challenges identified by the EP as regards AI-driven services. At international level, the ramifications in the use of AI systems and related challenges have also received significant attention. The Council of Europe started work on an international legal framework for the development, design and application of AI, based on the Council of Europe’s standards on human rights, democracy and rule of law. It has also recently issued guidelines and proposed safeguards and certain prohibitions of the use of facial recognition technology considered particularly intrusive and interfering with human rights. The OECD adopted a Council Recommendation on Artificial Intelligence. The G20 adopted human-centred AI Principles that draw on the OECD AI Principles. UNESCO is also starting to develop a global standard setting instrument on AI. Furthermore, the EU, together with many advanced economies, set up the Global Partnership on Artificial Intelligence (GPAI). As part of the EU-Japan Partnership for Sustainable Connectivity and Quality Infrastructure, concluded in September 2019, both sides reconfirmed their intention to continue promoting policies that boost innovation including in Artificial Intelligence, cloud, quantum computing and blockchain. In addition to EU and international initiatives, many countries around the world started to consider adopting their own ethical and accountability frameworks on AI and/or automated decision-making systems. In Canada, a Directive on Automated Decision-Making came into effect on April 1, 2020 and applies to the use of automated decision systems in the public sector that “provide external services and recommendations about a particular client, or whether an application should be approved or denied.” The Directive includes an Algorithmic Impact Assessment and transparency obligations vis-à-vis persons affected by the automated decision. In 2020, the Government of New Zealand, together with the World Economic Forum, was spearheading a multi-stakeholder, policy project, structured around three focus areas: 1) obtaining of a social licence for the use of AI through an inclusive national conversation; 2) the development of in-house understanding of AI to produce well-informed policies; and 3) the effective mitigation of risks associated with AI systems to maximize their benefits. In early 2020, the United States’ government adopted overall regulatory principles. On this basis, the White House released the first-ever guidance for Federal agencies on the regulation of artificial intelligence applications in the public sector that should comply with key principles for Trustworthy AI. Other countries with regulatory initiatives on AI include, for example, Singapore, Japan, Australia, the United Kingdom and China. Annex 5.1 summarises the main ongoing initiatives in these third countries undertaken to address the challenges posed by AI and to harness its potential for good.

1.5. Scope of the impact assessment This report assesses the case for an EU regulatory framework for the development and use of AI systems and examines the impact of different policy options. The use of AI for exclusive military purposes remains outside the scope of the present initiative due to its implications for the Common Foreign and Security Policy (CFSP). Insofar as ‘dual use’ products and technologies have AI features and can be used for both military and civil purposes, these goods will fall into the scope of the current initiative on AI. The forthcoming initiative on liability and the ongoing revisions of sectoral safety legislation are subject to separate initiatives and remain equally outside the scope of this impact assessment, as discussed in section 1.3 above. 2. Problem definition 2.1.What are the problems? The analysis of the available evidence suggests that there are six main related problems triggered by the development and use of AI systems that the current initiative aims to address. Table 2: Main problems MAIN PROBLEMS STAKEHOLDERS CONCERNED 1.Use of AI poses increased risks to safety and security of citizens Citizens, consumers and other victims Affected businesses 2.Use of AI poses increased risk of violations of citizens’ fundamental rights and Union values Citizens, consumers and other victims Whole groups of the society, Users of AI systems liable for fundamental rights violations 3.Authorities do not have powers, procedural frameworks and resources to ensure and monitor compliance of AI development and use with applicable rules National authorities responsible for compliance with safety and fundamental rights rules 4.Legal uncertainty and complexity on how existing rules apply to AI systems dissuade businesses from developing and using AI systems Businesses and other providers developing AI systems Businesses and other users using AI systems 5.Mistrust in AI would slow down AI development in Europe and reduce the global competitiveness of the EU economy Businesses and other users using AI systems Citizens using AI systems or being affected by them 6.Fragmented measures create obstacles for cross-border AI single market and threaten Union’s digital sovereignty Businesses developing AI, mainly SMEs affected Users of AI system, including consumers, businesses and public authorities Problem 1: The use of AI poses increased risks to safety and security of citizens The overall EU architecture of safety frameworks is based on a combination of horizontal and sectoral rules. This includes the horizontal GPSD and sector-specific legislation, as for example, the Machinery Directive. The EU safety legislation has significantly contributed to the high-level of safety of products put into circulation in the EU Single Market. However, it is increasingly confronted with the challenges posed by new technologies, some of which specifically relate to AI technologies. Two main reasons explain the limitations of the existing EU safety and security framework in relation to the application to AI technologies. Firstly, the nature of safety risks caused by AI: The main causes of safety risks generated by complex software or algorithms are qualitatively different from risks caused by physical products. The specific characteristics of AI systems, could lead to safety risks including: ·Biases in data or models: training data can have hidden biases that can be ‘learnt’ by the AI model and reproduced in its outputs. AI algorithms can also introduce biases in their reasoning mechanisms by preferring certain characteristics of the data. For example, a medical imaging recognition device trained on data biased towards a specific segment of population may lead in certain cases to safety risks due to misdiagnosis in patients not well represented by the data. ·Edge cases: unexpected or confusing input data can result in failures due to the limited ability that AI models might exhibit to generalise well from training data. For example, image recognition systems in an autonomous vehicle could malfunction in the case of unexpected road situations, endangering passengers and pedestrians. ·Negative side effects: the realization of a task by an autonomous AI system could lead to harmful effects if the scope of action of the system is not correctly defined and does not consider the context of use and the state of the environment. For instance, an industrial robot with an AI system designed to maximize the work rate in a workshop could damage property or accidentally hurt people in situations not foreseen in its design. AI models which are run on top of ICT infrastructure are composed of a diverse range of digital assets. Therefore, cybersecurity issues affecting this broader digital ecosystem also extend to AI systems and can result in important AI safety risks. This is particularly relevant considering the new systems enabled by AI, such as cyber-physical systems, where cybersecurity risks have direct safety implications. Moreover, AI systems can also be subject to malicious attempts to exploit AI specific vulnerabilities, including: ·Evasion – an attack when an attacker modifies input data, sometimes in an imperceptible manner, so that the AI model cannot correctly identify the input and this leads to wrong outputs. Examples include attacks to evade anti-spam filters, spoofing attacks against biometric verification systems or stickers added to stop signs to make autonomous vehicles to perceive them as speed signs. ·Data poisoning – an action aiming to modify the behaviour of the AI model by altering the training datasets, especially when the data used is scraped from the web, sourced from data exchanges or from open datasets. The ‘learning’ systems where model parameters are constantly updated using new data, are particularly sensitive to data poisoning. For example, poisoning the data used to train a chatbot could make it disclose sensitive information or adopt inappropriate behaviour.

·Model extraction – attacks that aim to build a surrogate system that imitates the targeted AI model. The goal is to get access to a representation of the AI model that will allow the attacker to understand and mimic the logic of the system and possibility build more sophisticated attacks, like evasion or data poisoning or steal sensitive information from training data. ·Backdoor – refers to a typical risk in programming which is not limited to AI applications, but is more difficult to detect and avoid in the case of AI due to its opacity. The presence of a so-called ‘backdoor’ makes unauthorised access to a system possible. AI specific risks are not or are only partly covered by the current Union safety and security legislation. While Union safety legislation covers more generally the risks stemming from software being a safety component (and usually embedded) in a hardware product, stand-alone software (except in the medical device framework) – including when used in services – or software uploaded to a hardware device after this device is placed on the market are not currently covered. Thus, services based on AI technology, such as transport services or infrastructure management, are not covered. Moreover, even when software is covered by EU safety legislation, no specific safety or performance requirements are set for AI systems. For example, there are no AI-technology specific requirements ensuring reliability and safety over its lifecycle. The EU legal framework on cybersecurity applies to ICT products, services and processes, and therefore could also potentially cover AI technologies. However, no scheme for AI currently exists and there are no established AI cybersecurity standards of best practice for developers in the design phase, notably when it comes to ‘security by design’ for AI. Moreover, the certification schemes for cybersecurity are of a voluntary nature. This lack of clear safety provisions covering specific AI risks, both for AI systems being safety components of products and AI systems used in services, can be highly problematic for users and consumers of AI applications. Secondly, the lifecycle of an AI product: Under the current legal framework, ex-ante conformity assessment procedures are mainly conceptualized for products that are ‘stable’ in time after deployment. The current safety legislation does not contain specific provisions for products that are possibly subject to evolution during their lifecycle. Yet, certain AI systems are subject to considerable change after their first deployment on the market. Such potential and unanticipated modifications to the performance of those AI systems after their placement on the market could still occur and cause injuries and physical damage. A product that is already on the market is normally required to undergo a new conformity assessment if there are substantial modifications of the product. However, the exact interpretation of the notion of “substantial modification” in an AI context needs to be clearly defined, also in light of the complexity of the AI value chain, lest it should lead to legal uncertainty. As a general conclusion, the specificities of AI applications might require the establishment of some specific safety requirements to ensure a high level of protection of users and consumers as well as to provide legal certainty for businesses, notably for use cases where the lack of proper performance or reliability of AI could have a severe impact on life and health of individuals. This is regardless of whether AI systems are a safety component of products or are used in services. Stakeholders views: In the Public consultation on White Paper on AI, 83% of all respondents consider that the fact that AI may endanger safety’ is ‘important’ (28%) or ‘very important’ (55%). Among SMEs, 72% found safety to be an important or very important concern, whereas only 12% said it was not important or not important at all. This position was even more pronounced among large businesses, with 83% saying that safety was (very) important and only 4% finding the issue unimportant. 80% of academic and other research institutions and 88% of civil society organisations agreed that safety was a (very) important concern. Among EU citizens, 73% found safety to be an important or very important issue. Of those stakeholders who said safety was not a (very) important concern, 43% were EU citizens (which make up 35% of all respondents) and 20% were SMEs (7%). Problem 2: Use of AI poses an increased risk of citizens’ fundamental rights and Union values violations The use of AI can have a significant impact on virtually all fundamental rights as enshrined in the EU Charter of Fundamental Rights. AI use can be positive, promoting certain rights (e.g. the right to education, health care) and contributing to important public interests (e.g. public security, public health, protection of financial interests). It can also help reduce some adverse impacts on fundamental rights by improving the accuracy or efficiency of decision-making processes and addressing biases, delays or errors in individual human decisions. On the other hand, AI used to replace or support human decision-making or for other activities such as surveillance may also infringe upon individual’s rights. This is not a flaw of the technology per se, but the responsibility of the humans who are designing and using it and who must ensure these violations do not happen in the first place. If breaches of fundamental rights do happen, these can also be very difficult to detect and prove, especially when the system is not transparent. This challenges the effective enforcement of the existing EU legislation aimed at safeguarding fundamental rights, as listed in section 1.3. While it remains difficult to quantify the real magnitude of the risks to fundamental rights, a growing amount of evidence suggests that Union citizens might be affected in an increasingly wide range. Moreover, a growing body of case law and legal challenges to the use of AI breaching fundamental rights is also emerging across different Member States. The following sections will focus on some of the most prominent fundamental rights risks. 2.1.1. Use of AI may violate human dignity and personal autonomy The right to human dignity is an inviolable right that requires every individual to be treated with respect as a human being and not as a mere ‘object’ and their personal autonomy respected. Depending on the circumstances of the case and the nature of the interaction, this might be challenging if people are misled in believing that they are interacting with another person when they are actually interacting with an AI system. Moreover, AI is often used to sort and classify traits and characteristics that emanate from datasets that are not based on the individual concerned. Organisations that use such data to determine individual’s status as a victim (e.g. women at risk of domestic violence) or individual’s likelihood to reoffend in case of predictive risk assessments could violate individuals’ right to human dignity since the assessment is no longer based on the personal individual situation and merits. AI can also be used for manipulation that can be particularly harmful for certain users. While psychological science shows that these problems are not new, the growing manipulative capabilities of algorithms that collect and can predict very sensitive and privacy intrusive personal information can make people extremely vulnerable, easily deceived or hyper-nudged towards specific decisions that do not align with their goals or run counter to their interests. Evidence suggests that AI supported products or services (toys, personal assistants etc.) can be intentionally designed or used in ways that appeal to the subliminal perception of individuals, thus causing them to take decisions that are beyond their cognitive capacities. Even if the techniques used are not subliminal, for certain categories of vulnerable subjects, in particular children, these might have the same adverse manipulative effects if their mental infirmity, age or credulity are exploited in harmful ways. As the AI application areas develop, these (mis)uses and risks will likely increase. 2.1.2. Use of AI may reduce privacy protection and violate the right to data protection The EU has a strong and modern legal framework on data protection with the Law Enforcement Directive and the General Data Protection Regulation recently evaluated as fit for purpose. Still, the use of AI systems might challenge the effective protection of individuals since the right to private life and other fundamental rights violations can occur even if non-personal, including anonymized data is processed.

The arbitrary use of algorithmic tools gives unprecedented opportunities for indiscriminate or mass surveillance, profiling and scoring of citizens and significant intrusion into people’s privacy and other fundamental rights. Beyond affecting the individuals concerned, such use of technology has also an impact on society as a whole and on broader Union values such as democracy, freedom, rule of law, etc. A particularly sensitive case is the increasing use of remote biometric identification systems in publicly accessible spaces. Currently, the most advanced variety of this family of applications is facial recognition, but other varieties exist, such as gait recognition or voice recognition. Wherever such a system is in operation, the whereabouts of persons included in the reference-database can be followed, thus impacting their personal data, privacy, autonomy and dignity. Moreover, freedom of expression, association and assembly might be undermined by the use of the technology resulting in a chilling effect on democracy. On the other hand, the use of such systems has been considered by some justified in limited cases when strictly necessary and proportionate for safeguarding important public interests of public security. Public and private operators are already using such systems in Europe, but because of privacy and other fundamental rights violations, their operation has been blocked by data protection authorities in schools or in other publicly accessible spaces. Despite these serious concerns and potential legal challenges, many countries consider using biometric identification systems at a much larger scale to cope with increasing security risks. While EU data protection rules in principle prohibit the processing of biometric data for the purpose of uniquely identifying a natural person except under specific conditions permitted by law, the White Paper on AI opened a discussion on the specific circumstances, if any, which might justify such use, and on common safeguards. Stakeholders views: In a recent survey, between 45% and 60% of consumers believed that AI will lead to more abuse of personal data (BEUC, Consumers see potential of artificial intelligence but raise serious concerns, 2020). 76.7 % of respondents to the White Paper on AI consider that the systems for remote biometric identification in public spaces have to be regulated in one way or another, 28.1% consider that they should never be authorized at publicly accessible spaces. Recently, 12 NGOs also started an EU-wide campaign called ‘Reclaim Your Face’ to urge EU to ban facial recognition in public spaces. The Commission has also registered a European Citizens' Initiative entitled ‘Civil society initiative for a ban on biometric mass surveillance practices'. Stakeholders views: In a recent survey, between 45% and 60% of consumers believed that AI will lead to more abuse of personal data (BEUC, Consumers see potential of artificial intelligence but raise serious concerns, 2020). 76.7 % of respondents to the White Paper on AI consider that the systems for remote biometric identification in public spaces have to be regulated in one way or another, 28.1% consider that they should never be authorized at publicly accessible spaces. Recently, 12 NGOs also started an EU-wide campaign called ‘Reclaim Your Face’ to urge EU to ban facial recognition in public spaces. The Commission has also registered a European Citizens' Initiative entitled ‘Civil society initiative for a ban on biometric mass surveillance practices'. 2.1.3. Use of AI may lead to discriminatory outcomes Algorithmic discrimination can occur for several reasons at many stages and it is often difficult to detect and mitigate. Problems may arise due to flawed design and developers who unconsciously embed their own biases and stereotypes when making the classification choices. Users might also misinterpret the AI output in concrete situations or use it in a way that is not fit for the intended purpose. Moreover, bias causes specific concerns for AI techniques dependent on data, which might be unrepresentative, incomplete or contain historical biases that can cement existing injustices with the ‘stamp’ of what appears to be scientific and evidence-based legitimacy. Developers or users could also intentionally or unintentionally use proxies that correlate with protected characteristics under EU non-discrimination legislation such as race, sex, disability etc. Although being based on seemingly neutral criteria, this may disproportionately affect certain protected groups giving rise to indirect discrimination (e.g., using proxies such as postal codes to account for ethnicity and race). As explained in the driver section 2.2., the algorithms can also introduce themselves biases in their reasoning mechanisms by favouring certain characteristics of the data on which they have been trained. Varying levels of accuracy in the performance of AI systems may also disproportionately affect certain groups, for example facial recognition systems that detect gender well for white men, but not for black women or that do not detect as person those using wheelchairs. The use of discriminatory AI systems notably in sectors such as employment, public administration, judiciary or law enforcement, might also violate many other fundamental rights (e.g. right to education, social security and social assistance, good administration etc.) and lead to broader societal consequences, reinforcing existing or creating new forms of structural discrimination and exclusion. For example, evidence suggests that in the employment sector AI is playing an increasingly key role in making hiring decisions mainly facilitated by intermediary tech service providers. This can negatively affect potential candidates in terms of discriminatory filtering at different moments of recruitment procedures or afterwards. Another problematic area is the administration of social welfare assistance, with some recent cases of suspected discriminatory profiling of unemployed people in Denmark, Poland or Austria. Financial institutions and other organisations might also use AI for assessing individual’s creditworthiness to support decisions determining the access to credit and other services such as housing. While this can increase the opportunities for some people to get access to credit on the basis of more diverse data points, there is also a risk that systems for assessing scores might unintentionally induce biases, if not properly designed and validated. In law enforcement and criminal justice, AI models trained with past data can be used to forecast trends in the development of criminality in certain geographic areas, to identify potential victims of criminal offences such as domestic violence or to assess the threats posed by individuals to commit offences based upon their criminal records and overall behaviour. In the EU, cases of these predictive policing systems exist in a number of Member States. At the borders, specific groups such as migrants and asylum seekers can also have their rights significantly affected if discriminatory AI systems are used by public authorities. Under the existing EU and national anti-discrimination law, it could be very difficult to launch a complaint as the affected person most likely do not know that an AI systems is used and even if they do, they are not aware how it functions and how its outputs are applied in practice. This makes it very difficult, if not impossible, for the persons concerned to establish the facts needed to establish prima facie discrimination, or prove it. It might also be very challenging for supervisory authorities and courts to detect and assess discrimination, in particular in cases when there is no readily available and relevant statistical evidence. Stakeholders views: 2020 BEUC survey - Consumers see potential of artificial intelligence but raise serious concerns: In a recent consumer organization survey in nine Member States, between 37% and 51% of respondents agree or strongly agree that AI will lead to unfair discrimination based on individual characteristics or social categories. In the public consultation on the AI White Paper, 89% of all respondents found that AI leading to discriminatory outcomes is an important or very important concern. This was a (very) important concern for 76% of SMEs and only 5% found it not important (at all). Large businesses were even more concerned: 89% said discrimination was a (very) important concern. Similarly, 91% of academic and research institutions and 90% of civil society organisations thought this was (very) important concern. Meanwhile, EU citizens were less concerned, although 78% still found this to be (very) important. Of those stakeholders stating that discriminatory outcomes were not important or very important concern, EU citizens (35%), academic and research institutions (19%) and SMEs (15%) were represented the most. For academic and research institutions and SMEs, this share was significantly larger than their representation in the overall sample. Stakeholders views: 2020 BEUC survey - Consumers see potential of artificial intelligence but raise serious concerns: In a recent consumer organization survey in nine Member States, between 37% and 51% of respondents agree or strongly agree that AI will lead to unfair discrimination based on individual characteristics or social categories. In the public consultation on the AI White Paper, 89% of all respondents found that AI leading to discriminatory outcomes is an important or very important concern. This was a (very) important concern for 76% of SMEs and only 5% found it not important (at all). Large businesses were even more concerned: 89% said discrimination was a (very) important concern. Similarly, 91% of academic and research institutions and 90% of civil society organisations thought this was (very) important concern. Meanwhile, EU citizens were less concerned, although 78% still found this to be (very) important. Of those stakeholders stating that discriminatory outcomes were not important or very important concern, EU citizens (35%), academic and research institutions (19%) and SMEs (15%) were represented the most. For academic and research institutions and SMEs, this share was significantly larger than their representation in the overall sample. 2.1.4. Use of AI might violate the right to an effective remedy, fair trial and good administration One prominent threat to the right to an effective remedy is the lack of transparency in the use and operation of AI systems. Without access given to relevant information, individuals may not be able to defend themselves and challenge any decision taken or supported by AI systems that might adversely affect them. This jeopardizes their right to be heard as well as the right to an effective remedy and fair trial. Furthermore, the use of automated decision-making in judicial proceedings might particularly affect the right of affected persons to access to court and to fair trial, if these systems are not subject to appropriate safeguards for transparency, accuracy, non-discrimination and human oversight. The opacity of AI could also hamper the ability of persons charged with a crime to defend themselves and challenge the evidence used against them. In the context of AI-enabled individual risk assessments increasingly used in law enforcement, singling out people without reasonable suspicion or on the basis of biased or flawed data might also threaten the presumption of innocence. Public authorities may also not be able to properly reason their individual administrative decisions which is required as part of the principle and the right to good administration. Stakeholders views: In the public consultation on the White Paper on AI, only 2% of all respondents said that AI breaching fundamental rights is not (at all) an important concern. 85% of SMEs considered this (very) important, while none found the issue to be unimportant. Similarly, 87% large businesses found this to be (very) important– only one respondent (1%) found it not important. 93% and 94% of academic/research institutions and civil society organisations, respectively, were (very) concerned about fundamental rights breaches. EU citizens were also concerned: 83% found potential breaches of fundamental rights (very) important. Among those stakeholders who found this not to be a (very) important concern, academic and research institutions were the largest group with 33% (much higher than their 14% share of the entire sample). Stakeholders views: In the public consultation on the White Paper on AI, only 2% of all respondents said that AI breaching fundamental rights is not (at all) an important concern. 85% of SMEs considered this (very) important, while none found the issue to be unimportant. Similarly, 87% large businesses found this to be (very) important– only one respondent (1%) found it not important. 93% and 94% of academic/research institutions and civil society organisations, respectively, were (very) concerned about fundamental rights breaches. EU citizens were also concerned: 83% found potential breaches of fundamental rights (very) important. Among those stakeholders who found this not to be a (very) important concern, academic and research institutions were the largest group with 33% (much higher than their 14% share of the entire sample). Problem 3: Competent authorities do not have powers, resources and/or procedural frameworks to ensure and monitor compliance of AI use with fundamental rights and safety rules The specific characteristics of many AI technologies, set out in section 2.2., often make it hard to verify how outputs and decisions have been reached where AI is used. As a consequence, it may become impossible to verify compliance with existing EU law meant to guarantee safety and protect fundamental rights. For example, to determine whether a recruitment decision is justified or involved discrimination, enforcement authorities need to determine how this decision was reached. Yet, since there is no requirement for producers and users of AI systems to keep proper documentation and ensure traceability of these decision-making processes, public authorities may not be able to properly investigate, prove and sanction a breach. The governance and enforcement mechanisms under existing sectoral legislation also suffer from shortcomings. Firstly, the use of AI systems may lead to situations where market surveillance and supervisory authorities may not be empowered to act and/or do not have the appropriate technical capabilities and expertise to inspect these systems. Secondly, existing secondary legislation on data protection, consumer protection and non-discrimination legislation relies primarily on ex-post mechanisms for enforcement and focuses on individual remedies for ‘data subjects’ or ’consumers’. To evaluate compliance with fundamental rights, the purpose and use of an application needs to be assessed in context and it is the responsibility of every actor to comply with their legal obligations. Unless legal compliance in view of the intended purpose and context is taken into account already at the design stage, harmful AI systems might be placed on the market and violate individual fundamental rights at scale before any enforcement action is taken by competent authorities.

Thirdly, as set out above, the current safety legislation does not provide yet for clear and specific requirements for AI systems that are embedded in products. Outside the scope of product safety legislation, there is also no binding obligation for prior testing and validation of the systems before they are placed on the market. Moreover, after systems are placed on the market and deployed, there is no strict ex post obligation for continuous monitoring which is, however, essential given the continuous learning capabilities of certain AI system or their changing performance due to regular software updates. Fourthly, the secondary legislation on fundamental rights primarily places the burden for compliance on the user and often leaves the provider of the AI system outside its scope. However, while users remain responsible for a possible breach of fundamental rights obligations, providers might be best placed to prevent and mitigate some of the risks already at an early development stage. Users are also often unable to fully understand the workings of AI applications if not provided with all the necessary information. Because of these gaps in the existing legislation, procedures by supervisory authorities may not result in useful findings. Finally, given the complexity and rapid speed of AI development, competent authorities often lack the necessary resources, expertise and technological tools to effectively supervise risks posed by the use of AI systems to safety and fundamental rights. They also do not have sufficient tools for cooperation with authorities in other Member States for carrying out joint investigations, or even at national level where, for example, various sectoral legislation might intersect and lead to violations of multiple fundamental rights or to risks to both safety and fundamental rights. Problem 4: Legal uncertainty and complexity on how to ensure compliance with rules applicable to AI systems dissuade businesses from developing and using the technology Due to the specific characteristics of AI set out in section 2.2., businesses using AI technology are also facing increasing legal uncertainty and complexity on how to comply with existing legislation. Considering the various sources of risks at all different levels, organisations involved in the complex AI value chain might be unclear who exactly should ensure compliance with the different requirements under the existing legislation. For example, providers of AI systems might be unclear on what measures they should integrate to minimize the risks to safety and fundamental rights’ violation, while users might not be able to remedy features of the design that are inadequate for the context of application. In this context, the evolving nature of risks also pose particular problems to correctly attribute responsibilities. Providers of AI systems may have limited information with regard to the harm that AI can produce post deployment (especially if the application context has not been taken into account in the design of the system), while users may be unable to exercise due care when operating the AI system if not properly informed about its nature and provided with guidance about the required oversight and ex post control. The lack of clear distribution of obligations across the AI value chain taking into account all these specific features of the AI technology leads to significant legal uncertainty for companies, while failing to effectively minimise the increasing risks to safety and fundamental rights, as identified above. Another problem is that there are no harmonized standards under EU law as to how general principles or requirements on security, non-discrimination, transparency, accuracy, human oversight should be implemented specifically as regards AI systems in the design and development stage. This results in legal uncertainty on the business side, which affects both the developer and the user of the AI system. Also, there are no clear red lines when companies should not engage in the use of AI for certain particularly harmful practices beyond the practices explicitly listed in the Unfair Commercial Practice Directive. Certification for trustworthy AI product and services that are currently available on the Union market is also missing and creates uncertainties across the AI value chain. Without a clear legal framework, start-ups and developers working in this field will not be able to attract the required investments. Similarly, without certainty on applicable rules and clear common standards on what is required for a trustworthy, safe and lawful AI, developers and providers of AI systems and other innovators are less likely to pursue developments in this field. As a result, businesses and public authorities using AI technology fear becoming responsible for fundamental rights infringements, which may dissuade them from using AI. At European level, 69% of companies cite a ‘need for new laws or regulation’ as an obstacle to adoption of AI. SMEs were more likely than large companies to say that this was not a challenge or barrier, whereas small enterprises were more likely to identify this as a major barrier (31% for companies with 5 to 9 employees and 28% for those with 10 to 49 employees) compared to medium and large companies. Figure 2: Obstacles to the use of AI (by companies) Source: Ipsos Survey, 2020 At national level, a survey of 500 companies by the German Technical Inspection Association TÜV found that 42% of businesses expect legal challenges due to the use of AI, while 84% said that there was uncertainty around AI within their company. As a result, 84% wanted AI products to be clearly marked for users, and 87% were in favour of a risk-based approach to regulation. Yet, a McKinsey global survey (2017) showed that companies that are committed to AI have significantly higher profit margins across sectors. These companies also expect a margin increase of up to five percentage points more than industry average in the following three years. Hence, direct costs from legal uncertainty in the market of AI development and use are also accompanied by a missed potential for business innovation. In the end, even consumers will suffer, as they will miss out beneficial products and services not developed by businesses due to the fear of legal consequences. Problem 5: Mistrust in AI would slow down AI development in Europe and reduce the global competitiveness of the EU economy If citizens observe that AI repeatedly endangers the safety of individuals or infringes their fundamental rights, they are unlikely to be willing to accept the use of AI technologies for themselves or by other users. In a recent survey aimed at consumers in nine EU Member States, around 56% of respondents said they had low trust in authorities to exert effective control over AI. Similarly, only 12% of the survey respondents in Sweden and Finland reported to trust private companies’ abilities to address the ethical dilemmas AI brings. 77% thought that companies that are developing new AI solutions should be bound to ethical guidelines and regulation. Faced with reluctant private and business customers, businesses will find it then more difficult to invest in and adopt AI than if customers embrace AI. As a result, demand for new and innovative AI applications will be sub-optimal. This mistrust will hamper innovation because companies will be more hesitant in offering innovative services for which they first have to establish credibility, which will be particularly challenging in a context of growing public fears about the risks of AI. That is why in a recent study, 42% of EU executives surveyed see “guidelines or regulations to safeguard ethical and transparent use” as part of a policy to promote AI to the benefit of Europe. Similarly, a recent survey of European companies found that for 65% of respondents a lack of trust among citizens is an obstacle to the adoption of AI. There is already a substantial share of companies not preparing for the AI-enabled future: 40% of the companies neither use any AI application whatsoever nor intend to do so. The problem is particularly acute for SMEs that cannot rely on their brand to reassure customers that they are trustworthy. Significantly fewer large companies (34%) than SMEs (between 41% and 43% depending on company size) saw lack of trust as not an obstacle. The share of companies citing this as a major obstacle was relatively evenly distributed across company sizes (between 26% and 28%).

Figure 3: Trust in AI applications Source: Ipsos survey 20-28 September 2018, quoted by Statista Without a sound common framework for trustworthy AI, Europe could lose out on the beneficial impact of AI on competitiveness. Yet, the benefits of rapid adoption of AI are generally estimated as being very significant. McKinsey estimates that if Europe (EU28) on average develops and distributes AI according to its current assets and digital position relative to the world, it could add some €2.7 trillion, or 20%, to its combined economic output by 2030. According to a European Value Added Assessment, prepared by the European Parliament, a common EU framework on the ethics of AI has the potential to bring the European Union € 294.9 billion in additional GDP and 4.6 million additional jobs by 2030. Thus, a slower adoption of AI would have significant economic costs and would hamper innovation. It would also mean foregoing part of the wide array of the societal benefits that AI is poised to bring in areas such as health, transport or pollution reduction. It would thus negatively affect not just businesses and the economy, but consumers and the society as well. Problem 6: Fragmented measures create obstacles for a cross-border AI single market and threaten the Union’s digital sovereignty In the absence of a common European framework to address the risks examined before and build trust in AI technology, Member States can be expected to start taking action at a national level to deal with these specific challenges. While national legislation is within the Member States’ sovereign competences, there is a risk that diverging national approaches will lead to market fragmentation and could create obstacles especially for smaller companies to enter multiple national markets and scale up across the EU Single Market. Yet, as noted in section 1.2., AI applications are rapidly increasing in scale. Where advanced models work with billions of parameters, companies need to scale up their models to remain competitive. Since the high mobility of AI producers could lead to a race to the bottom where companies move to Member States with the lightest regulation and serve the entire EU market from there, other Member States may take measures to limit access from other Member States, leading to further market fragmentation. That is why Member States in general support a common European approach to AI. In a recent position paper 14 Member States recognise the risk of market fragmentation and emphasise that the ‘main aim must be to create a common framework where trustworthy and human-centric AI goes hand in hand with innovation, economic growth and competitiveness’. Earlier, in its conclusions of 9 June 2020, the Council called upon the Commission ‘to put forward concrete proposals, taking existing legislation into consideration, which follow a risk-based, proportionate and, if necessary, regulatory approach for artificial intelligence.’ While waiting for a European proposal, some Member States are already considering national legislative or soft-law measures to address the risks, build trust in AI and support innovation. For example, the German Data Ethics Commission has called for a five-level risk-based system of horizontal regulation on AI that would go from no regulation for the most innocuous AI systems to a complete ban for the most dangerous ones. Denmark has just launched the prototype of a Data Ethics Seal, whilst Malta has introduced a voluntary certification system for AI. Spain is in the process of adopting a Code of Ethics and considering certification of AI products and services. Finland issued recommendations for self-regulation and the development of responsibility standards for the private sector, while Italy envisages certificates to validate and to monitor AI applications developed in an ethically sound way. Moreover, several Member States (e.g. Belgium, Sweden, Netherlands and Portugal) are considering the need for binding legislation on the legal and ethical aspects of AI. In addition to this increasingly patchy national landscape, there is an ongoing proliferation of voluntary international technical standards for various aspects of ‘Trustworthy AI’ adopted by international standardisation organisations (e.g. IEEE, ISO/IEC, ETSI, ITU-T, NIST). While these standards can in principle be very helpful in ensuring safe and trustworthy AI systems, there is also a growing risk of divergence between them since they are adopted by different international organisations. Moreover, these technical standards may not be fully compliant with existing EU legislation (e.g., on data protection or non-discrimination), which creates additional liability risks and legal uncertainty for companies adhering to them. In addition, there is also a proliferation of national technical standards on AI that are adopted or being developed by a number of Member States and many other third countries around the world. This means national standards risk not being fully interoperable or compatible with each other, which will create obstacles to cross-border movement and to the scaling up of AI-driven services and products across different Member States. The impact of this increasing fragmentation is disproportionately affecting small companies. This is because large companies, especially global ones, can spread the additional costs for operating across an increasingly fragmented single market over their larger sales, especially when they already have established a dominant position in some markets. Meanwhile, SMEs and start-ups which do not have the market power or the same resources may be deterred from entering the markets of other Member States and thus fail to profit from the single market. This problem is further exacerbated since big tech players have not only a technological advantage but also exclusive access to large and quality data necessary for the development of AI. They may try to use this information asymmetry to seek economic advantages and further harm smaller companies. These dominant tech companies may also try to free ride on political efforts aiming to increase consumer protection by ensuring that the adopted standards for AI are in line with their own business practices to the detriment of newcomers and smaller players. This risk is significantly higher when the AI market is fragmented with individual Member States taking unilateral actions. All these diverging measures stand in the way of a seamless and well-functioning single market for trustworthy AI in the Union and pose particular legal barriers for SMEs and start-ups. This in turn negatively affects the global competiveness of the EU industry, both regarding AI providers and the industries using AI, giving advantage to companies from third countries that are already dominant on the global market. Beyond the purely market dimension, there is a growing risk that the ‘digital sovereignty’ of the Union and the Member States might be threatened since such AI-driven products and services from foreign companies might not completely comply with Union values and/or legislation or they might even pose security risks and make the European infrastructure more vulnerable. As stated by the President of the Commission von der Leyen, to ensure ‘tech sovereignty’, the EU should strengthen its capability to make its own choices, based on its own values, respecting its own rules. Aside from strengthening the EU internal market, such tech sovereignty will also facilitate the development and the leverage of Union’s tools and regulatory power to shape global rules and standards on AI. 2.2. What are the main problem drivers? The uptake of AI systems has a strong potential to bring benefits, economic growth and enhance EU innovation and global competitiveness. However, as mentioned in the section above, in certain cases, the use of AI systems can also create problems for businesses and national authorities as well as new risks to safety and fundamental rights of individuals. The key cause explaining the analysed problems are the specific characteristics of AI systems which make them qualitatively different from previous technological advancements. Table 3 below explains what each characteristic means and why they can create problems for fundamental rights and safety. Table 3: Five specific characteristics of AI and Problem Drivers CHARACTERISTICS EXPLANATION (simplified) WHY IS IT A PROBLEM?/ DRIVERS Opacity/ (lack of transparency) Limited ability of human mind to understand how certain AI systems operate A lack of transparency (opacity)) in AI (due to complexity or how the algorithm was realized in code or how the application is realised) makes it difficult to monitor, identify and prove possible breaches of laws, including legal provisions that protect fundamental rights. Complexity Multiplicity of different components and processes of an AI system and their interlinks The complexity of AI makes it difficult to monitor, identify and prove possible breaches of laws, including legal provisions that protect fundamental rights. Continuous adaptation and unpredictability Functional ability of some AI systems to continuously ‘learn’ and ‘adapt’ as they operate, sometimes leading to unpredictable outcomes. Some AI systems change and evolve over time and may even change their own behaviour in unforeseen ways. This can give rise to new risks that are not adequately addressed by the existing legislation. Autonomous behaviour Functional ability of some AI systems to generate outputs such as ‘decisions’ with limited or no human intervention The autonomous behaviour of AI systems can affect safety because of the functional ability of an AI system to perform a task with minimum or no direct human intervention. Data Functional dependence on data and the quality of data The dependence of AI systems on data and their quality, the AI’s ‘ability’ to infer correlations from data input and learn from data, including proxies, can reinforce systemic biases and errors, and exacerbate discriminatory and adverse results. Certain AI systems may include only some of the characteristics. However, as a rule, the more specific characteristics a given AI system has, the higher the probability that it becomes a ‘black box’. The term ‘black box’ reflects the limited ability of even most advanced experts to monitor an AI system. This stands in considerable difference with the ability to monitor other technologies. The OECD report explains this ‘black box’ effect with the example of neural networks as follows: “Neural networks iterate on the data they are trained on. They find complex, multi-variable probabilistic correlations that become part of the model that they build. However, they do not indicate how data would interrelate. The data are far too complex for the human mind to understand.” ‘Black box’ AI systems present new challenges for public policy compared to traditional and other technologies. These specific characteristics of AI systems may create new (1) safety and security and (2) fundamental rights risks and accelerate the probability or intensity of the existing risks, as well as (3) make it hard for enforcement authorities to verify compliance with and enforce the existing rules. This set of problems leads in turn to (4) legal uncertainty for companies, (5) potentially slower uptake of AI technologies, due to the lack of trust, by businesses and citizens as well as (6) regulatory responses by Member States to mitigate possible externalities. Consequently, problems triggered by specific characteristics of AI may lead to safety risks and breaches of fundamental rights and challenge the effective application of and compliance with the EU legal framework for the protection of fundamental rights and safety. Table 4: Problem Tree

Rapid developments and uptake of AI systems increase this challenge. The Ipsos 2019 survey of European business indicates that 42% of enterprises currently use at least one AI technology, a quarter of them use at least two types, and 18% have plans to adopt AI technologies in the next two years. Moreover, the intensity of use of AI technology by business is expected to grow in the next two years. This data suggests that in the next two years, it is likely that more than half of all EU businesses will be using AI systems. Thus, AI systems already affect business and consumers in the EU on a large scale. According to the European Commission Better Regulation Guidelines and accompanying Toolbox, a public policy intervention may be justified, among other grounds, when regulations fail or when protection and fulfilment of fundamental rights afforded to citizens of the Union provide grounds for intervention. Several factors may cause regulatory failures, including, when existing public intervention becomes “out of date as the world evolves.” Protection and fulfilment of fundamental rights afforded to citizens of the Union may also provide important reasons for policy intervention ‘because even a perfectly competitive and efficient economy can produce outcomes that are unacceptable in terms of equity’. 2.3. How will the problem evolve? Given the increasing public awareness of the potential for violation of safety and fundamental rights that AI has, it is likely that the proliferation of ethics principles will continue. Companies would adopt these principle unilaterally in an effort to reassure their potential customers. However, such non-binding ethical principles cannot build the necessary trust as they cannot be enforced by affected parties and no external party or regulator is actually empowered to check whether these principles are duly respected by the companies and the public authorities developing and using AI systems. Moreover, the multiplicity of commitments would require consumers to spend an extraordinary amount of time understanding which commitments apply to which application. As a consequence, and given the significant commercial opportunities offered by AI solutions, ‘untrustworthy’ AI solutions could ensue, with a likely backlash against AI technology as a whole by citizens and businesses. If and when this happens, European citizens will lose out on the benefits of AI and European companies will be placed at a significant disadvantage compared to their overseas competitors with a dynamic home market. Over time, technological developments in the fields of algorithmic transparency, accountability and fairness could improve the situation, but progress and impact will be uncertain and uneven across Europe. On the contrary, as AI develops, it can be implemented in more and more situations and sectors, so that the problems identified above apply to an ever-growing share of citizens’ life. It cannot be excluded that over the long run and after a sufficient number of incidents, consumers will prefer companies with a proven track record of trustworthy AI. However, apart from the damage done in the meantime, this would have the consequence of favouring large companies, who can rely on their brand image, over SMEs who will face increasing legal barriers to enter the market. 3.WHY SHOULD THE EU ACT? 3.1. Legal basis The initiative constitutes a core part of the EU single market strategy given that artificial intelligence has already found its way into a vast majority of services and products and will only continue to do so in the future. EU action on the basis of Article 114 of the Treaty on the Functioning of the European Union can be taken for the purposes of the approximation of the provisions laid down by law, regulation or administrative action in the Member States when it has as its object the establishment and functioning of the internal market. The measures must be intended to improve the conditions for the establishment and functioning of the internal market and must genuinely have that object, actually contributing to the elimination of obstacles to the free movement of goods or services, or to the removal of distortions of competition. Article 114 TFEU may be used as a legal basis to prevent the occurrence of these obstacles resulting from diverging national laws and approaches how to address the legal uncertainties and gaps in the existing legal frameworks applicable to AI. The present initiative aims to improve the functioning of the internal market by setting harmonized rules on the development, placing on the Union market and the use of products and services making use of the AI technology or provided as stand-alone AI applications. Some Member States are already considering national rules to ensure AI is safe and is developed and used in compliance with fundamental rights obligations. This will likely lead to a further fragmentation of the internal market and increasing legal uncertainty for providers and users on how existing and new rules will apply to AI systems. Furthermore, the Court of Justice has recognised that applying heterogeneous technical requirements could be valid grounds to trigger Article 114 TFEU. The new initiative will aim to address that problem by proposing harmonised technical standards for the implementation of common requirements applicable to the design and development of certain AI systems before they are placed on the market. The initiative will also address the situation after AI systems have been placed on the market by harmonising the way in which ex-post controls are conducted. Based on the above, Article 114 TFEU is the applicable legal basis for the present initiative. In addition, considering that this Regulation contains certain specific rules, unrelated to the functioning of the internal market, restricting the use of AI systems for ‘real-time’ remote biometric identification by the law enforcement authorities of the Member States, which necessarily limits the processing of biometric data by those authorities, it is appropriate to base this Regulation, in as far as those specific rules are concerned, on Article 16 of the Treaty. 3.2. Subsidiarity: Necessity of EU action The intrinsic nature of AI which often relies on large and varied datasets and which might be embedded in any product or service circulating freely within the internal market mean that the objectives of the initiative cannot effectively be achieved by Member States alone. An emerging patchy framework of potentially divergent national rules will hamper the seamless provision of AI systems across the EU and is ineffective in ensuring the safety and protection of fundamental rights and Union values across the different Member States. Such an approach is unable to solve the problems of ineffective enforcement and governance mechanisms and will not create common conditions for building trust in the technology across all Member States. National approaches in addressing the problems will only create additional legal uncertainty, legal barriers and will slow market uptake of AI even further. Companies could be prevented from seamlessly expanding into other Member States, depriving consumers and other users of the benefits of their services and products and affecting negatively the competitiveness of European companies and the economy. 3.3. Subsidiarity: Added value of EU action The objectives of the initiative can be better achieved at Union level so as to avoid a further fragmentation of the Single Market into potentially contradictory national frameworks preventing the free circulation of goods and services embedding AI. In their positions to the White paper on AI all Member States support a coordinated action at EU level to prevent the risk of fragmentation and create the necessary conditions for a single market of safe, lawful and trustworthy AI in Europe. A solid European regulatory framework for trustworthy AI will also ensure a level playing field and protect all European citizens, while strengthening Europe’s competitiveness and industrial basis in AI. A common EU legislative action on AI could boost the internal market and has great potential to provide European industry with a competitive edge at the global scene and economies of scale that cannot be achieved by individual Member States alone. Setting up the governance structures and mechanisms for a coordinated European approach to AI across all sectors and Member States will enhance safety and the respect of fundamental rights, while allowing businesses, public authorities and users of AI systems to capitalise on the scale of the internal market and use safe and trustworthy AI products and services. Only common action at EU level can also protect Union’s tech sovereignty and leverage its tools and regulatory powers to shape global rules and standards. 4.Objectives: What is to be achieved? The problems analysed in section 2 above are complex and cannot be fully addressed by any single policy intervention. This is why the Commission proposes to address emerging problems related specifically to AI systems gradually. The objectives of this initiative are defined accordingly.

Table 5: General/Specific objectives 1. 2. 3. 4. 4.1.General objectives The general objective of the intervention is to ensure the proper functioning of the single market by creating the conditions for the development and use of trustworthy artificial intelligence in the Union. 4.2.Specific objectives The specific objectives of this initiative are as follows: ·set requirements specific to AI systems and obligations on all value chain participants in order to ensure that AI systems placed on the market and used are safe and respect the existing law on fundamental rights and Union values; ·ensure legal certainty to facilitate investment and innovation in AI by making it clear what essential requirements, obligations, as well as conformity and compliance procedures must be followed to place, or use an AI system in the Union market; ·enhance governance and effective enforcement of the existing law on fundamental rights and safety requirements applicable to AI systems by providing new powers, resources and clear rules for relevant authorities on conformity assessment and ex post monitoring procedures and the division of governance and supervision tasks between national and EU levels; ·facilitate the development of a single market for lawful, safe and trustworthy AI applications and prevent market fragmentation by taking EU action to set minimum requirement for AI systems to be placed and used in the Union market in compliance with the existing law on fundamental rights and safety. 4.2.1.Ensure that AI systems placed on the market and used are safe and respect the existing law on fundamental rights and Union values Safeguarding the safety and fundamental rights of EU citizens is a cornerstone of European values. However, the emergence of AI creates new challenges regarding safety and protection of fundamental rights and hinder the enforcement of these rights, due to the specific features of this technology (see section 2.2.). However, the same rights and rules that apply in the analogue world should also be respected when AI systems are used. The first specific objective of the initiative is, therefore, to ensure that AI systems that are developed, placed on the market and/or used in the Union are safe and respect the existing law on fundamental rights and Union values by setting requirements specific to AI systems and obligations on all value chain participants. The ongoing review of the sectoral safety legislation will pursue a similar objective to ensure safety of products embedding AI technology, but focusing on the overall safety of the whole product and the safe integration of the AI system into the product. 4.2.2. Ensure legal certainty to facilitate investment and innovation in AI Due to the specific characteristics of AI, businesses using AI technology are also facing increasing legal uncertainty and complexity on how to comply with existing legislation. As long as the challenges and risks to safety and fundamental rights have not been addressed, companies must also calculate the risk that legislation or other requirements will be introduced, without knowing what this will imply for their business models. Such legal uncertainty is detrimental for investment and especially for innovation. The second objective is therefore to promote investment and innovation by creating single market-wide legal certainty and an appropriate framework that stimulates innovation by making it clear what essential requirements, obligations as well as conformity and compliance procedures must be followed to place or use an AI system in the Union market. The complementary initiative on liability rules will also aim to increase legal certainty in the use of AI technology, but by ensuring a high-level of protection of victims who have suffered harms caused by certain AI systems. 4.2.3.Enhance governance and effective enforcement of the existing law on fundamental rights and safety requirements applicable to AI systems The technological features of AI such as opacity and autonomous behaviour might cause violations of safety rules and the existing law on fundamental rights that may not even be noticed by the concerned person, and that, even when they are noticed, are often difficult to prove. Existing competent authorities might also face difficulties in auditing the compliance of certain AI systems with the existing legislation due to the specific technological features of AI. They might also lack powers to intervene against actors who are outside their jurisdiction or lack sufficient resources and a mechanism for cooperation and joint investigations with other competent authorities. The enforcement and governance system needs to be adapted to these new challenges so as to ensure that possible breaches can be effectively detected and sanctioned by enforcement authorities and those affected. The third objective is therefore to improve the governance mechanism and effective enforcement of the existing law on fundamental rights and safety requirements applicable to AI by providing new powers, resources and clear rules for relevant authorities on conformity assessment and ex post monitoring procedures and the division of governance and supervision tasks between national and EU levels.

4.2.4. Facilitate the development of a single market for lawful, safe and trustworthy AI applications and prevent market fragmentation The safety and fundamental rights risks posed by AI may lead citizens and consumers to mistrust this technology, inciting in turn Member States to address these problems with national measures which may create obstacles to cross border sales, especially for SMEs. The fourth objective is, hence, to foster trustworthy AI, which will reduce the incentives for national and potentially mutually incompatible legislations and will remove legal barriers and obstacles to cross border movement of products and services embedding the AI technology by taking EU action to set minimum requirements for AI systems to be placed and used in the Union market in compliance with the existing law on fundamental rights and safety. The complementary initiative on liability rules would also aim to increase trust in the AI technology, but by ensuring a high-level of protection of victims who have suffered harms caused by certain AI systems. 4.3. Objectives tree/intervention logic. Figure 4: Intervention logic The specific characteristics of certain AI systems (opacity, complexity, autonomous behaviour, unpredictability and data dependency) may create (1) safety and security and (2) fundamental rights risks and (3) make it hard for enforcement authorities to verify compliance with and enforce the existing rules.