dominates the development of advanced general-purpose AI systems. American institutions produced the majority (54%) of large language and multimodal models in 2022 (562). Industry now surpasses academia in producing significant machine learning models (32 vs three in 2022) and industry co-authorship on papers presented at the top ten leading AI conferences rose from 22% in 2000 to 38% in 2020 (563).

**The rising 'compute divide' is contributing to disparities in the distribution of computing resources, and unequal participation in general-purpose AI development.** The term 'compute divide' describes the different extent to which large industrial AI labs and typical academic AI labs have access to computing resources (556). In recent years, this divide has widened (556, 557). Estimates show that US technology companies are the major buyers of NVIDIA H100 GPUs, one of the most powerful GPU chip types on the market explicitly designed for AI (564*). Amazon, Meta, Google, and Microsoft have all recently announced custom AI chips to reduce their dependence on the AI chip supply chain, potentially paving the way for more widespread access to GPUs. However, the exceptionally high cost of GPUs ($15,000 for top-tier models such as the H100 at the time of writing) could hinder academic institutions and less wealthy countries from affording this level of AI infrastructure.

**The delegation of lower-level AI work to workers in low-income countries has led to a 'ghost work' industry.** From content moderation to proofreading to data labelling, a lot of human labour that the typical consumer is usually not aware of – sometimes referred to as 'ghost work' – is necessary for many products of large technology companies (565). The increasing demand for data to train general-purpose AI systems, including human feedback to aid in training, has further increased the reliance on ghost work including the creation of firms helping big technology companies to outsource various aspects of data production, including data collection, cleaning, and annotation. This trend has played a significant role in the development of notable machine learning benchmark datasets such as ImageNet (566). Data production, a crucial aspect of general-purpose AI advancement, often relies on workers in countries with lower average wages. These workers may face exposure to graphic content, erratic schedules, heavy workloads, and have limited social and economic mobility (567, 568, 569, 570). This can lead to harms against marginalised workers, widening the AI divide, and increasing the disparity of who reaps the benefits from advanced general-purpose AI development.

### 4.3.3 Market concentration risks and single points of failure

#### KEY INFORMATION

* Developing state-of-the-art, general-purpose AI models requires substantial up-front investment. These very high costs create barriers to entry, disproportionately benefiting large technology companies.

* Market power is concentrated among a few companies that are the only ones able to build the leading general-purpose AI models.

* Widespread adoption of a few general-purpose AI models and systems by critical sectors including finance, cybersecurity, and defence creates systemic risk because any flaws, vulnerabilities, bugs, or inherent biases in the dominant general-purpose AI models and systems could cause simultaneous failures and disruptions on a broad scale across these interdependent sectors.

The development of state-of-the-art AI systems currently costs hundreds of millions, or even billions of US dollars. The biggest upfront investments are specialised computational resources, AI expertise and access to large, often proprietary datasets (see 2.3.1 Recent trends in compute, data, and algorithms). The significant costs associated with these inputs are a barrier to entry for new firms (571,
dominates the development of advanced general-purpose AI systems. American institutions produced the majority (54%) of large language and multimodal models in 2022 (562). Industry now surpasses academia in producing significant machine learning models (32 vs three in 2022) and industry co-authorship on papers presented at the top ten leading AI conferences rose from 22% in 2000 to 38% in 2020 (563).

**The rising 'compute divide' is contributing to disparities in the distribution of computing resources, and unequal participation in general-purpose AI development.** The term 'compute divide' describes the different extent to which large industrial AI labs and typical academic AI labs have access to computing resources (556). In recent years, this divide has widened (556, 557). Estimates show that US technology companies are the major buyers of NVIDIA H100 GPUs, one of the most powerful GPU chip types on the market explicitly designed for AI (564*). Amazon, Meta, Google, and Microsoft have all recently announced custom AI chips to reduce their dependence on the AI chip supply chain, potentially paving the way for more widespread access to GPUs. However, the exceptionally high cost of GPUs ($15,000 for top-tier models such as the H100 at the time of writing) could hinder academic institutions and less wealthy countries from affording this level of AI infrastructure.

**The delegation of lower-level AI work to workers in low-income countries has led to a 'ghost work' industry.** From content moderation to proofreading to data labelling, a lot of human labour that the typical consumer is usually not aware of – sometimes referred to as 'ghost work' – is necessary for many products of large technology companies (565). The increasing demand for data to train general-purpose AI systems, including human feedback to aid in training, has further increased the reliance on ghost work including the creation of firms helping big technology companies to outsource various aspects of data production, including data collection, cleaning, and annotation. This trend has played a significant role in the development of notable machine learning benchmark datasets such as ImageNet (566). Data production, a crucial aspect of general-purpose AI advancement, often relies on workers in countries with lower average wages. These workers may face exposure to graphic content, erratic schedules, heavy workloads, and have limited social and economic mobility (567, 568, 569, 570). This can lead to harms against marginalised workers, widening the AI divide, and increasing the disparity of who reaps the benefits from advanced general-purpose AI development.

### 4.3.3 Market concentration risks and single points of failure

#### KEY INFORMATION

* Developing state-of-the-art, general-purpose AI models requires substantial up-front investment. These very high costs create barriers to entry, disproportionately benefiting large technology companies.

* Market power is concentrated among a few companies that are the only ones able to build the leading general-purpose AI models.

* Widespread adoption of a few general-purpose AI models and systems by critical sectors including finance, cybersecurity, and defence creates systemic risk because any flaws, vulnerabilities, bugs, or inherent biases in the dominant general-purpose AI models and systems could cause simultaneous failures and disruptions on a broad scale across these interdependent sectors.

The development of state-of-the-art AI systems currently costs hundreds of millions, or even billions of US dollars. The biggest upfront investments are specialised computational resources, AI expertise and access to large, often proprietary datasets (see 2.3.1 Recent trends in compute, data, and algorithms). The significant costs associated with these inputs are a barrier to entry for new firms (571,
572, 573, 574). Large technology companies are well-positioned thanks to their existing access to the necessary resources and ability to make substantial financial investments.

In addition, general-purpose Al systems benefit from scale. More compute-intensive large-scale models tend to outperform smaller ones (110*), giving rise to economies of scale: large-scale general-purpose Al systems are in higher demand due to their superior performance, driving down their costs per customer. High user numbers also have network effects: as more users interact with these models, they generate large amounts of additional training data that can be used to improve the models' performance (575).

These tendencies towards market concentration in the general-purpose Al industry are particularly concerning because of general-purpose Al's potential to enable greater centralisation of decision-making in a few companies than ever before. Since society at large could benefit as well as suffer from these decisions, this raises questions about the appropriate governance of these few large-scale systems. A single general-purpose Al model could potentially influence decision-making across many organisations and sectors (571) in ways which might be benign, subtle, inadvertent, or deliberately exploited. There is the potential for the malicious use of general-purpose Al as a powerful tool for manipulation, persuasion and control by a few companies or governments. Potentially harmful biases such as demographic, personality traits, and geographical bias, which might be present in any dominant general-purpose Al model that become embedded in multiple sectors, could propagate widely. For example, popular text-to-image models like DALL-E 2 and Stable Diffusion exhibit various demographic biases across occupations, personality traits, and geographical contexts (576).

The increasing dependence on a few Al systems across critical sectors introduces systemic risks. Errors, bugs, or cyberattacks targeting these systems could cause widespread disruption. Different scenarios have been proposed that illustrate potential disruptions. For example, a denial-of-service attack on a widely used Al API could disrupt critical public infrastructure which relies on that technology. In finance, the adoption of homogeneous Al systems by multiple institutions could destabilise markets by synchronising participants' decisions (577): If several banks rely on one model, they may inadvertently make similar choices, creating systemic vulnerabilities (2*). Comparable risks could potentially arise in domains, like defence or cybersecurity, if Al systems with similar functionality are widely deployed (see also 4.4. Cross-cutting risk factors).

### 4.3.4 Risks to the environment

#### KEY INFORMATION

* Growing compute use in general-purpose Al development and deployment has rapidly increased energy usage associated with general-purpose Al.
* This trend might continue, potentially leading to strongly increasing CO2 emissions.

The recent rapid growth in demand for computing power ('compute') used for Al, and particularly general-purpose Al, development and deployment could make Al a major, and potentially the largest, contributor to data centre electricity consumption in the near future. This is because compute demand is expected to far outpace hardware efficiency improvements.

Today, data centres, servers and data transmission networks account for between 1% to 1.5% of global electricity demand (578); roughly 2% in the EU, 4% in the US, and close to 3% in China (69, 579, 580). Al likely accounts for well under half of data centre electricity consumption currently, but if the rapid growth of Al's computational requirements continues, Al could become the primary consumer of data centre electricity over the coming years and increase its share of global electricity demand. In 2023, the largest general-purpose Al training runs used around 5e25 FLOP (65). Using H100 GPUs running at 1400 watts per GPU for operation and cooling, this consumes around 40 GWh. If this figure were to
572, 573, 574). Large technology companies are well-positioned thanks to their existing access to the necessary resources and ability to make substantial financial investments.

In addition, general-purpose Al systems benefit from scale. More compute-intensive large-scale models tend to outperform smaller ones (110*), giving rise to economies of scale: large-scale general-purpose Al systems are in higher demand due to their superior performance, driving down their costs per customer. High user numbers also have network effects: as more users interact with these models, they generate large amounts of additional training data that can be used to improve the models' performance (575).

These tendencies towards market concentration in the general-purpose Al industry are particularly concerning because of general-purpose Al's potential to enable greater centralisation of decision-making in a few companies than ever before. Since society at large could benefit as well as suffer from these decisions, this raises questions about the appropriate governance of these few large-scale systems. A single general-purpose Al model could potentially influence decision-making across many organisations and sectors (571) in ways which might be benign, subtle, inadvertent, or deliberately exploited. There is the potential for the malicious use of general-purpose Al as a powerful tool for manipulation, persuasion and control by a few companies or governments. Potentially harmful biases such as demographic, personality traits, and geographical bias, which might be present in any dominant general-purpose Al model that become embedded in multiple sectors, could propagate widely. For example, popular text-to-image models like DALL-E 2 and Stable Diffusion exhibit various demographic biases across occupations, personality traits, and geographical contexts (576).

The increasing dependence on a few Al systems across critical sectors introduces systemic risks. Errors, bugs, or cyberattacks targeting these systems could cause widespread disruption. Different scenarios have been proposed that illustrate potential disruptions. For example, a denial-of-service attack on a widely used Al API could disrupt critical public infrastructure which relies on that technology. In finance, the adoption of homogeneous Al systems by multiple institutions could destabilise markets by synchronising participants' decisions (577): If several banks rely on one model, they may inadvertently make similar choices, creating systemic vulnerabilities (2*). Comparable risks could potentially arise in domains, like defence or cybersecurity, if Al systems with similar functionality are widely deployed (see also 4.4. Cross-cutting risk factors).

### 4.3.4 Risks to the environment

#### KEY INFORMATION

* Growing compute use in general-purpose Al development and deployment has rapidly increased energy usage associated with general-purpose Al.
* This trend might continue, potentially leading to strongly increasing CO2 emissions.

The recent rapid growth in demand for computing power ('compute') used for Al, and particularly general-purpose Al, development and deployment could make Al a major, and potentially the largest, contributor to data centre electricity consumption in the near future. This is because compute demand is expected to far outpace hardware efficiency improvements.

Today, data centres, servers and data transmission networks account for between 1% to 1.5% of global electricity demand (578); roughly 2% in the EU, 4% in the US, and close to 3% in China (69, 579, 580). Al likely accounts for well under half of data centre electricity consumption currently, but if the rapid growth of Al's computational requirements continues, Al could become the primary consumer of data centre electricity over the coming years and increase its share of global electricity demand. In 2023, the largest general-purpose Al training runs used around 5e25 FLOP (65). Using H100 GPUs running at 1400 watts per GPU for operation and cooling, this consumes around 40 GWh. If this figure were to
grow at a rate of 3x/year, then at the end of the decade, the largest training run would consume 90 TWh, over half of total US data centre electricity consumption in 2022.

There are several potential mitigations for the increasing energy use of widespread general-purpose Al systems. Specialised Al hardware and other hardware efficiency improvements can enhance the performance-per-watt of machine learning workloads over time (78). Moreover, new machine learning techniques and architectures can help reduce energy consumption (78). The energy efficiency of computation generally improves by an estimated 26% annually (68). However, even with additional optimisation for Al, the growing demand for computing power used for Al training, which has been increasing by a factor of approximately 4x each year, is so far significantly to outpacing energy efficiency improvements (17).

The CO2 emissions resulting from Al development and deployment depend on the extent and sources of its energy consumption as well as several factors. The carbon intensity of the energy source is a key variable, with renewable sources like solar power contributing substantially less CO2 emissions throughout their life cycle compared to fossil fuels (581*). Al firms often rely on renewable energy (76, 78), a significant portion of Al training globally still relies on high-carbon sources such as coal or natural gas (581*). Other important factors affecting CO2 emissions include the geographic location of data centres, their efficiency, and the efficiency of the hardware used. As a result, the actual CO2 emissions for a given amount of energy consumed in Al can vary considerably.

The 'embodied carbon footprint' of Al hardware, which includes emissions from manufacturing, transportation, the physical building infrastructure, and disposal (as opposed to not running the hardware), contributes a substantial portion to emissions - depending on the location this could be as high as 50% (76). As hardware efficiency improves, the embodied carbon footprint could become a larger proportion of the total carbon footprint (76, 78).

Water consumption might be another noteworthy area of environmental risk from Al. Given the increases in compute used for training and deploying models, cooling demands increase, too, leading to higher water consumption. Water consumption by current models and the methodology to assess it are still subject to scientific debate, but some researchers predict that water consumption by Al could ramp up to billions of cubic metres by 2027 (76, 582). In the context of concerns around global freshwater scarcity, and assuming no obvious short-term cooling alternatives exist, Al water footprint might be a substantial cause of environmental concern.

### 4.3.5 Risks to privacy

#### KEY INFORMATION

* General-purpose Al models or systems can 'leak' information about individuals whose data was used in training. For future models trained on sensitive personal data like health or financial data, this may lead to particularly serious privacy leaks.

* General-purpose Al models could enhance privacy abuse. For instance, Large Language Models might facilitate more efficient and effective search for sensitive data (for example, on internet text or in breached data leaks), and also enable users to infer sensitive information about individuals.

General-purpose Al systems rely on and process vast amounts of personal data, and this could pose significant and potentially wide-reaching privacy risks. Such risks include loss of data confidentiality for people whose data was used to train these systems, loss of transparency and control over how data-driven decisions are made, and new forms of abuse that these systems could enable.

Privacy, broadly speaking, refers to a person's right to control others' access to their sensitive or personal information. In the context of Al, privacy is a complex and multi-faceted concept, which
grow at a rate of 3x/year, then at the end of the decade, the largest training run would consume 90 TWh, over half of total US data centre electricity consumption in 2022.

There are several potential mitigations for the increasing energy use of widespread general-purpose Al systems. Specialised Al hardware and other hardware efficiency improvements can enhance the performance-per-watt of machine learning workloads over time (78). Moreover, new machine learning techniques and architectures can help reduce energy consumption (78). The energy efficiency of computation generally improves by an estimated 26% annually (68). However, even with additional optimisation for Al, the growing demand for computing power used for Al training, which has been increasing by a factor of approximately 4x each year, is so far significantly to outpacing energy efficiency improvements (17).

The CO2 emissions resulting from Al development and deployment depend on the extent and sources of its energy consumption as well as several factors. The carbon intensity of the energy source is a key variable, with renewable sources like solar power contributing substantially less CO2 emissions throughout their life cycle compared to fossil fuels (581*). Al firms often rely on renewable energy (76, 78), a significant portion of Al training globally still relies on high-carbon sources such as coal or natural gas (581*). Other important factors affecting CO2 emissions include the geographic location of data centres, their efficiency, and the efficiency of the hardware used. As a result, the actual CO2 emissions for a given amount of energy consumed in Al can vary considerably.

The 'embodied carbon footprint' of Al hardware, which includes emissions from manufacturing, transportation, the physical building infrastructure, and disposal (as opposed to not running the hardware), contributes a substantial portion to emissions - depending on the location this could be as high as 50% (76). As hardware efficiency improves, the embodied carbon footprint could become a larger proportion of the total carbon footprint (76, 78).

Water consumption might be another noteworthy area of environmental risk from Al. Given the increases in compute used for training and deploying models, cooling demands increase, too, leading to higher water consumption. Water consumption by current models and the methodology to assess it are still subject to scientific debate, but some researchers predict that water consumption by Al could ramp up to billions of cubic metres by 2027 (76, 582). In the context of concerns around global freshwater scarcity, and assuming no obvious short-term cooling alternatives exist, Al water footprint might be a substantial cause of environmental concern.

### 4.3.5 Risks to privacy

#### KEY INFORMATION

* General-purpose Al models or systems can 'leak' information about individuals whose data was used in training. For future models trained on sensitive personal data like health or financial data, this may lead to particularly serious privacy leaks.

* General-purpose Al models could enhance privacy abuse. For instance, Large Language Models might facilitate more efficient and effective search for sensitive data (for example, on internet text or in breached data leaks), and also enable users to infer sensitive information about individuals.

General-purpose Al systems rely on and process vast amounts of personal data, and this could pose significant and potentially wide-reaching privacy risks. Such risks include loss of data confidentiality for people whose data was used to train these systems, loss of transparency and control over how data-driven decisions are made, and new forms of abuse that these systems could enable.

Privacy, broadly speaking, refers to a person's right to control others' access to their sensitive or personal information. In the context of Al, privacy is a complex and multi-faceted concept, which
encompasses issues of confidentiality, transparency, and control. Privacy is a challenging concept to
define (583). In the context of AI it encompasses:

* Data confidentiality and protection of personal data collected or used for training purposes, or during inference (584)

* Transparency, and controls over how personal information is used in AI systems (585), for example the ability for individuals to opt-out from personal data being collected for training, or the post-hoc ability to make a general-purpose AI system 'unlearn' specific information about an individual (586);

* Individual and collective harms that may occur as a result of data use or malicious use, for example the creation of deepfakes (587).

General-purpose AI systems may expose their training data. The training of general-purpose AI models generally requires large amounts of training data. Academic studies have shown that some of this training data may be memorised by the general-purpose AI model, or may be extractable using adversarial inputs, enabling users to infer information about individuals whose data was collected (588, 589, 590) or even reconstruct entire training examples (591, 592, 593, 594). However, definitions of memorisation vary, so it is challenging to make any concrete claims about the harms that might arise from memorisation (595).

Many systems are trained on publicly available data containing personal information without the
knowledge or consent of the individuals it pertains to. This information could then be outputted by a
general-purpose AI system in undesired contexts. There is a risk that training models on sensitive data
containing personal information (such as medical or financial data) could result in serious privacy
leaks. It is difficult to assess the likelihood or potential impact of these risks: for example, existing
medical general-purpose AI systems such as Google's Gemini-Med (596*) are only trained on
anonymised public patient data, and the rate at which such models regurgitate training data has not
yet been studied. General-purpose AI systems that continuously learn from interactions with users
(e.g. chatbots such as ChatGPT) might also leak such interactions to other users, although at the time
of writing, there are no well-documented cases of this occurring.

General-purpose AI systems could enable privacy abuse. Some studies have found that general-
purpose AI systems have privacy-relevant capabilities that may be exploited by malicious users of
these systems. For example, fine-grained internet-wide search capabilities, such as powerful reverse
image search or forms of writing style detection, which allow individuals to be identified and tracked
across online platforms, or sensitive personal characteristics to be inferred, further eroding individual
privacy (597, 598). Large language models could also enable more efficient and effective search for
sensitive information on the internet, or in breached datasets. General-purpose AI-generated content,
such as non-consensual deepfakes, could be used to manipulate or harm individuals, raising concerns
about the harm caused by the malicious use of personal data and the erosion of trust in online
content (255, 256, 373, 599).

4.3.6 Copyright infringement

KEY INFORMATION

* The use of large amounts of copyrighted data for training general-purpose AI models poses a challenge to traditional intellectual property laws, and to systems of consent, compensation, and control over data.

* The use of copyrighted data at scale by organisations developing general-purpose AI is likely to alter incentives around creative expression.

* An unclear copyright regime disincentivizes general-purpose AI developers from following best practices for data transparency.
encompasses issues of confidentiality, transparency, and control. Privacy is a challenging concept to define (583). In the context of AI it encompasses:

* Data confidentiality and protection of personal data collected or used for training purposes, or during inference (584)

* Transparency, and controls over how personal information is used in AI systems (585), for example the ability for individuals to opt-out from personal data being collected for training, or the post-hoc ability to make a general-purpose AI system 'unlearn' specific information about an individual (586);

* Individual and collective harms that may occur as a result of data use or malicious use, for example the creation of deepfakes (587).

General-purpose AI systems may expose their training data. The training of general-purpose AI models generally requires large amounts of training data. Academic studies have shown that some of this training data may be memorised by the general-purpose AI model, or may be extractable using adversarial inputs, enabling users to infer information about individuals whose data was collected (588, 589, 590) or even reconstruct entire training examples (591, 592, 593, 594). However, definitions of memorisation vary, so it is challenging to make any concrete claims about the harms that might arise from memorisation (595).

Many systems are trained on publicly available data containing personal information without the knowledge or consent of the individuals it pertains to. This information could then be outputted by a general-purpose AI system in undesired contexts. There is a risk that training models on sensitive data containing personal information (such as medical or financial data) could result in serious privacy leaks. It is difficult to assess the likelihood or potential impact of these risks: for example, existing medical general-purpose AI systems such as Google's Gemini-Med (596*) are only trained on anonymised public patient data, and the rate at which such models regurgitate training data has not yet been studied. General-purpose AI systems that continuously learn from interactions with users (e.g. chatbots such as ChatGPT) might also leak such interactions to other users, although at the time of writing, there are no well-documented cases of this occurring.

General-purpose AI systems could enable privacy abuse. Some studies have found that general-purpose AI systems have privacy-relevant capabilities that may be exploited by malicious users of these systems. For example, fine-grained internet-wide search capabilities, such as powerful reverse image search or forms of writing style detection, which allow individuals to be identified and tracked across online platforms, or sensitive personal characteristics to be inferred, further eroding individual privacy (597, 598). Large language models could also enable more efficient and effective search for sensitive information on the internet, or in breached datasets. General-purpose AI-generated content, such as non-consensual deepfakes, could be used to manipulate or harm individuals, raising concerns about the harm caused by the malicious use of personal data and the erosion of trust in online content (255, 256, 373, 599).

### 4.3.6 Copyright infringement

#### KEY INFORMATION

* The use of large amounts of copyrighted data for training general-purpose AI models poses a challenge to traditional intellectual property laws, and to systems of consent, compensation, and control over data.

* The use of copyrighted data at scale by organisations developing general-purpose AI is likely to alter incentives around creative expression.

* An unclear copyright regime disincentivizes general-purpose AI developers from following best practices for data transparency.
* There is very limited infrastructure for sourcing and filtering legally and ethically permissible data from the internet for training general-purpose AI models.

**General-purpose AI models are usually trained on large data sets sourced online, giving rise to concerns over breaches of copyright, lack of creator compensation, and the potential for economic disruption.** Copyright laws aim to protect intellectual property and encourage written and creative expression (600, 601). They grant the creators of original works the exclusive right to copy, distribute, adapt, and perform their own work. However, the third-party use of copyrighted data as training data may be legally permissible in certain circumstances, for instance on the basis of the 'fair use' exception in the US (602), by the 'text and data mining' exception in the EU (603), by the amended Copyright Act in Japan (604), under Israeli copyright law (605), and by the Copyright Act 2021 in Singapore (606). Beyond copyright, artists and other individuals sometimes feel their style, voice, and likeness are not sufficiently protected, which may implicate other forms of intellectual property such as trademarks and brands.

Recent advances in general-purpose AI capabilities have largely resulted from large-scale web scraping and aggregation of data to train general-purpose AI models (607, 608), often containing copyrighted works, or used without consent from the data's creators. This applies to creative works including text, images, videos, and speech, and other modalities that are increasingly used to develop general-purpose AI models. The extent to which this is legally permissible is complex and can vary by country. In the US, the fair use exception has been argued for in the case of training general-purpose AI models (222, 609, 610, 611), as well as legally challenged (612). Many issues related to dataset creation and use across its lifecycle make copyright concerns for training AI models very complicated (613). These issues include the questions of whether datasets are assembled specifically for machine learning or originally for other purposes (614), whether the infringement analysis applies to model inputs or model outputs (615), and issues of jurisdiction, among others (236, 616, 617). This also presents questions on who is liable for infringement or harmful model outputs (618). While there are technical strategies for mitigating the risks of copyright infringement from model outputs, these risks are difficult to eliminate entirely (619, 620).

As general-purpose AI systems become more capable, they increasingly have the potential to disrupt labour markets, and in particular creative industries (250, 621), (also see 4.3.1 *Labour market risks*). The legal determinations regarding copyright infringement in the AI training phase will affect the ability for general-purpose AI developers to build powerful and performant models. They may also impact data creators' ability to exert control over their data, which may disincentivize creative expression.

**An unclear copyright regime disincentivizes general-purpose AI developers from improving data transparency.** Transparency about general-purpose AI model training data is useful for understanding various potential risks and harms of a general-purpose AI system (259). However, this type of transparency is often lacking for major general-purpose AI developers (244). Fears of legal risk, especially over copyright infringements, may disincentivize these developers from disclosing their training data (622).

**The infrastructure to source and filter for legally permissible data is under-developed, making it hard for developers to comply with copyright law.** The permissibility of using copyrighted works as part of training data without a licence is an active area of litigation. Tools to source and identify available data without copyright concerns are limited. For instance, recent work shows that ~60% of popular datasets in the most widely used openly accessible dataset repositories have incorrect or missing licence information (236). Similarly, there are limitations to the current tools for discerning copyright-free data in web scrapes (607, 623). However, practitioners are developing new standards for data documentation and new protocols for data creators to signal their consent for use in training AI models (235, 624).
* There is very limited infrastructure for sourcing and filtering legally and ethically permissible data from the internet for training general-purpose AI models.

**General-purpose AI models are usually trained on large data sets sourced online, giving rise to concerns over breaches of copyright, lack of creator compensation, and the potential for economic disruption.** Copyright laws aim to protect intellectual property and encourage written and creative expression (600, 601). They grant the creators of original works the exclusive right to copy, distribute, adapt, and perform their own work. However, the third-party use of copyrighted data as training data may be legally permissible in certain circumstances, for instance on the basis of the 'fair use' exception in the US (602), by the 'text and data mining' exception in the EU (603), by the amended Copyright Act in Japan (604), under Israeli copyright law (605), and by the Copyright Act 2021 in Singapore (606). Beyond copyright, artists and other individuals sometimes feel their style, voice, and likeness are not sufficiently protected, which may implicate other forms of intellectual property such as trademarks and brands.

Recent advances in general-purpose AI capabilities have largely resulted from large-scale web scraping and aggregation of data to train general-purpose AI models (607, 608), often containing copyrighted works, or used without consent from the data's creators. This applies to creative works including text, images, videos, and speech, and other modalities that are increasingly used to develop general-purpose AI models. The extent to which this is legally permissible is complex and can vary by country. In the US, the fair use exception has been argued for in the case of training general-purpose AI models (222, 609, 610, 611), as well as legally challenged (612). Many issues related to dataset creation and use across its lifecycle make copyright concerns for training AI models very complicated (613). These issues include the questions of whether datasets are assembled specifically for machine learning or originally for other purposes (614), whether the infringement analysis applies to model inputs or model outputs (615), and issues of jurisdiction, among others (236, 616, 617). This also presents questions on who is liable for infringement or harmful model outputs (618). While there are technical strategies for mitigating the risks of copyright infringement from model outputs, these risks are difficult to eliminate entirely (619, 620).

As general-purpose AI systems become more capable, they increasingly have the potential to disrupt labour markets, and in particular creative industries (250, 621), (also see 4.3.1 *Labour market risks*). The legal determinations regarding copyright infringement in the AI training phase will affect the ability for general-purpose AI developers to build powerful and performant models. They may also impact data creators' ability to exert control over their data, which may disincentivize creative expression.

**An unclear copyright regime disincentivizes general-purpose AI developers from improving data transparency.** Transparency about general-purpose AI model training data is useful for understanding various potential risks and harms of a general-purpose AI system (259). However, this type of transparency is often lacking for major general-purpose AI developers (244). Fears of legal risk, especially over copyright infringements, may disincentivize these developers from disclosing their training data (622).

**The infrastructure to source and filter for legally permissible data is under-developed, making it hard for developers to comply with copyright law.** The permissibility of using copyrighted works as part of training data without a licence is an active area of litigation. Tools to source and identify available data without copyright concerns are limited. For instance, recent work shows that ~60% of popular datasets in the most widely used openly accessible dataset repositories have incorrect or missing licence information (236). Similarly, there are limitations to the current tools for discerning copyright-free data in web scrapes (607, 623). However, practitioners are developing new standards for data documentation and new protocols for data creators to signal their consent for use in training AI models (235, 624).
## 4.4 Cross-cutting risk factors

### 4.4.1 Cross-cutting technical risk factors

#### KEY INFORMATION

* This section covers seven cross-cutting *technical risk factors* – technical factors that each contribute to many general-purpose Al risks.

a. General-purpose Al systems can be applied in many ways and contexts, making it hard to test and assure their trustworthiness across all realistic use-cases.

b. General-purpose Al developers have a highly limited understanding of how general-purpose Al models and systems function internally to achieve the capabilities they output.

c. General-purpose Al systems can act in accordance with unintended goals, leading to potentially harmful outputs, despite testing and mitigation efforts by Al developers.

d. A general-purpose Al system can be rapidly deployed to very large numbers of users, so if a faulty system is deployed at scale, resulting harm could be rapid and global.

e. Currently, risk assessment and evaluation methods for general-purpose Al systems are immature and can require significant effort, time, resources, and expertise.

f. Despite attempting to debug and diagnose, developers are not able to prevent overtly harmful behaviours across all circumstances in which general-purpose Al systems are used.

g. Some developers are working to create general-purpose Al systems that can act with increasing autonomy, which could increase the risks by enabling more widespread applications of general-purpose Al systems with less human oversight.

Risk factors are distinct from risks. They are conditions that increase the likelihood and/or impact of risks occurring. This section covers seven cross-cutting *technical risk factors* i.e.– factors that each contribute to multiple general-purpose Al risks.

**General-purpose Al systems can be applied in many ways and contexts, making it hard to test and assure their trustworthiness across all possible use cases.** The relative safety of general-purpose Al systems depends on the context in which they are used. General-purpose Al systems' outputs are often open-ended, such as free-form dialogue or code generation. This makes it difficult to design safe systems because it is not tractable to exhaustively evaluate all possible downstream use cases. Users can also 'jailbreak' general-purpose Al models to make them comply with potentially harmful requests (see 5.2.3 Improving robustness to failures). At present, computer scientists are unable to give guarantees of the form "System X will not do Y" about general-purpose Al systems (625). As discussed in 3. Methodology to assess and understand general-purpose Al systems, assessing the risks of general-purpose Al systems in real-world applications and making strong assurances against general-purpose Al-related harms is extremely difficult with current methods.

**General-purpose Al developers have a highly limited understanding of how general-purpose Al models and systems function internally.** A key feature of general-purpose Al systems is that their capabilities are mainly achieved through learning rather than from top-down design. As a result, unlike most human-engineered systems, state-of-the-art general-purpose Al models do not come with blueprints, and their structure does not conform to common design principles. This gives rise to concerns around understanding or explaining general-purpose Al, which are used interchangeably in this report to refer to the ability to provide human-understandable accounts of how general-purpose Al arrives at outputs and decisions from inputs and objectives. There are different views around what
## 4.4 Cross-cutting risk factors

### 4.4.1 Cross-cutting technical risk factors

#### KEY INFORMATION

* This section covers seven cross-cutting *technical risk factors* – technical factors that each contribute to many general-purpose AI risks.

a. General-purpose AI systems can be applied in many ways and contexts, making it hard to test and assure their trustworthiness across all realistic use-cases.

b. General-purpose AI developers have a highly limited understanding of how general-purpose AI models and systems function internally to achieve the capabilities they output.

c. General-purpose AI systems can act in accordance with unintended goals, leading to potentially harmful outputs, despite testing and mitigation efforts by AI developers.

d. A general-purpose AI system can be rapidly deployed to very large numbers of users, so if a faulty system is deployed at scale, resulting harm could be rapid and global.

e. Currently, risk assessment and evaluation methods for general-purpose AI systems are immature and can require significant effort, time, resources, and expertise.

f. Despite attempting to debug and diagnose, developers are not able to prevent overtly harmful behaviours across all circumstances in which general-purpose AI systems are used.

g. Some developers are working to create general-purpose AI systems that can act with increasing autonomy, which could increase the risks by enabling more widespread applications of general-purpose AI systems with less human oversight.

Risk factors are distinct from risks. They are conditions that increase the likelihood and/or impact of risks occurring. This section covers seven cross-cutting *technical risk factors* i.e.– factors that each contribute to multiple general-purpose AI risks.

**General-purpose AI systems can be applied in many ways and contexts, making it hard to test and assure their trustworthiness across all possible use cases.** The relative safety of general-purpose AI systems depends on the context in which they are used. General-purpose AI systems' outputs are often open-ended, such as free-form dialogue or code generation. This makes it difficult to design safe systems because it is not tractable to exhaustively evaluate all possible downstream use cases. Users can also 'jailbreak' general-purpose AI models to make them comply with potentially harmful requests (see 5.2.3 Improving robustness to failures). At present, computer scientists are unable to give guarantees of the form "System X will not do Y" about general-purpose AI systems (625). As discussed in 3. Methodology to assess and understand general-purpose AI systems, assessing the risks of general-purpose AI systems in real-world applications and making strong assurances against general-purpose AI-related harms is extremely difficult with current methods.

**General-purpose AI developers have a highly limited understanding of how general-purpose AI models and systems function internally.** A key feature of general-purpose AI systems is that their capabilities are mainly achieved through learning rather than from top-down design. As a result, unlike most human-engineered systems, state-of-the-art general-purpose AI models do not come with blueprints, and their structure does not conform to common design principles. This gives rise to concerns around understanding or explaining general-purpose AI, which are used interchangeably in this report to refer to the ability to provide human-understandable accounts of how general-purpose AI arrives at outputs and decisions from inputs and objectives. There are different views around what
constitutes a human-understandable account. A thorough discussion of these views lies outside the
scope of this report, but key questions include:

* What forms it may take, e.g. whether it has to be easily comprehensible language or can be complex mathematical information, and whether the size of the description should be bounded to allow humans to make sense of the explanation.

* How comprehensive it must be, e.g. whether it should include retrospective analysis of how an explained decision process was established in training;

* Whether it must be counterfactual, i.e. whether it must allow hypothetical statements about what outputs would have resulted from different inputs, different objectives or different training.

Currently, scientists' understanding of general-purpose AI systems is more analogous to that of brains or cells than aeroplanes or power plants. Some researchers believe that it may be possible to develop general-purpose AI systems that can be proven safe, or are 'safe by design' (626) by focusing the validation on interpretable outputs of the neural networks rather than their internal states, which humans currently cannot understand. However, it has not yet been possible for scientists to achieve such quantitative safety guarantees for state-of-the-art general-purpose AI models (58). Research on thoroughly understanding how AI systems operate has been limited to 'toy' systems that are much smaller and less capable than general-purpose AI models (627, 628, 629, 630), or are unreliable and require major simplifying assumptions (631, 632*). In practice, techniques for interpreting the inner workings of neural networks can be misleading (213, 288*, 289, 290, 336), and can fail sanity checks or prove unhelpful in downstream uses (218, 297, 298, 299). As discussed in 3. Methodology to assess and understand general-purpose AI systems, these research methods are being developed, and new improvements might yield further insights, especially with sufficient investment in further research. However, it is unclear yet whether interpreting the inner structures of neural networks will offer sufficient safety assurances.

**Ensuring that general-purpose AI systems pursue the goals intended by their developers and users is difficult.** Although general-purpose AI systems can appear to excel at learning what they are 'told' to do, their behaviour may not necessarily be what their designers intended (489, 633, 634, 635). Even subtle differences between a designer's goals and the incentives given to a system can lead to unexpected failures. For example, general-purpose AI chatbots are often trained to produce text that will be rated positively by human evaluators, but user approval is an imperfect proxy for user benefit: widely-used chatbots can learn to pander to users' biases instead of prioritising truth (237, 238). Even when a general-purpose AI system receives correct feedback during training, it may still develop a solution that does not generalise well when applied to novel situations during new situations once deployed (636, 637, 638) because the training data may not adequately represent real-world scenarios. For example, some researchers have found that chatbots are more likely to comply with harmful requests in languages that are under-represented in their training data (309). See 4.2.3. Loss of control and 5.2. Training more trustworthy models for further discussion of these challenges.

**Because general-purpose AI systems can proliferate rapidly, like other software, a new fault or harmful capability can rapidly have a global and sometimes irreversible impact.** A small number of proprietary and freely available (open-source) general-purpose AI models reach many millions of users (4.3.3.Market concentration risks and single points of failure). Both proprietary and open-source models can therefore have rapid and global impacts when they are released, although in different ways. Once a model is made available open-source, there is no practical way to erase the model from the market in case it has faults or capabilities that enable malicious use (639) (see 4.1. Malicious use risks). For model faults, however, open-sourcing a model allows a much greater and more diverse number of practitioners to discover them which can improve the understanding of risks and possible mitigations (640) (see 3. Methodology to assess and understand general-purpose AI systems). Developers or others can then repair faults and encourage users to update to a new model version. Furthermore, open-sourcing models allows more actors to customise these models. If one version of the model has flaws, other customised versions may not share the same issues (640). However, neither repairing model faults nor customising can prevent deliberate malicious use (639). The risk of deliberate malicious use depends on a model's marginal risk compared to available closed source
constitutes a human-understandable account. A thorough discussion of these views lies outside the
scope of this report, but key questions include:

* What forms it may take, e.g. whether it has to be easily comprehensible language or can be complex mathematical information, and whether the size of the description should be bounded to allow humans to make sense of the explanation.

* How comprehensive it must be, e.g. whether it should include retrospective analysis of how an explained decision process was established in training;

* Whether it must be counterfactual, i.e. whether it must allow hypothetical statements about what outputs would have resulted from different inputs, different objectives or different training.

Currently, scientists' understanding of general-purpose AI systems is more analogous to that of brains or cells than aeroplanes or power plants. Some researchers believe that it may be possible to develop general-purpose AI systems that can be proven safe, or are 'safe by design' (626) by focusing the validation on interpretable outputs of the neural networks rather than their internal states, which humans currently cannot understand. However, it has not yet been possible for scientists to achieve such quantitative safety guarantees for state-of-the-art general-purpose AI models (58). Research on thoroughly understanding how AI systems operate has been limited to 'toy' systems that are much smaller and less capable than general-purpose AI models (627, 628, 629, 630), or are unreliable and require major simplifying assumptions (631, 632*). In practice, techniques for interpreting the inner workings of neural networks can be misleading (213, 288*, 289, 290, 336), and can fail sanity checks or prove unhelpful in downstream uses (218, 297, 298, 299). As discussed in 3. Methodology to assess and understand general-purpose AI systems, these research methods are being developed, and new improvements might yield further insights, especially with sufficient investment in further research. However, it is unclear yet whether interpreting the inner structures of neural networks will offer sufficient safety assurances.

**Ensuring that general-purpose AI systems pursue the goals intended by their developers and users is difficult.** Although general-purpose AI systems can appear to excel at learning what they are 'told' to do, their behaviour may not necessarily be what their designers intended (489, 633, 634, 635). Even subtle differences between a designer's goals and the incentives given to a system can lead to unexpected failures. For example, general-purpose AI chatbots are often trained to produce text that will be rated positively by human evaluators, but user approval is an imperfect proxy for user benefit: widely-used chatbots can learn to pander to users' biases instead of prioritising truth (237, 238). Even when a general-purpose AI system receives correct feedback during training, it may still develop a solution that does not generalise well when applied to novel situations during new situations once deployed (636, 637, 638) because the training data may not adequately represent real-world scenarios. For example, some researchers have found that chatbots are more likely to comply with harmful requests in languages that are under-represented in their training data (309). See 4.2.3. Loss of control and 5.2. Training more trustworthy models for further discussion of these challenges.

**Because general-purpose AI systems can proliferate rapidly, like other software, a new fault or harmful capability can rapidly have a global and sometimes irreversible impact.** A small number of proprietary and freely available (open-source) general-purpose AI models reach many millions of users (4.3.3.Market concentration risks and single points of failure). Both proprietary and open-source models can therefore have rapid and global impacts when they are released, although in different ways. Once a model is made available open-source, there is no practical way to erase the model from the market in case it has faults or capabilities that enable malicious use (639) (see 4.1. Malicious use risks). For model faults, however, open-sourcing a model allows a much greater and more diverse number of practitioners to discover them which can improve the understanding of risks and possible mitigations (640) (see 3. Methodology to assess and understand general-purpose AI systems). Developers or others can then repair faults and encourage users to update to a new model version. Furthermore, open-sourcing models allows more actors to customise these models. If one version of the model has flaws, other customised versions may not share the same issues (640). However, neither repairing model faults nor customising can prevent deliberate malicious use (639). The risk of deliberate malicious use depends on a model's marginal risk compared to available closed source
models and other technologies such as internet search (640) (see 4.1. Malicious use risks). The above factors are relevant to the specific possibility of rapid, widespread, and irreversible impacts of general-purpose Al models, but this report does not provide an assessment of the overall impacts of open-source models. Even when a system is not open-sourced, its capabilities can still be accessed by a wide user base. Within two months of launch, ChatGPT had over 100 million users and set a record for the fastest-growing user base of any consumer application (641). Each time a general-purpose Al system is updated, a new version of it rapidly reaches a large user base, so any vulnerabilities or harmful tendencies can potentially have a global impact quickly (see also 4.3.3. Market concentration and single points of failure.)

**Despite attempting to debug and diagnose, developers are not able to prevent even overtly harmful behaviours across all circumstances in which general-purpose Al systems are used.**

Empirically, harmful behaviours have included revealing private or copyrighted information (221, 642*, 643); generating hate speech (225, 644); spreading social and political biases (239, 455, 645); pandering to user biases (238); hallucinating inaccurate content (46, 47*, 49); exhibiting vulnerabilities to a variety of attacks on their safety protections (108, 208, 209, 210, 211, 309, 646, 647, 648*); and assisting in overtly harmful tasks (368, 649, 650). Users can circumvent general-purpose Al model safeguards with relative ease (650, 651), for example through 'jailbreaking' techniques (see 3. Methodology to assess and understand general-purpose Al systems). While some have called for safety measures that rule out all overtly harmful behaviours across all situations (626), current general-purpose Al development fails to meet a lower standard than this: ruling out any specific overtly harmful behaviour across foreseeable situations (such as situations where users try to jailbreak a model.)

## Al developers are working to create 'agent' general-purpose Al systems that can accomplish tasks with little to no human involvement, which could increase risks of accident and malicious use

Today, general-purpose Al systems are primarily used directly as tools by humans. For example, a human may ask a chatbot to write computer code to help them accomplish a task, which naturally requires a 'human in the loop'. However, developers are increasingly designing systems that allow general-purpose Al systems to act autonomously, by controlling software tools such as a web browser or controlling the execution of code rather than only writing code. This enables some forms of reasoning about problems, constructing plans, and executing plans step-by-step (13, 26, 188, 189, 652, 653, 654, 655*, 656). Such systems have included web-browsing virtual agents (657*), research assistants (415), and writing, fixing, and running code autonomously (441).

The main purpose of general-purpose Al 'agents' is to reduce the need for human involvement and oversight, allowing for faster and cheaper applications of general-purpose Al. This property also reduces human oversight, potentially increasing the risk of accidents (see 4.1 Malicious use risks), and allowing the automation workflows for malicious uses (see 4.2.1 Risks from product functionality issues use) while also being relevant to the risk of loss of control (see 4.2.3 Loss of control) (481*, 658).

**General-purpose Al agents have early forms of many autonomous capabilities, but lack reliability at performing complex tasks autonomously.** Current state-of-the-art general-purpose Al systems are capable of autonomously executing many simple tasks, but some evaluations have shown that they struggle with more complex ones (183, 509). They are particularly unreliable at performing tasks that involve many steps. As of 2023, general-purpose Al agents scored low in benchmarks designed to measure their performance on complex and economically useful tasks (183, 441). However, given current efforts and growing investment in developing general-purpose Al systems with greater autonomy, the capabilities of general-purpose Al agents could continue to increase.

**Although current general-purpose Al agents are unreliable, advancement has been rapid.** For example, for two challenging benchmarks for general-purpose Al agents that measure general
models and other technologies such as internet search (640) (see 4.1. Malicious use risks). The above factors are relevant to the specific possibility of rapid, widespread, and irreversible impacts of general-purpose Al models, but this report does not provide an assessment of the overall impacts of open-source models. Even when a system is not open-sourced, its capabilities can still be accessed by a wide user base. Within two months of launch, ChatGPT had over 100 million users and set a record for the fastest-growing user base of any consumer application (641). Each time a general-purpose Al system is updated, a new version of it rapidly reaches a large user base, so any vulnerabilities or harmful tendencies can potentially have a global impact quickly (see also 4.3.3. Market concentration and single points of failure.)

**Despite attempting to debug and diagnose, developers are not able to prevent even overtly harmful behaviours across all circumstances in which general-purpose Al systems are used.**

Empirically, harmful behaviours have included revealing private or copyrighted information (221, 642*, 643); generating hate speech (225, 644); spreading social and political biases (239, 455, 645); pandering to user biases (238); hallucinating inaccurate content (46, 47*, 49); exhibiting vulnerabilities to a variety of attacks on their safety protections (108, 208, 209, 210, 211, 309, 646, 647, 648*); and assisting in overtly harmful tasks (368, 649, 650). Users can circumvent general-purpose Al model safeguards with relative ease (650, 651), for example through 'jailbreaking' techniques (see 3. Methodology to assess and understand general-purpose Al systems). While some have called for safety measures that rule out all overtly harmful behaviours across all situations (626), current general-purpose Al development fails to meet a lower standard than this: ruling out any specific overtly harmful behaviour across foreseeable situations (such as situations where users try to jailbreak a model.)

## Al developers are working to create 'agent' general-purpose Al systems that can accomplish tasks with little to no human involvement, which could increase risks of accident and malicious use

Today, general-purpose Al systems are primarily used directly as tools by humans. For example, a human may ask a chatbot to write computer code to help them accomplish a task, which naturally requires a 'human in the loop'. However, developers are increasingly designing systems that allow general-purpose Al systems to act autonomously, by controlling software tools such as a web browser or controlling the execution of code rather than only writing code. This enables some forms of reasoning about problems, constructing plans, and executing plans step-by-step (13, 26, 188, 189, 652, 653, 654, 655*, 656). Such systems have included web-browsing virtual agents (657*), research assistants (415), and writing, fixing, and running code autonomously (441).

The main purpose of general-purpose Al 'agents' is to reduce the need for human involvement and oversight, allowing for faster and cheaper applications of general-purpose Al. This property also reduces human oversight, potentially increasing the risk of accidents (see 4.1 Malicious use risks), and allowing the automation workflows for malicious uses (see 4.2.1 Risks from product functionality issues use) while also being relevant to the risk of loss of control (see 4.2.3 Loss of control) (481*, 658).

**General-purpose Al agents have early forms of many autonomous capabilities, but lack reliability at performing complex tasks autonomously.** Current state-of-the-art general-purpose Al systems are capable of autonomously executing many simple tasks, but some evaluations have shown that they struggle with more complex ones (183, 509). They are particularly unreliable at performing tasks that involve many steps. As of 2023, general-purpose Al agents scored low in benchmarks designed to measure their performance on complex and economically useful tasks (183, 441). However, given current efforts and growing investment in developing general-purpose Al systems with greater autonomy, the capabilities of general-purpose Al agents could continue to increase.

**Although current general-purpose Al agents are unreliable, advancement has been rapid.** For example, for two challenging benchmarks for general-purpose Al agents that measure general
problem-solving (183) and autonomous software engineering (441) accuracy has improved over a period of a few months by factors of 2.2x and 7.1x respectively compared to strong baselines using GPT-4. The performance of the general-purpose Al models (such as LLMs) which underlie these general-purpose Al agents is key to how reliable they are, so newer generations of general-purpose Al models typically increase agent capabilities. One potential way to increase agent capabilities could be through combining LLMs with search and planning methods. In board games like Go and Stratego, combining deep learning with methods like Monte Carlo Tree Search (MCTS) and self-play has led to above-human level performance (659, 660). Early work on combining LLMs with search has yielded improvements in simple settings (661). However, careful monitoring of general-purpose Al agent capabilities is needed to assess many of the risks discussed in 4. Risks.

## 4.4.2 Cross-cutting societal risk factors

### KEY INFORMATION

* This section covers four cross-cutting societal risk factors – non-technical aspects of general-purpose Al development and deployment that each contribute to many risks from general-purpose Al:

a. Al developers competing for market share may have limited incentives to invest in mitigating risks.

b. As general-purpose Al advances rapidly, regulatory or enforcement efforts can struggle to keep pace.

c. Lack of transparency makes liability harder to determine, potentially hindering governance and enforcement.

d. It is very difficult to track how general-purpose Al models and systems are trained, deployed and used.

Risk factors are distinct from risks – they are conditions that increase the likelihood and/or impact of risks occurring. There are societal aspects of general-purpose Al development and deployment that increase not one but several general-purpose Al risks. This section discusses these ‘cross-cutting societal risk factors’.

**General-purpose Al developers, who are competing for market share in a dynamic market where getting a product out quickly is vital, may have limited incentives to invest in mitigating risks.** The one-time cost of developing a state-of-the-art general-purpose Al model is very high, while the marginal costs of distributing such a model to (additional) users are relatively low. This can lead to 'winner takes all' dynamics, creating strong incentives for developers to build the most capable model at any cost, because doing so might allow immediately capturing a large market share. In recent years, there has been intense competition between general-purpose Al developers to rapidly build and deploy models. This has raised concern about potential 'race to the bottom' scenarios, where actors compete to develop general-purpose Al models as quickly as possible while under-investing in measures to ensure safety and ethics (662, 663). This could contribute to situations in which it is challenging for general-purpose Al developers to commit unilaterally to stringent safety standards, as doing so might put them at a competitive disadvantage (664). Similar dynamics can also occur at the international level regarding regulation efforts. Without global coordination regarding the regulation of general-purpose Al, a regulatory 'race to the bottom' could see countries attempt to attract Al companies through lax regulation that might be insufficient for ensuring safety domestically and abroad, a dynamic that has been described for several types of regulation like labour law (665).

**As general-purpose Al markets advance rapidly, regulatory or enforcement efforts can struggle to keep pace.** A recurring theme in the discourse on general-purpose Al risk is the mismatch between
problem-solving (183) and autonomous software engineering (441) accuracy has improved over a period of a few months by factors of 2.2x and 7.1x respectively compared to strong baselines using GPT-4. The performance of the general-purpose Al models (such as LLMs) which underlie these general-purpose Al agents is key to how reliable they are, so newer generations of general-purpose Al models typically increase agent capabilities. One potential way to increase agent capabilities could be through combining LLMs with search and planning methods. In board games like Go and Stratego, combining deep learning with methods like Monte Carlo Tree Search (MCTS) and self-play has led to above-human level performance (659, 660). Early work on combining LLMs with search has yielded improvements in simple settings (661). However, careful monitoring of general-purpose Al agent capabilities is needed to assess many of the risks discussed in 4. Risks.

## 4.4.2 Cross-cutting societal risk factors

### KEY INFORMATION

* This section covers four cross-cutting societal risk factors – non-technical aspects of general-purpose Al development and deployment that each contribute to many risks from general-purpose Al:

a. Al developers competing for market share may have limited incentives to invest in mitigating risks.

b. As general-purpose Al advances rapidly, regulatory or enforcement efforts can struggle to keep pace.

c. Lack of transparency makes liability harder to determine, potentially hindering governance and enforcement.

d. It is very difficult to track how general-purpose Al models and systems are trained, deployed and used.

Risk factors are distinct from risks – they are conditions that increase the likelihood and/or impact of risks occurring. There are societal aspects of general-purpose Al development and deployment that increase not one but several general-purpose Al risks. This section discusses these ‘cross-cutting societal risk factors’.

**General-purpose Al developers, who are competing for market share in a dynamic market where getting a product out quickly is vital, may have limited incentives to invest in mitigating risks.** The one-time cost of developing a state-of-the-art general-purpose Al model is very high, while the marginal costs of distributing such a model to (additional) users are relatively low. This can lead to 'winner takes all' dynamics, creating strong incentives for developers to build the most capable model at any cost, because doing so might allow immediately capturing a large market share. In recent years, there has been intense competition between general-purpose Al developers to rapidly build and deploy models. This has raised concern about potential 'race to the bottom' scenarios, where actors compete to develop general-purpose Al models as quickly as possible while under-investing in measures to ensure safety and ethics (662, 663). This could contribute to situations in which it is challenging for general-purpose Al developers to commit unilaterally to stringent safety standards, as doing so might put them at a competitive disadvantage (664). Similar dynamics can also occur at the international level regarding regulation efforts. Without global coordination regarding the regulation of general-purpose Al, a regulatory 'race to the bottom' could see countries attempt to attract Al companies through lax regulation that might be insufficient for ensuring safety domestically and abroad, a dynamic that has been described for several types of regulation like labour law (665).

**As general-purpose Al markets advance rapidly, regulatory or enforcement efforts can struggle to keep pace.** A recurring theme in the discourse on general-purpose Al risk is the mismatch between
the pace of technological innovation and the development of governance structures (666). While existing legal and governance frameworks apply to some uses of general-purpose AI systems and several jurisdictions (like the European Union, China, the USA or Canada) have initiated or completed efforts to regulate AI and general-purpose AI specifically, there often remain regulatory gaps. In a market that is as fast-moving as the general-purpose AI market currently is, it is very difficult to fill such gaps ex-post, because by the time a regulatory fix is implemented it might already be outdated. Policymakers therefore face the challenge of creating a flexible regulatory environment that ensures the pace of general-purpose AI development and deployment remains manageable from a public safety perspective.

**General-purpose AI systems' inherent lack of transparency makes legal liability hard to determine, potentially hindering governance and enforcement.** How current legal frameworks apply in cases where it is suspected that a general-purpose AI system caused harm is often unclear. This raises many issues for accountability, liability and justice. In principle, people and corporate entities are held accountable, not the technology, which is why many critical areas maintain a 'human in the loop' policy. However, tracing harm back to the responsible individuals who developed or deployed the AI is very challenging (667, 668, 669), as is gathering evidence of error. This accountability issue is compounded by the opaque nature of proprietary general-purpose AI models, where commercially sensitive training data, methodologies, and decision-making processes are usually not open to public scrutiny and there is a lack of widely shared standard operating procedures for using and interpreting AI systems (291, 292, 293, 294*, 295, 670, 671). Some also argue that general-purpose AI systems can exhibit 'emergent' behaviours that were not explicitly programmed or intended by their developers, raising questions about who should be held liable for resulting harm. The distributed nature of general-purpose AI development, involving multiple actors such as data providers, model trainers, and deployers, also makes it challenging to assign liability to a single entity (669).

**It is very difficult to track how general-purpose AI models and systems are trained, deployed and used.** Tracking the use of general-purpose AI models and systems is not only important for establishing liability for potential harms caused by the use of general-purpose AI models and systems, but also for monitoring and evidencing malicious use, and noticing malfunctions (658, 672, 673). Comprehensive safety governance is common in safety-critical fields such as automotive, pharmaceuticals, and energy (674, 675, 676, 677), but it often relies on broadly accepted standards that are currently missing in general-purpose AI governance.
the pace of technological innovation and the development of governance structures (666). While existing legal and governance frameworks apply to some uses of general-purpose AI systems and several jurisdictions (like the European Union, China, the USA or Canada) have initiated or completed efforts to regulate AI and general-purpose AI specifically, there often remain regulatory gaps. In a market that is as fast-moving as the general-purpose AI market currently is, it is very difficult to fill such gaps ex-post, because by the time a regulatory fix is implemented it might already be outdated. Policymakers therefore face the challenge of creating a flexible regulatory environment that ensures the pace of general-purpose AI development and deployment remains manageable from a public safety perspective.

**General-purpose AI systems' inherent lack of transparency makes legal liability hard to determine, potentially hindering governance and enforcement.** How current legal frameworks apply in cases where it is suspected that a general-purpose AI system caused harm is often unclear. This raises many issues for accountability, liability and justice. In principle, people and corporate entities are held accountable, not the technology, which is why many critical areas maintain a 'human in the loop' policy. However, tracing harm back to the responsible individuals who developed or deployed the AI is very challenging (667, 668, 669), as is gathering evidence of error. This accountability issue is compounded by the opaque nature of proprietary general-purpose AI models, where commercially sensitive training data, methodologies, and decision-making processes are usually not open to public scrutiny and there is a lack of widely shared standard operating procedures for using and interpreting AI systems (291, 292, 293, 294*, 295, 670, 671). Some also argue that general-purpose AI systems can exhibit 'emergent' behaviours that were not explicitly programmed or intended by their developers, raising questions about who should be held liable for resulting harm. The distributed nature of general-purpose AI development, involving multiple actors such as data providers, model trainers, and deployers, also makes it challenging to assign liability to a single entity (669).

**It is very difficult to track how general-purpose AI models and systems are trained, deployed and used.** Tracking the use of general-purpose AI models and systems is not only important for establishing liability for potential harms caused by the use of general-purpose AI models and systems, but also for monitoring and evidencing malicious use, and noticing malfunctions (658, 672, 673). Comprehensive safety governance is common in safety-critical fields such as automotive, pharmaceuticals, and energy (674, 675, 676, 677), but it often relies on broadly accepted standards that are currently missing in general-purpose AI governance.
# 5 Technical approaches to mitigate risks

This section of the report discusses technical approaches to increase general-purpose Al safety through mitigating general-purpose Al-related risks: to reduce the scale of harms or the likelihood of their occurrence. This report ultimately finds that, while there are many technical approaches that reduce risk, existing methods are insufficient to prove that systems are safe.

The scope of this report does not include nontechnical (political, legal, or social) interventions, but these are equally important for addressing risks. Moreover, technical and nontechnical aspects of managing risks from general-purpose Al are highly intertwined; no technical solutions are implemented in a vacuum. The last months and years have seen increased interest in Al regulation from policymakers, and several jurisdictions (like the European Union, China, the USA or Canada) have initiated or completed efforts to regulate Al and general-purpose Al specifically. Effective approaches will require the resources and political will to implement technical solutions as well as an interplay between multiple technical and nontechnical safeguards against harms from general-purpose Al.

## 5.1 Risk management and safety engineering

### KEY INFORMATION

* Developing and incentivising systematic risk management practices for general-purpose Al is difficult. This is because current general-purpose Al is progressing rapidly, is not well-understood, and has a wide range of applications. Methodologies for assessing general-purpose Al risk are too nascent for good quantitative analysis of risk to be available.

* While many other fields offer lessons for how such approaches could be developed, there are currently no well-established risk management and safety engineering practices for general-purpose Al systems.

* Since no single existing method can provide full or partial guarantees of safety, a practical strategy is *defence in depth* – layering multiple risk mitigation measures. This is a common way to manage technological risks.

* An important consideration for effective risk management of general-purpose Al is who to involve in the process in order to identify and assess high-priority risks. This can include experts from multiple domains but also representatives of impacted communities.

**Risk** is the combination of the probability of an occurrence of harm and the severity of that harm if it occurs (339). This technically includes both positive and negative outcomes, but in common usage, the focus is on the negative outcomes. Some (but not all) of the common risks associated with general-purpose Al are described in 4. Risks.

Risk increases with the severity of the potential harm and the probability of the harm materialising. Whether and to what extent the outcomes of a system are considered undesirable has to be conceptualised with respect to contextual human values. Various stakeholders may disagree how undesirable any particular outcome is.

**Risk surface/exposure:** The risk surface of a technology consists of all the ways it can cause harm through accidents or malicious use. The more general-purpose a technology is, the more extensive its risk exposure is expected to be. General-purpose Al models can be fine-tuned and applied in numerous application domains and used by a wide variety of users (4.4.1. Cross-cutting technical risk
# 5 Technical approaches to mitigate risks

This section of the report discusses technical approaches to increase general-purpose Al safety through mitigating general-purpose Al-related risks: to reduce the scale of harms or the likelihood of their occurrence. This report ultimately finds that, while there are many technical approaches that reduce risk, existing methods are insufficient to prove that systems are safe.

The scope of this report does not include nontechnical (political, legal, or social) interventions, but these are equally important for addressing risks. Moreover, technical and nontechnical aspects of managing risks from general-purpose Al are highly intertwined; no technical solutions are implemented in a vacuum. The last months and years have seen increased interest in Al regulation from policymakers, and several jurisdictions (like the European Union, China, the USA or Canada) have initiated or completed efforts to regulate Al and general-purpose Al specifically. Effective approaches will require the resources and political will to implement technical solutions as well as an interplay between multiple technical and nontechnical safeguards against harms from general-purpose Al.

## 5.1 Risk management and safety engineering

### KEY INFORMATION

* Developing and incentivising systematic risk management practices for general-purpose Al is difficult. This is because current general-purpose Al is progressing rapidly, is not well-understood, and has a wide range of applications. Methodologies for assessing general-purpose Al risk are too nascent for good quantitative analysis of risk to be available.

* While many other fields offer lessons for how such approaches could be developed, there are currently no well-established risk management and safety engineering practices for general-purpose Al systems.

* Since no single existing method can provide full or partial guarantees of safety, a practical strategy is *defence in depth* – layering multiple risk mitigation measures. This is a common way to manage technological risks.

* An important consideration for effective risk management of general-purpose Al is who to involve in the process in order to identify and assess high-priority risks. This can include experts from multiple domains but also representatives of impacted communities.

**Risk** is the combination of the probability of an occurrence of harm and the severity of that harm if it occurs (339). This technically includes both positive and negative outcomes, but in common usage, the focus is on the negative outcomes. Some (but not all) of the common risks associated with general-purpose Al are described in 4. Risks.

Risk increases with the severity of the potential harm and the probability of the harm materialising. Whether and to what extent the outcomes of a system are considered undesirable has to be conceptualised with respect to contextual human values. Various stakeholders may disagree how undesirable any particular outcome is.

**Risk surface/exposure:** The risk surface of a technology consists of all the ways it can cause harm through accidents or malicious use. The more general-purpose a technology is, the more extensive its risk exposure is expected to be. General-purpose Al models can be fine-tuned and applied in numerous application domains and used by a wide variety of users (4.4.1. Cross-cutting technical risk
factors), leading to extremely broad risk surfaces and exposure, challenging effective risk management.

**Risk management** consists of the *identification, assessment, and prioritisation* of risks and utilising resources to minimise, monitor, and control high-priority risks.

**System safety engineering** is defined very similarly but with an emphasis on the importance of the interactions of multiple parts of a larger system (678). In the case of AI, this approach entails taking into account all the constituent parts of a general-purpose AI system, as well as the broader context in which it operates. Industries such as finance, insurance, health, and cybersecurity have well-established risk management practices (679). The AI risk management framework by NIST (680) is among the few prominent recent efforts to come up with a framework of risk management specific to AI systems.

## 5.1.1 Risk assessment

When the scope of applicability and use of an AI system is narrow (e.g., consider spam filtering as an example), salient types of risk (e.g., the likelihood of false positives) can be measured with relatively high confidence. In contrast, assessing general-purpose AI models' risks, such as the generation of toxic language, is much more challenging, in part due to a lack of consensus on what should be considered toxic and the interplay between toxicity and contextual factors (including the prompt and the intention of the user).

There is a broad range of risk assessment techniques that could and are already being used with regard to general-purpose AI (216, 679), including evaluations, red-teaming, auditing, and qualitative evaluation of general-purpose AI systems (see 3. Methodology to assess and understand general-purpose AI systems). Other risk methods for assessment, some of which draw on established practices in other fields, include:

* Uplift studies, where the aim is to test how much more competent a human is at accomplishing a potentially harmful task when they have access to a general-purpose AI system versus when they do not (166*).

* Forecasting to inform high-stakes decisions (323*)

* Delphi studies aggregating forecasts from groups of relevant experts (681).

* Field testing aims to detect and document risks arising in the environment in which the model is to be used and the specific dangers posed in that setting.

* Benchmarking tasks and datasets to assess the prevalence and severity of specific types of risks (e.g., toxicity, certain forms of bias, grasp of scientific domains, dangerous capabilities).

Current risk assessment methodologies often fail to produce reliable assessments of the risk posed by general-purpose AI systems. Some of the key challenges of utilising such risk assessment methodologies for highly capable models are:

* Specifying the relevant/high-priority flaws and vulnerabilities is highly influenced by who is at the table and how the discussion is organised, meaning it is easy to miss or mis-define areas of concern.

* Guarding against potential malicious uses requires understanding likely and viable threats, including the estimation of resources available to malicious actors (e.g., compute, access, and expertise) and their incentives.

* The general-purpose nature of these technologies adds uncertainty with regard to their use in deployment. An example illustrating this point is applications involving open-ended interactions (e.g., chatbots), which can generate a vast array of potential outputs.

* The rapid pace of technological advancement (2.3.1 Recent trends in capabilities and 4.4.1 Cross-cutting technical risk factors) worsens the above challenges.
factors), leading to extremely broad risk surfaces and exposure, challenging effective risk management.

**Risk management** consists of the *identification, assessment, and prioritisation* of risks and utilising resources to minimise, monitor, and control high-priority risks.

**System safety engineering** is defined very similarly but with an emphasis on the importance of the interactions of multiple parts of a larger system (678). In the case of AI, this approach entails taking into account all the constituent parts of a general-purpose AI system, as well as the broader context in which it operates. Industries such as finance, insurance, health, and cybersecurity have well-established risk management practices (679). The AI risk management framework by NIST (680) is among the few prominent recent efforts to come up with a framework of risk management specific to AI systems.

## 5.1.1 Risk assessment

When the scope of applicability and use of an AI system is narrow (e.g., consider spam filtering as an example), salient types of risk (e.g., the likelihood of false positives) can be measured with relatively high confidence. In contrast, assessing general-purpose AI models' risks, such as the generation of toxic language, is much more challenging, in part due to a lack of consensus on what should be considered toxic and the interplay between toxicity and contextual factors (including the prompt and the intention of the user).

There is a broad range of risk assessment techniques that could and are already being used with regard to general-purpose AI (216, 679), including evaluations, red-teaming, auditing, and qualitative evaluation of general-purpose AI systems (see 3. Methodology to assess and understand general-purpose AI systems). Other risk methods for assessment, some of which draw on established practices in other fields, include:

* Uplift studies, where the aim is to test how much more competent a human is at accomplishing a potentially harmful task when they have access to a general-purpose AI system versus when they do not (166*).

* Forecasting to inform high-stakes decisions (323*)

* Delphi studies aggregating forecasts from groups of relevant experts (681).

* Field testing aims to detect and document risks arising in the environment in which the model is to be used and the specific dangers posed in that setting.

* Benchmarking tasks and datasets to assess the prevalence and severity of specific types of risks (e.g., toxicity, certain forms of bias, grasp of scientific domains, dangerous capabilities).

Current risk assessment methodologies often fail to produce reliable assessments of the risk posed by general-purpose AI systems. Some of the key challenges of utilising such risk assessment methodologies for highly capable models are:

* Specifying the relevant/high-priority flaws and vulnerabilities is highly influenced by who is at the table and how the discussion is organised, meaning it is easy to miss or mis-define areas of concern.

* Guarding against potential malicious uses requires understanding likely and viable threats, including the estimation of resources available to malicious actors (e.g., compute, access, and expertise) and their incentives.

* The general-purpose nature of these technologies adds uncertainty with regard to their use in deployment. An example illustrating this point is applications involving open-ended interactions (e.g., chatbots), which can generate a vast array of potential outputs.

* The rapid pace of technological advancement (2.3.1 Recent trends in capabilities and 4.4.1 Cross-cutting technical risk factors) worsens the above challenges.
Red teaming, for example, only assesses *whether* a model can produce some output, not the extent to which it will do so in real-world contexts nor how harmful doing so would be. Instead, they tend to provide qualitative information that informs judgments on what risk the system poses.

## 5.1.2 Risk management

There are a range of risk assessment techniques that could and are already being used with regard to general-purpose AI (see 3. Methodology to assess and understand general-purpose AI systems). To address the limitations of these existing tools, researchers can look to established practice in risk management in other domains. Some of the common risk management tools in other safety-critical industries are:

* Planned audits and inspection.

* Ensuring traceability using standardised documentation.

* Redundant defence mechanisms against critical risks and failures.

* Risk management guidelines prescribing processes, evaluations, and deliverables at all stages of a safety-critical system's life cycle.

Similar ideas have been proposed for managing the risks associated with general-purpose AI systems. Many of these existing approaches are not directly applicable to highly capable general-purpose AI models, or their efficacy is not well-studied, but efforts are underway to extend existing guidelines to Generative AI.

**Safety and reliability engineering:** The practice of safety engineering has a long history in various safety-critical engineering systems, such as the construction of bridges, skyscrapers, aircraft, and nuclear power plants. At a high level, safety engineering assures that a *life-critical* system acts as intended and with minimal harm, even when certain components of the system fail. Reliability engineering is broader in scope and address non-critical failures as well. These approaches offer several techniques that could be useful for risk assessment in general-purpose AI:

* Safety by Design (SbD) is an approach that centres user safety in the design and development of products and services. For general-purpose AI products and services, this may take the form of restricting access to the model (e.g., through limiting the length of user interactions with the model).

* Safety analysis aims to understand the causal dependencies between the functionality of the individual components and the overall system so that component failures, which can lead to system-level hazards (e.g., aircraft crash or nuclear reactor core meltdown), can be anticipated and prevented to the extent possible.

* 'Safety of the intended function' (SOTIF) approaches require engineers to provide evidence that the system is safe when operating as intended.

* Some risk assessment methods, such as for the nuclear power sector, leverage mathematical models that are designed to quantify risk as a function of various design and engineering choices, accompanied by quantitative risk thresholds set by regulators (682).<fnref n="5" /> A key advantage of this approach is that it lets a publicly accountable body define what risk is considered acceptable, in a way that is legible to the public and external experts.

However, translating best practices from these fields to general-purpose AI is difficult. Quantitative risk assessment methodologies for general-purpose AI are very nascent and it is not yet clear how quantitative safety guarantees could be obtained. Experience of other risk assessments in general-

<footnote n="5">⁵ For example, some regulatory commissions mandate that nuclear reactor operators produce probabilistic risk assessments, and ensures that the estimated risk of certain events are kept below specified thresholds. For example, operators are mandated to keep the estimated chance of radioactive release beyond the power plant below one in a million (683).</footnote>
Red teaming, for example, only assesses *whether* a model can produce some output, not the extent to which it will do so in real-world contexts nor how harmful doing so would be. Instead, they tend to provide qualitative information that informs judgments on what risk the system poses.

## 5.1.2 Risk management

There are a range of risk assessment techniques that could and are already being used with regard to general-purpose AI (see 3. Methodology to assess and understand general-purpose AI systems). To address the limitations of these existing tools, researchers can look to established practice in risk management in other domains. Some of the common risk management tools in other safety-critical industries are:

* Planned audits and inspection.

* Ensuring traceability using standardised documentation.

* Redundant defence mechanisms against critical risks and failures.

* Risk management guidelines prescribing processes, evaluations, and deliverables at all stages of a safety-critical system's life cycle.

Similar ideas have been proposed for managing the risks associated with general-purpose AI systems. Many of these existing approaches are not directly applicable to highly capable general-purpose AI models, or their efficacy is not well-studied, but efforts are underway to extend existing guidelines to Generative AI.

**Safety and reliability engineering:** The practice of safety engineering has a long history in various safety-critical engineering systems, such as the construction of bridges, skyscrapers, aircraft, and nuclear power plants. At a high level, safety engineering assures that a *life-critical* system acts as intended and with minimal harm, even when certain components of the system fail. Reliability engineering is broader in scope and address non-critical failures as well. These approaches offer several techniques that could be useful for risk assessment in general-purpose AI:

* Safety by Design (SbD) is an approach that centres user safety in the design and development of products and services. For general-purpose AI products and services, this may take the form of restricting access to the model (e.g., through limiting the length of user interactions with the model).

* Safety analysis aims to understand the causal dependencies between the functionality of the individual components and the overall system so that component failures, which can lead to system-level hazards (e.g., aircraft crash or nuclear reactor core meltdown), can be anticipated and prevented to the extent possible.

* 'Safety of the intended function' (SOTIF) approaches require engineers to provide evidence that the system is safe when operating as intended.

* Some risk assessment methods, such as for the nuclear power sector, leverage mathematical models that are designed to quantify risk as a function of various design and engineering choices, accompanied by quantitative risk thresholds set by regulators (682).<fnref n="5" /> A key advantage of this approach is that it lets a publicly accountable body define what risk is considered acceptable, in a way that is legible to the public and external experts.

However, translating best practices from these fields to general-purpose AI is difficult. Quantitative risk assessment methodologies for general-purpose AI are very nascent and it is not yet clear how quantitative safety guarantees could be obtained. Experience of other risk assessments in general-

<footnote n="5">⁵ For example, some regulatory commissions mandate that nuclear reactor operators produce probabilistic risk assessments, and ensures that the estimated risk of certain events are kept below specified thresholds. For example, operators are mandated to keep the estimated chance of radioactive release beyond the power plant below one in a million (683).</footnote>
purpose Al suggests that many areas of concern may not be amenable to quantification (for example, bias and misinformation). If quantitative risk assessments are too uncertain to be relied on, they may still be an important complement to inform high-stakes decisions, clarify the assumptions used to assess risk levels and evaluate the appropriateness of other decision procedures (e.g. those tied to model capabilities). Further, "risk" and "safety" are contentious concepts – for instance, one might ask "safe to whom?" – which may require the involvement of diverse sets of experts and potentially impacted populations (307).

While there are currently no well-established safety engineering practices for general-purpose Al systems, the pipeline-aware approach to mitigating Al's harm takes inspiration from safety engineering and proposes scrutinising numerous design choices made through the general-purpose Al lifecycle, from ideation and problem formulation, to design, development, and deployment, both as individual components and in relation to one another (684, 685). Further work is needed to extend these ideas from traditional Al to generative Al.

**Safety cases:** Developers of safety-critical technologies such as aviation, medical devices, and defence software are required to make safety cases, which put the burden of proof on the developer to demonstrate that their product does not exceed maximum risk thresholds set by the regulator (686, 687, 688, 689). A safety case is a structured argument supported by evidence, where the developer identifies hazards, models risk scenarios, and evaluates the mitigations taken. Safety cases would be easier to make for general-purpose Al systems with limited capabilities since less capable models often pose less risk. Thereby, they are robust to scenarios of both slow and rapid progress in general-purpose Al capabilities (see 2.4 Capability progress in coming years). Safety cases leverage the technical expertise of the technology developer but still require that the regulator (or a suitable third party) has the technical expertise to evaluate safety cases. Safety cases often address only a subset of risks and threat models, leaving out important ones (690, 691). One mitigation to these limitations is to review safety cases alongside risk cases produced by a red team of third-party experts (689).

**The 'Swiss-cheese' model for general-purpose Al safety engineering:** The general-purpose, rapidly evolving, and inscrutable nature of highly capable models makes it increasingly difficult to develop, assess, and incentivise systematic risk management practices. Effectively managing the risks of highly capable general-purpose Al systems might therefore require the involvement of multiple stakeholder groups, including experts from multiple domains and impacted communities, to identify and assess high-priority risks. Further, it suggests that no single line of defence should be relied upon. Instead, multiple independent and overlapping layers of defence against those risks may be advisable, such that if one fails, others will still be effective. This is sometimes referred to as the Swiss Cheese model of defence in depth (692).

**Current risk management among general-purpose Al developers:** Though not universally adhered to, it is common practice to test models for some dangerous capabilities ahead of release, including via red-teaming and benchmarking, and publishing those results in a 'model card' (257). Further, some developers have internal decision-making panels that deliberate on how to safely and responsibly release new systems. An increasingly common practice among these developers is to constrain decisions through voluntary pre-defined capabilities thresholds (693*, 694*). Such thresholds determine that specific model capabilities must be met with specific mitigations that are meant to keep risks to an acceptable level (682). Such capabilities thresholds have the advantage of being observable in advance of capabilities being developed. However, more work is needed to assess whether adhering to some specific set of thresholds indeed does keep risk to an acceptable level and to assess the practicality of accurately specifying appropriate thresholds in advance.
purpose Al suggests that many areas of concern may not be amenable to quantification (for example, bias and misinformation). If quantitative risk assessments are too uncertain to be relied on, they may still be an important complement to inform high-stakes decisions, clarify the assumptions used to assess risk levels and evaluate the appropriateness of other decision procedures (e.g. those tied to model capabilities). Further, "risk" and "safety" are contentious concepts – for instance, one might ask "safe to whom?" – which may require the involvement of diverse sets of experts and potentially impacted populations (307).

While there are currently no well-established safety engineering practices for general-purpose Al systems, the pipeline-aware approach to mitigating Al's harm takes inspiration from safety engineering and proposes scrutinising numerous design choices made through the general-purpose Al lifecycle, from ideation and problem formulation, to design, development, and deployment, both as individual components and in relation to one another (684, 685). Further work is needed to extend these ideas from traditional Al to generative Al.

**Safety cases:** Developers of safety-critical technologies such as aviation, medical devices, and defence software are required to make safety cases, which put the burden of proof on the developer to demonstrate that their product does not exceed maximum risk thresholds set by the regulator (686, 687, 688, 689). A safety case is a structured argument supported by evidence, where the developer identifies hazards, models risk scenarios, and evaluates the mitigations taken. Safety cases would be easier to make for general-purpose Al systems with limited capabilities since less capable models often pose less risk. Thereby, they are robust to scenarios of both slow and rapid progress in general-purpose Al capabilities (see 2.4 Capability progress in coming years). Safety cases leverage the technical expertise of the technology developer but still require that the regulator (or a suitable third party) has the technical expertise to evaluate safety cases. Safety cases often address only a subset of risks and threat models, leaving out important ones (690, 691). One mitigation to these limitations is to review safety cases alongside risk cases produced by a red team of third-party experts (689).

**The 'Swiss-cheese' model for general-purpose Al safety engineering:** The general-purpose, rapidly evolving, and inscrutable nature of highly capable models makes it increasingly difficult to develop, assess, and incentivise systematic risk management practices. Effectively managing the risks of highly capable general-purpose Al systems might therefore require the involvement of multiple stakeholder groups, including experts from multiple domains and impacted communities, to identify and assess high-priority risks. Further, it suggests that no single line of defence should be relied upon. Instead, multiple independent and overlapping layers of defence against those risks may be advisable, such that if one fails, others will still be effective. This is sometimes referred to as the Swiss Cheese model of defence in depth (692).

**Current risk management among general-purpose Al developers:** Though not universally adhered to, it is common practice to test models for some dangerous capabilities ahead of release, including via red-teaming and benchmarking, and publishing those results in a 'model card' (257). Further, some developers have internal decision-making panels that deliberate on how to safely and responsibly release new systems. An increasingly common practice among these developers is to constrain decisions through voluntary pre-defined capabilities thresholds (693*, 694*). Such thresholds determine that specific model capabilities must be met with specific mitigations that are meant to keep risks to an acceptable level (682). Such capabilities thresholds have the advantage of being observable in advance of capabilities being developed. However, more work is needed to assess whether adhering to some specific set of thresholds indeed does keep risk to an acceptable level and to assess the practicality of accurately specifying appropriate thresholds in advance.
## 5.2 Training more trustworthy models

### KEY INFORMATION

* There is progress in training general-purpose AI systems to function more safely, but there is currently no approach that can ensure that general-purpose AI systems will be harmless in all circumstances.

* Companies have proposed strategies to train general-purpose AI systems to be more helpful and harmless: however, the viability and reliability of these approaches for such advanced systems remains limited.

* Current techniques for aligning the behaviour of general-purpose AI systems with developer intentions rely heavily on data from humans such as human feedback. This makes them subject to human error and bias. Increasing the quantity and quality of this feedback is an avenue for improvement.

* Developers train models to be more robust to inputs that are designed to make them fail ('adversarial training'). Despite this, adversaries can typically find alternative inputs that reduce the effectiveness of safeguards with low to moderate effort.

* Limiting a general-purpose AI system's capabilities to a specific use case can help to reduce risks from unforeseen failures or malicious use.

* Researchers are beginning to learn to analyse the inner workings of general-purpose AI models. Progress in this area could help developers understand and edit general-purpose AI model functionality more reliably.

* Researchers are exploring how to obtain AI systems that are safe by design or provably safe, although many open problems remain to scale these methods to general-purpose AI systems.

### 5.2.1 Aligning general-purpose AI systems with developer intentions

‘AI alignment’ refers to the challenge of making general-purpose AI systems act in accordance with their developer’s goals and interests (see 5.4 Technical approaches to fairness and representation in general-purpose AI systems for a discussion of the challenges to alignment posed by conflicting values of different stakeholders).

#### 5.2.1.1 Two alignment challenges

There are two challenges involved in training aligned general-purpose AI systems: firstly, ensuring that they are trained with an objective that incentivises the intended goals; and secondly, ensuring that the outputs translate from their training contexts to the real world as intended, especially in high-stakes situations.

**It is challenging to precisely specify an objective for general-purpose AI systems in a way that does not unintentionally incentivise undesirable behaviours.** Currently, researchers do not know how to specify abstract human preferences and values in a way that can be used to train general-purpose AI systems. Moreover, given the complex socio-technical relationships embedded in general-purpose AI systems, it is not clear whether such specification is possible. General-purpose AI systems are generally trained to optimise for objectives that are imperfect proxies for the developer's true goals (634). For example, AI chatbots are often trained to produce text that will be rated positively by human evaluators, but user approval is an imperfect proxy for user benefit. Research has shown that
## 5.2 Training more trustworthy models

### KEY INFORMATION

* There is progress in training general-purpose Al systems to function more safely, but there is currently no approach that can ensure that general-purpose Al systems will be harmless in all circumstances.

* Companies have proposed strategies to train general-purpose Al systems to be more helpful and harmless: however, the viability and reliability of these approaches for such advanced systems remains limited.

* Current techniques for aligning the behaviour of general-purpose Al systems with developer intentions rely heavily on data from humans such as human feedback. This makes them subject to human error and bias. Increasing the quantity and quality of this feedback is an avenue for improvement.

* Developers train models to be more robust to inputs that are designed to make them fail ('adversarial training'). Despite this, adversaries can typically find alternative inputs that reduce the effectiveness of safeguards with low to moderate effort.

* Limiting a general-purpose Al system's capabilities to a specific use case can help to reduce risks from unforeseen failures or malicious use.

* Researchers are beginning to learn to analyse the inner workings of general-purpose Al models. Progress in this area could help developers understand and edit general-purpose Al model functionality more reliably.

* Researchers are exploring how to obtain Al systems that are safe by design or provably safe, although many open problems remain to scale these methods to general-purpose Al systems.

### 5.2.1 Aligning general-purpose Al systems with developer intentions

‘Al alignment’ refers to the challenge of making general-purpose Al systems act in accordance with their developer’s goals and interests (see 5.4 Technical approaches to fairness and representation in general-purpose Al systems for a discussion of the challenges to alignment posed by conflicting values of different stakeholders).

#### 5.2.1.1 Two alignment challenges

There are two challenges involved in training aligned general-purpose Al systems: firstly, ensuring that they are trained with an objective that incentivises the intended goals; and secondly, ensuring that the outputs translate from their training contexts to the real world as intended, especially in high-stakes situations.

**It is challenging to precisely specify an objective for general-purpose Al systems in a way that does not unintentionally incentivise undesirable behaviours.** Currently, researchers do not know how to specify abstract human preferences and values in a way that can be used to train general-purpose Al systems. Moreover, given the complex socio-technical relationships embedded in general-purpose Al systems, it is not clear whether such specification is possible. General-purpose Al systems are generally trained to optimise for objectives that are imperfect proxies for the developer’s true goals (634). For example, Al chatbots are often trained to produce text that will be rated positively by human evaluators, but user approval is an imperfect proxy for user benefit. Research has shown that
several widely-used chatbots sometimes match their stated views to a user's views regardless of
truth (238). This is an ongoing challenge for general-purpose Al and similar Al systems (489, 633, 634,
695*).

**Ensuring general-purpose Al systems learn behaviours that translate from their training contexts to real-world, high-stakes deployment contexts is also highly challenging.** Just as general-purpose Al systems are trained to optimise for imperfect proxy goals, the training context can also fail to adequately represent the real-world situations they will encounter after they are deployed. In such cases, general-purpose Al systems can still learn to take harmful actions even if they are trained with correct human-provided feedback (636, 637, 638). For example, some researchers have found that chatbots are more likely to take harmful actions in languages that are under-represented in their training data (309). Using more multilingual data and oversight may be able to more mitigate this type of failure.

5.2.1.2 Alignment techniques

To elicit the desired behaviours from state-of-the-art general-purpose Al systems, developers train
models using human oversight. Improving performance in real-world contexts benefits from large
amounts of data.

**State-of-the-art alignment techniques rely on feedback or demonstrations from humans and, as such, are constrained by human error and bias.** As discussed in [2.1. How does General-Purpose Al gain its capabilities?](https://arxiv.org/abs/1905.05706), developers fine-tune state-of-the-art general-purpose Al systems using a large amount of human involvement. In practice, this involves techniques that leverage human-generated examples of desired actions (18) or human-generated feedback on examples from models (19, 20, 21*, 303). This is done at scale, making it labour-intensive and expensive. However, human attention, comprehension, and trustworthiness are not perfect (303), which limits the quality of the resulting general-purpose Al systems (696, 697*, 698). Even slight imperfections in feedback from humans can be amplified when used to train highly capable systems with potentially serious consequences (see [4.1. Malicious use risks](https://arxiv.org/abs/1608.02672) and [4.2.3. Loss of control](https://arxiv.org/abs/1803.05746)).

Improving the quality and quantity of human oversight can help to train more aligned models.

Some research has shown that using richer, more detailed forms of feedback from humans can
provide a better oversight signal, but at the cost of increased time and effort for data collection (699,
700, 701). To gather larger datasets, leveraging general-purpose Al systems to partially automate the
feedback process can greatly increase the volume of data (158*, 702*). However, in practice, the
amount of explicit human oversight used during finetuning is very small compared to the trillions of
data points used in pre-training on internet data and may, therefore, be unable to fully remove harmful
knowledge or capabilities from pre-training. Short of rethinking the way state-of-the-art general-
purpose Al systems are trained, improving fine-tuning feedback data is unlikely to be a solution on its
own.

**Maintaining uncertainty over goals can reduce risky actions.** Some researchers have proposed methods that involve incorporating uncertainty into the goals that general-purpose Al systems learn to pursue (703, 704, 705, 706*, 707, 708). By requiring general-purpose Al systems to act in a way that respects uncertainty about their objectives, these methods can reduce the risk of unexpected actions and encourage information-seeking or deference to humans in response to ambiguity. However, these methods have yet to be incorporated into state-of-the-art, highly capable Al.

**Some researchers are working toward safe-by-design approaches which might be able to provide quantitative safety guarantees.** Similar to the above approaches to estimate uncertainty about goals and predictions, it may be possible to design Al systems that are constructed to achieve quantified levels of safety (626). The advantage of mathematical guarantees and bounds is that they may provide safety assurances even outside of the domain where the Al has been trained and tested, in contrast with empirical trial-and-error methods that are currently the standard for designing deep learning systems. Currently, however, practically-useful, provable guarantees of safety are not possible
several widely-used chatbots sometimes match their stated views to a user's views regardless of
truth (238). This is an ongoing challenge for general-purpose Al and similar Al systems (489, 633, 634,
695*).

**Ensuring general-purpose Al systems learn behaviours that translate from their training contexts to real-world, high-stakes deployment contexts is also highly challenging.** Just as general-purpose Al systems are trained to optimise for imperfect proxy goals, the training context can also fail to adequately represent the real-world situations they will encounter after they are deployed. In such cases, general-purpose Al systems can still learn to take harmful actions even if they are trained with correct human-provided feedback (636, 637, 638). For example, some researchers have found that chatbots are more likely to take harmful actions in languages that are under-represented in their training data (309). Using more multilingual data and oversight may be able to more mitigate this type of failure.

5.2.1.2 Alignment techniques

To elicit the desired behaviours from state-of-the-art general-purpose Al systems, developers train
models using human oversight. Improving performance in real-world contexts benefits from large
amounts of data.

**State-of-the-art alignment techniques rely on feedback or demonstrations from humans and, as such, are constrained by human error and bias.** As discussed in [2.1. How does General-Purpose Al gain its capabilities?](https://arxiv.org/abs/1905.05706), developers fine-tune state-of-the-art general-purpose Al systems using a large amount of human involvement. In practice, this involves techniques that leverage human-generated examples of desired actions (18) or human-generated feedback on examples from models (19, 20, 21*, 303). This is done at scale, making it labour-intensive and expensive. However, human attention, comprehension, and trustworthiness are not perfect (303), which limits the quality of the resulting general-purpose Al systems (696, 697*, 698). Even slight imperfections in feedback from humans can be amplified when used to train highly capable systems with potentially serious consequences (see [4.1. Malicious use risks](https://arxiv.org/abs/1606.01397) and [4.2.3. Loss of control](https://arxiv.org/abs/1803.05746)).

Improving the quality and quantity of human oversight can help to train more aligned models.

Some research has shown that using richer, more detailed forms of feedback from humans can
provide a better oversight signal, but at the cost of increased time and effort for data collection (699,
700, 701). To gather larger datasets, leveraging general-purpose Al systems to partially automate the
feedback process can greatly increase the volume of data (158*, 702*). However, in practice, the
amount of explicit human oversight used during finetuning is very small compared to the trillions of
data points used in pre-training on internet data and may, therefore, be unable to fully remove harmful
knowledge or capabilities from pre-training. Short of rethinking the way state-of-the-art general-
purpose Al systems are trained, improving fine-tuning feedback data is unlikely to be a solution on its
own.

**Maintaining uncertainty over goals can reduce risky actions.** Some researchers have proposed methods that involve incorporating uncertainty into the goals that general-purpose Al systems learn to pursue (703, 704, 705, 706*, 707, 708). By requiring general-purpose Al systems to act in a way that respects uncertainty about their objectives, these methods can reduce the risk of unexpected actions and encourage information-seeking or deference to humans in response to ambiguity. However, these methods have yet to be incorporated into state-of-the-art, highly capable Al.

**Some researchers are working toward safe-by-design approaches which might be able to provide quantitative safety guarantees.** Similar to the above approaches to estimate uncertainty about goals and predictions, it may be possible to design Al systems that are constructed to achieve quantified levels of safety (626). The advantage of mathematical guarantees and bounds is that they may provide safety assurances even outside of the domain where the Al has been trained and tested, in contrast with empirical trial-and-error methods that are currently the standard for designing deep learning systems. Currently, however, practically-useful, provable guarantees of safety are not possible
with general-purpose AI models and methods, and many open questions however remain in order to
achieve those objectives for large-scale AI systems.

**It is unclear if and how humans might be able to oversee general-purpose AI systems with capabilities exceeding those of humans.** Future general-purpose AI models or systems that surpass the abilities of human experts across many or all domains would pose a particularly difficult challenge (see **4.2.3. Loss of control**). Some research efforts, especially concentrated in the leading AI laboratories, study the extent to which humans might be able to oversee general-purpose AI with capabilities that exceed the human overseers generally or in a given domain (known as 'scalable oversight'). Some empirical research has studied the ability of less capable general-purpose AI systems to oversee more capable ones under the assumption that similar dynamics may exist for general-purpose AI exceeding human capabilities (**709*, **710***). Researchers have also proposed theoretical approaches that can, under certain assumptions, allow for stronger assurances than existing methods (**711, 712*, **713***). However, published research on these methods is highly preliminary.

## 5.2.2 Reducing the hallucination of falsehoods

**The hallucination of falsehoods is a challenge, but it can be reduced.** In AI, 'hallucination' refers to the propensity of general-purpose AI systems to output falsehoods and made-up content. For example, language models commonly hallucinate non-existent citations, biographies, or facts (**46, 47***, **48, 49, 50**), which could pose legal and ethical problems involving the spread of misinformation (**714**). It is possible but challenging to reduce general-purpose AI systems' tendency to hallucinate untrue outputs. Fine-tuning general-purpose AI models explicitly to make them more truthful – both in the accuracy of their answers and analysis of their competence – is one approach to tackling this challenge (**715***). Additionally, allowing general-purpose AI systems to access knowledge databases when they are asked to perform tasks helps to improve the reliability of language model generations (**716, 717**). Alternative approaches attempt to detect hallucinations rather than remove them from the model, and inform the user if the generated output is not to be trusted (**718**). However, reducing hallucination remains a very active area of research.

## 5.2.3 Improving robustness to failures

Sometimes, unfamiliar inputs a general-purpose AI system encounters in deployment can cause unexpected failures (719), and users or attackers can construct inputs that are specifically designed to make a system fail (720).

**Adversarial training helps improve robustness in state-of-the-art AI systems.** Adversarial training involves first, constructing 'attacks' designed to make a model act undesirably and second, training the system to handle these attacks appropriately. Attacks against AI systems can take many forms and can be either human- or algorithm-generated. Once an adversarial attack has been produced, training on these examples can proceed as usual. Adversarial training has become a principal method by which models are made more robust to failures (2*, 3*, 22*, 204*, 207, 721).

### 5.2.3.2 Open problems in robustness

While adversarial training is a valuable tool, it is not sufficient by itself (219).

**Making systems more robust to unforeseen classes of failure modes is a challenging open problem.** Adversarial training generally requires examples of a failure to fix it, (722*, 723). These limitations have resulted in ongoing games of 'cat and mouse' in which some developers continually update models in response to newly discovered vulnerabilities. A partial solution to this problem is to simply produce and train on more adversarial examples. Automated methods for generating attacks can help scale up adversarial training (203, 210, 237). However, the exponentially large number of possible inputs for general-purpose AI systems makes it intractable to thoroughly search for all types
with general-purpose Al models and methods, and many open questions however remain in order to
achieve those objectives for large-scale Al systems.

**It is unclear if and how humans might be able to oversee general-purpose Al systems with capabilities exceeding those of humans.** Future general-purpose Al models or systems that surpass the abilities of human experts across many or all domains would pose a particularly difficult challenge (see **4.2.3. Loss of control**). Some research efforts, especially concentrated in the leading Al laboratories, study the extent to which humans might be able to oversee general-purpose Al with capabilities that exceed the human overseers generally or in a given domain (known as 'scalable oversight'). Some empirical research has studied the ability of less capable general-purpose Al systems to oversee more capable ones under the assumption that similar dynamics may exist for general-purpose Al exceeding human capabilities (**709*, **710***). Researchers have also proposed theoretical approaches that can, under certain assumptions, allow for stronger assurances than existing methods (**711, 712*, **713***). However, published research on these methods is highly preliminary.

5.2.2 Reducing the hallucination of falsehoods

**The hallucination of falsehoods is a challenge, but it can be reduced.** In AI, 'hallucination' refers to the propensity of general-purpose AI systems to output falsehoods and made-up content. For example, language models commonly hallucinate non-existent citations, biographies, or facts (46, 47*, 48, 49, 50), which could pose legal and ethical problems involving the spread of misinformation (714). It is possible but challenging to reduce general-purpose AI systems' tendency to hallucinate untrue outputs. Fine-tuning general-purpose AI models explicitly to make them more truthful – both in the accuracy of their answers and analysis of their competence – is one approach to tackling this challenge (715*). Additionally, allowing general-purpose AI systems to access knowledge databases when they are asked to perform tasks helps to improve the reliability of language model generations (716, 717). Alternative approaches attempt to detect hallucinations rather than remove them from the model, and inform the user if the generated output is not to be trusted (718). However, reducing hallucination remains a very active area of research.

5.2.3 Improving robustness to failures

Sometimes, unfamiliar inputs a general-purpose Al system encounters in deployment can cause
unexpected failures (719), and users or attackers can construct inputs that are specifically designed to
make a system fail (720).

**Adversarial training helps improve robustness in state-of-the-art Al systems.** Adversarial training involves first, constructing 'attacks' designed to make a model act undesirably and second, training the system to handle these attacks appropriately. Attacks against Al systems can take many forms and can be either human- or algorithm-generated. Once an adversarial attack has been produced, training on these examples can proceed as usual. Adversarial training has become a principal method by which models are made more robust to failures (2*, 3*, 22*, 204*, 207, 721).

5.2.3.2 Open problems in robustness

While adversarial training is a valuable tool, it is not sufficient by itself (219).

**Making systems more robust to unforeseen classes of failure modes is a challenging open problem.** Adversarial training generally requires examples of a failure to fix it, (722*, 723). These limitations have resulted in ongoing games of 'cat and mouse' in which some developers continually update models in response to newly discovered vulnerabilities. A partial solution to this problem is to simply produce and train on more adversarial examples. Automated methods for generating attacks can help scale up adversarial training (203, 210, 237). However, the exponentially large number of possible inputs for general-purpose Al systems makes it intractable to thoroughly search for all types
of attacks. One way to address this by applying adversarial training to general-purpose AI models'
internal states, instead of inputs, has been proposed (724), but research on this remains preliminary.
Mathematical proofs to certify a model's robustness would theoretically be a way to cover all possible
attacks (626), but this is not possible with current models and methods.

**Adversarial training can sometimes harm a model's performance or robustness.** In vision models, there is often a trade-off between robustness to adversarial attacks and performance on non-adversarial data (725, 726, 727). Even when adversarial training is helpful to improve worst-case performance, it may not be used when it harms average case performance. Adversarial training can also sometimes make language models less robust to certain attacks that were not trained on (722*, 724). However, some refined approaches to adversarial training may be able to improve trade-offs between performance on clean and adversarial data (728, 729, 730).

5.2.4 Removing hazardous capabilities

**'Machine unlearning' can help to remove certain undesirable capabilities from general-purpose AI systems.** For example, removing certain capabilities that could aid malicious users in making explosives, bioweapons, chemical weapons, and cyberattacks would improve safety (408). Unlearning as a way of negating the influence of undesirable training data was originally proposed as a way to protect privacy and copyright (586) which is discussed in [5.5 Privacy methods for general-purpose AI systems](#). Unlearning methods to remove hazardous capabilities (731, 732) include methods based on fine-tuning (733\*) and editing the inner workings of models (408). Ideally, unlearning should make a model unable to exhibit the unwanted behaviour even when subject to knowledge-extraction attacks, novel situations (e.g. foreign languages), or small amounts of fine-tuning. However, unlearning methods can often fail to perform unlearning robustly and may introduce unwanted side effects (734) on desirable model knowledge.

5.2.5 Analysing and editing the inner workings of models

**Studying the inner workings of models can help to establish the presence or lack of specific capabilities.** One technique that researchers use is to analyse a general-purpose AI model's internal states to better understand what concepts they reason with and what knowledge they have (278, 279, 735). For example, these approaches have been used to study features related to fairness in visual classifiers (736) and what knowledge language models have (737, 738\*). However, methods for assessing a general-purpose AI model's internal representations are imprecise (739, 740, 741). They are also not currently competitively used over other types of evaluations for understanding a general-purpose AI model's capabilities (see [3. Methodology to assess and understand general-purpose AI systems](#)).

**Understanding a model's internal computations might help to investigate whether they have learned trustworthy solutions.** 'Mechanistic interpretability' refers to studying the inner workings of state-of-the-art AI models. However, state-of-the-art neural networks are large and complex, and mechanistic interpretability has not yet been useful and competitive with other ways to analyse models for practical applications. Nonetheless, some researchers have provided thorough investigations of how very small neural networks perform very simple tasks (628, 629, 630). Some recent works have attempted more scalable techniques for designing human-interpretable models (282, 631, 632\*, 742). This type of approach cannot be used to rule out the possibility of harmful or unexpected actions, but it could offer a useful lens into how models work that could be useful for understanding how safe a model is. Instead of trying to interpret internal computations of neural networks, one could use neural networks to generate interpretable and verifiable explanations that could yield quantitative safety guarantees (626), although how to do this efficiently remains an open problem.

**Understanding a model's internal workings can sometimes be used to guide edits to usefully change its behaviour.** Despite the difficulty of understanding models' inner workings, some
of attacks. One way to address this by applying adversarial training to general-purpose AI models'
internal states, instead of inputs, has been proposed (724), but research on this remains preliminary.
Mathematical proofs to certify a model's robustness would theoretically be a way to cover all possible
attacks (626), but this is not possible with current models and methods.

**Adversarial training can sometimes harm a model's performance or robustness.** In vision models, there is often a trade-off between robustness to adversarial attacks and performance on non-adversarial data (725, 726, 727). Even when adversarial training is helpful to improve worst-case performance, it may not be used when it harms average case performance. Adversarial training can also sometimes make language models less robust to certain attacks that were not trained on (722*, 724). However, some refined approaches to adversarial training may be able to improve trade-offs between performance on clean and adversarial data (728, 729, 730).

5.2.4 Removing hazardous capabilities

**'Machine unlearning' can help to remove certain undesirable capabilities from general-purpose AI systems.** For example, removing certain capabilities that could aid malicious users in making explosives, bioweapons, chemical weapons, and cyberattacks would improve safety (408). Unlearning as a way of negating the influence of undesirable training data was originally proposed as a way to protect privacy and copyright (586) which is discussed in [5.5 Privacy methods for general-purpose AI systems](#). Unlearning methods to remove hazardous capabilities (731, 732) include methods based on fine-tuning (733\*) and editing the inner workings of models (408). Ideally, unlearning should make a model unable to exhibit the unwanted behaviour even when subject to knowledge-extraction attacks, novel situations (e.g. foreign languages), or small amounts of fine-tuning. However, unlearning methods can often fail to perform unlearning robustly and may introduce unwanted side effects (734) on desirable model knowledge.

5.2.5 Analysing and editing the inner workings of models

**Studying the inner workings of models can help to establish the presence or lack of specific capabilities.** One technique that researchers use is to analyse a general-purpose AI model's internal states to better understand what concepts they reason with and what knowledge they have (278, 279, 735). For example, these approaches have been used to study features related to fairness in visual classifiers (736) and what knowledge language models have (737, 738\*). However, methods for assessing a general-purpose AI model's internal representations are imprecise (739, 740, 741). They are also not currently competitively used over other types of evaluations for understanding a general-purpose AI model's capabilities (see [3. Methodology to assess and understand general-purpose AI systems](#)).

**Understanding a model's internal computations might help to investigate whether they have learned trustworthy solutions.** 'Mechanistic interpretability' refers to studying the inner workings of state-of-the-art AI models. However, state-of-the-art neural networks are large and complex, and mechanistic interpretability has not yet been useful and competitive with other ways to analyse models for practical applications. Nonetheless, some researchers have provided thorough investigations of how very small neural networks perform very simple tasks (628, 629, 630). Some recent works have attempted more scalable techniques for designing human-interpretable models (282, 631, 632\*, 742). This type of approach cannot be used to rule out the possibility of harmful or unexpected actions, but it could offer a useful lens into how models work that could be useful for understanding how safe a model is. Instead of trying to interpret internal computations of neural networks, one could use neural networks to generate interpretable and verifiable explanations that could yield quantitative safety guarantees (626), although how to do this efficiently remains an open problem.

**Understanding a model's internal workings can sometimes be used to guide edits to usefully change its behaviour.** Despite the difficulty of understanding models' inner workings, some
techniques can be used to guide specific edits to them. Compared to finetuning, these methods can sometimes be more compute- or data-efficient ways of modifying their functionality. Researchers have used a variety of methods for this, based on changes to their internal parameters (743, 744, 745, 746, 747, 748, 749), and neurons (275, 282, 750), or representations (281, 751, 752, 753, 754). However, these techniques are imperfect (299) and typically introduce unintended side effects on model behaviour (755). They remain an active area of research.

## 5.3 Monitoring and intervention

### KEY INFORMATION

* There are several techniques for identifying general-purpose AI system risks, inspecting general-purpose AI model actions, and evaluating performance once a general-purpose AI model has been deployed. These practices are often referred to as 'monitoring'. Meanwhile, 'interventions' refers to techniques that prevent harmful actions from general-purpose AI models.

* Techniques which are being developed to explain general-purpose AI actions could be used to detect and then intervene to block a risky action. However, the application of these techniques to general-purpose AI systems is still nascent.

* Techniques for detecting and watermarking general-purpose AI-generated content can help to avoid some harmful uses of generative general-purpose AI systems by unsophisticated users. However, these techniques are imperfect and can be circumvented by moderately skilled users.

* Techniques for identifying unusual behaviours from general-purpose AI systems can enable improved oversight and interventions.

* Having humans in the loop, and other checks before and during the deployment of general-purpose AI systems increase oversight and provide multiple layers of defence against failures. However, such measures can slow down general-purpose AI system outputs, may compromise privacy and could conflict with the economic incentives for companies that use general-purpose AI systems.

Monitoring during general-purpose AI system deployment refers to the ongoing identification of risks, inspection of model actions, and evaluation of performance. Interventions prevent potentially harmful outputs. Whereas [3. Methodology to assess and understand general-purpose AI systems](#) discusses how systems are evaluated in order to allow for more informed decisions around their use, this section discusses how some techniques for monitoring and intervention can be built into AI systems themselves.

Different strategies that researchers are developing for general-purpose AI system monitoring and intervention are discussed below, including detecting Al-generated content, detecting risky situations, identifying harmful actions, explaining model actions, and intervening to override or block them.

### 5.3.1 Detecting general-purpose AI-generated content

Content generated by general-purpose AI systems – particularly deepfakes – could have widespread harmful effects (756, 757) (see [4.1. Malicious use risks](#)). The ability to distinguish between genuine and general-purpose AI-generated content to prevent the malicious use of generative models.

**Some unreliable techniques exist for detecting general-purpose AI-generated content.** Just as different humans have unique artistic and writing styles, so do generative AI models. Some procedures have been developed to distinguish AI-generated text from human-generated text (374, 375, 379, 380, 758) and images (759, 760). Detection methods are typically based on specialised classifiers or
techniques can be used to guide specific edits to them. Compared to finetuning, these methods can sometimes be more compute- or data-efficient ways of modifying their functionality. Researchers have used a variety of methods for this, based on changes to their internal parameters (743, 744, 745, 746, 747, 748, 749), and neurons (275, 282, 750), or representations (281, 751, 752, 753, 754). However, these techniques are imperfect (299) and typically introduce unintended side effects on model behaviour (755). They remain an active area of research.

## 5.3 Monitoring and intervention

### KEY INFORMATION

* There are several techniques for identifying general-purpose AI system risks, inspecting general-purpose AI model actions, and evaluating performance once a general-purpose AI model has been deployed. These practices are often referred to as 'monitoring'. Meanwhile, 'interventions' refers to techniques that prevent harmful actions from general-purpose AI models.

* Techniques which are being developed to explain general-purpose AI actions could be used to detect and then intervene to block a risky action. However, the application of these techniques to general-purpose AI systems is still nascent.

* Techniques for detecting and watermarking general-purpose AI-generated content can help to avoid some harmful uses of generative general-purpose AI systems by unsophisticated users. However, these techniques are imperfect and can be circumvented by moderately skilled users.

* Techniques for identifying unusual behaviours from general-purpose AI systems can enable improved oversight and interventions.

* Having humans in the loop, and other checks before and during the deployment of general-purpose AI systems increase oversight and provide multiple layers of defence against failures. However, such measures can slow down general-purpose AI system outputs, may compromise privacy and could conflict with the economic incentives for companies that use general-purpose AI systems.

Monitoring during general-purpose AI system deployment refers to the ongoing identification of risks, inspection of model actions, and evaluation of performance. Interventions prevent potentially harmful outputs. Whereas [3. Methodology to assess and understand general-purpose AI systems](#) discusses how systems are evaluated in order to allow for more informed decisions around their use, this section discusses how some techniques for monitoring and intervention can be built into AI systems themselves.

Different strategies that researchers are developing for general-purpose AI system monitoring and intervention are discussed below, including detecting Al-generated content, detecting risky situations, identifying harmful actions, explaining model actions, and intervening to override or block them.

### 5.3.1 Detecting general-purpose AI-generated content

Content generated by general-purpose AI systems – particularly deepfakes – could have widespread harmful effects (756, 757) (see [4.1. Malicious use risks](#)). The ability to distinguish between genuine and general-purpose AI-generated content to prevent the malicious use of generative models.

**Some unreliable techniques exist for detecting general-purpose AI-generated content.** Just as different humans have unique artistic and writing styles, so do generative AI models. Some procedures have been developed to distinguish AI-generated text from human-generated text (374, 375, 379, 380, 758) and images (759, 760). Detection methods are typically based on specialised classifiers or
assessing how likely it is that a given example was generated by a specific general-purpose Al model. However, existing methods are limited and are prone to false positives because general-purpose Al systems tend to memorise examples that appear in their training data, so common text snippets or images of famous objects may be falsely identified as being Al-generated. As general-purpose Al-generated content becomes even more realistic, it may be even more challenging to detect general-purpose Al-generated content.

**Watermarks make distinguishing Al-generated content easier, but they can be removed.** A 'watermark' refers to a subtle style or motif that can be inserted into a file which is difficult for a human to notice but easy for an algorithm to detect. Watermarks for images typically take the form of imperceptible patterns inserted into image pixels (761), while watermarks for text typically take the form of stylistic or word-choice biases (382, 762). Watermarks are useful, but they are an imperfect strategy for detecting Al-generated content because they can be removed (374, 383). However, this does not mean that they are not useful. As an analogy, fingerprints are easy to avoid or remove, but they are still very useful in forensic science.

**Watermarks can also be used to indicate genuine content.** In contrast to inserting watermarks into general-purpose Al-generated content, a contrasting approach is to put encrypted watermarks in non-Al-generated content (763). However, this would require changes in the hardware and software of physical recording devices, and it is not clear whether these methods could be bypassed by tampering.

## 5.3.2 Detecting anomalies and attacks

**Detecting anomalies and attacks on general-purpose Al systems allows precautions to be taken when they are identified.** Some methods have been developed that can help detect unusual inputs or behaviours from Al systems (764, 765). Other technical approaches aim to detect when the model outputs for a given input are uncertain, which can indicate that there is an attack or a risk of false outputs (766). Once detected, these examples can be sent to a fault-handling process or flagged for further investigation. It is also sometimes possible to detect and filter a significant proportion of malicious attacks before they are passed into a general-purpose Al model (723, 767) or detect potentially harmful outputs so that they can be blocked before they are sent to a user (768, 769).

## 5.3.3 Explaining model actions

**Techniques to explain why deployed general-purpose Al systems act the way they do are nascent and not widely applied yet, but there are some helpful methods.** The actions of general-purpose Al systems can be hard to understand. However, understanding the reasons why models act the way they do is important for evaluation and determining accountability for harms caused by general-purpose Al systems (269, 770). Unfortunately, simply asking general-purpose Al language models for explanations of their decisions tends to produce misleading answers (771). To increase the reliability of model explanations, researchers are working on improved prompting and training strategies (772*, 773*). Other techniques for explaining general-purpose Al model actions (774, 775) have been shown to help with debugging (207). However, correctly explaining general-purpose Al model actions is a difficult problem because the size and complexity of general-purpose Al systems are beyond easy human understanding. State-of-the-art general-purpose Al systems are trained to produce outputs that are positively reinforced – not to do so for the desired reasons or in a self-consistent way.

## 5.3.4 Building safeguards into Al systems

As discussed in [5.1. Risk management and safety engineering](#), although there is no perfect safety measure, having multiple layers of protective measures and redundant safeguards in place increases
assessing how likely it is that a given example was generated by a specific general-purpose Al model. However, existing methods are limited and are prone to false positives because general-purpose Al systems tend to memorise examples that appear in their training data, so common text snippets or images of famous objects may be falsely identified as being Al-generated. As general-purpose Al-generated content becomes even more realistic, it may be even more challenging to detect general-purpose Al-generated content.

**Watermarks make distinguishing Al-generated content easier, but they can be removed.** A 'watermark' refers to a subtle style or motif that can be inserted into a file which is difficult for a human to notice but easy for an algorithm to detect. Watermarks for images typically take the form of imperceptible patterns inserted into image pixels (761), while watermarks for text typically take the form of stylistic or word-choice biases (382, 762). Watermarks are useful, but they are an imperfect strategy for detecting Al-generated content because they can be removed (374, 383). However, this does not mean that they are not useful. As an analogy, fingerprints are easy to avoid or remove, but they are still very useful in forensic science.

**Watermarks can also be used to indicate genuine content.** In contrast to inserting watermarks into general-purpose Al-generated content, a contrasting approach is to put encrypted watermarks in non-Al-generated content (763). However, this would require changes in the hardware and software of physical recording devices, and it is not clear whether these methods could be bypassed by tampering.

## 5.3.2 Detecting anomalies and attacks

**Detecting anomalies and attacks on general-purpose Al systems allows precautions to be taken when they are identified.** Some methods have been developed that can help detect unusual inputs or behaviours from Al systems (764, 765). Other technical approaches aim to detect when the model outputs for a given input are uncertain, which can indicate that there is an attack or a risk of false outputs (766). Once detected, these examples can be sent to a fault-handling process or flagged for further investigation. It is also sometimes possible to detect and filter a significant proportion of malicious attacks before they are passed into a general-purpose Al model (723, 767) or detect potentially harmful outputs so that they can be blocked before they are sent to a user (768, 769).

## 5.3.3 Explaining model actions

**Techniques to explain why deployed general-purpose Al systems act the way they do are nascent and not widely applied yet, but there are some helpful methods.** The actions of general-purpose Al systems can be hard to understand. However, understanding the reasons why models act the way they do is important for evaluation and determining accountability for harms caused by general-purpose Al systems (269, 770). Unfortunately, simply asking general-purpose Al language models for explanations of their decisions tends to produce misleading answers (771). To increase the reliability of model explanations, researchers are working on improved prompting and training strategies (772*, 773*). Other techniques for explaining general-purpose Al model actions (774, 775) have been shown to help with debugging (207). However, correctly explaining general-purpose Al model actions is a difficult problem because the size and complexity of general-purpose Al systems are beyond easy human understanding. State-of-the-art general-purpose Al systems are trained to produce outputs that are positively reinforced – not to do so for the desired reasons or in a self-consistent way.

## 5.3.4 Building safeguards into Al systems

As discussed in [5.1. Risk management and safety engineering](#), although there is no perfect safety measure, having multiple layers of protective measures and redundant safeguards in place increases
the level of assurance. Once detected, interventions can identify and protect against potentially harmful actions from deployed general-purpose AI systems.

**Having a human in the loop allows for direct oversight and manual overrides.** Humans in the loop are expensive compared to automated systems. However, in high-stakes decision-making situations, they are essential. Instead of teaching general-purpose AI systems to act on behalf of a human, the human-Al cooperation paradigm aims to combine the skills and strengths of both general-purpose Al systems and humans and is seen as generally preferable in complex situations with potentially harmful ramifications (703, 776, 777*, 778, 779). However, having a human in the loop is not practical in many situations, when decision-making happens too quickly and cannot be slowed down (such as chat applications with millions of users), when the human does not have sufficient domain knowledge, or when human bias or error can exacerbate risks (780). As a result, humans in the loop can only be useful in certain situations.

**Automated processing and filtering methods can offer additional but generally imperfect layers of protection.** Some cyber-attacks against general-purpose Al systems often take the form of subtle, brittle patterns in their inputs. As a result, researchers have developed input pre-processing methods that can remove these patterns (723, 781, 782, 783, 784, 785). Sometimes attempted attacks can be detected and blocked before they are passed to a general-purpose Al model (723, 767), while harmful outputs may be detected before they are sent to a user (768, 769). These methods can significantly reduce risks from some failures, but can be vulnerable to attacks.

**Secure interfaces can be designed for general-purpose Al systems with potentially dangerous capabilities.** General-purpose Al systems that can act autonomously and open-endedly on the web or in the physical world pose elevated risks (see 4.4.1 Cross-cutting technical risk factors). For general-purpose Al systems with highly risky capabilities, limiting the ways in which they can directly influence people or objects is a proposed way to reduce potential risks (625, 786).

## 5.4 Technical approaches to fairness and representation in general-purpose Al systems

### KEY INFORMATION

* General-purpose Al models can capture and, at times, amplify biases in their training data. This contributes to unequal resource allocation, inadequate representation, and discriminatory decisions.

* Fairness lacks a universally agreed-upon definition with variations across cultural, social, and disciplinary contexts.

* From a technical perspective, the cause of bias is often the data, which may fail to adequately represent minorities of a target population. Bias can also stem from poor system design or the type of general-purpose Al technique used. These choices depend on the involvement of diverse perspectives throughout the general-purpose Al lifecycle.

* Mitigation of bias should be addressed throughout the lifecycle of the general-purpose Al system, including design, training, deployment, and usage.

* It is very challenging to entirely prevent bias occurring in current general-purpose Al systems because it requires systematic training data collection, ongoing evaluation, and effective identification of bias, trading off fairness with other objectives such as accuracy, and deciding what is useful knowledge and what is an undesirable bias that should not be reflected in the outputs.

* There are differing views about how feasible it is to achieve meaningful fairness in general-purpose Al systems. Some argue that it is impossible for a general-purpose Al system to be
the level of assurance. Once detected, interventions can identify and protect against potentially harmful actions from deployed general-purpose Al systems.

**Having a human in the loop allows for direct oversight and manual overrides.** Humans in the loop are expensive compared to automated systems. However, in high-stakes decision-making situations, they are essential. Instead of teaching general-purpose Al systems to act on behalf of a human, the human-Al cooperation paradigm aims to combine the skills and strengths of both general-purpose Al systems and humans and is seen as generally preferable in complex situations with potentially harmful ramifications (703, 776, 777*, 778, 779). However, having a human in the loop is not practical in many situations, when decision-making happens too quickly and cannot be slowed down (such as chat applications with millions of users), when the human does not have sufficient domain knowledge, or when human bias or error can exacerbate risks (780). As a result, humans in the loop can only be useful in certain situations.

**Automated processing and filtering methods can offer additional but generally imperfect layers of protection.** Some cyber-attacks against general-purpose Al systems often take the form of subtle, brittle patterns in their inputs. As a result, researchers have developed input pre-processing methods that can remove these patterns (723, 781, 782, 783, 784, 785). Sometimes attempted attacks can be detected and blocked before they are passed to a general-purpose Al model (723, 767), while harmful outputs may be detected before they are sent to a user (768, 769). These methods can significantly reduce risks from some failures, but can be vulnerable to attacks.

**Secure interfaces can be designed for general-purpose Al systems with potentially dangerous capabilities.** General-purpose Al systems that can act autonomously and open-endedly on the web or in the physical world pose elevated risks (see 4.4.1 Cross-cutting technical risk factors). For general-purpose Al systems with highly risky capabilities, limiting the ways in which they can directly influence people or objects is a proposed way to reduce potential risks (625, 786).

## 5.4 Technical approaches to fairness and representation in general-purpose Al systems

### KEY INFORMATION

* General-purpose Al models can capture and, at times, amplify biases in their training data. This contributes to unequal resource allocation, inadequate representation, and discriminatory decisions.

* Fairness lacks a universally agreed-upon definition with variations across cultural, social, and disciplinary contexts.

* From a technical perspective, the cause of bias is often the data, which may fail to adequately represent minorities of a target population. Bias can also stem from poor system design or the type of general-purpose Al technique used. These choices depend on the involvement of diverse perspectives throughout the general-purpose Al lifecycle.

* Mitigation of bias should be addressed throughout the lifecycle of the general-purpose Al system, including design, training, deployment, and usage.

* It is very challenging to entirely prevent bias occurring in current general-purpose Al systems because it requires systematic training data collection, ongoing evaluation, and effective identification of bias, trading off fairness with other objectives such as accuracy, and deciding what is useful knowledge and what is an undesirable bias that should not be reflected in the outputs.

* There are differing views about how feasible it is to achieve meaningful fairness in general-purpose Al systems. Some argue that it is impossible for a general-purpose Al system to be
completely 'fair', while others think that from a practical perspective, near-complete fairness is achievable.

Fairness has no universally agreed-upon definition, and varies according to the cultural, social, and disciplinary contexts (333, 787, 788, 789, 790). Philosophically, fairness entails a deep ethical reflection on principles, values, and the distribution of resources and opportunities, while legal definitions may hinge on constitutional principles and case law. Achieving consensus on fairness proves to be elusive due to its multifaceted nature, necessitating engagement with diverse perspectives and context-specific factors.

The rise of AI applications brings algorithmic fairness to the forefront as a significant concern. Fairness in AI attempts to correct algorithmic bias in automated decision-making or content generation. AI fairness can be defined and measured in various ways (such as 'individual fairness'<fnref n="6" /> versus 'group fairness'<fnref n="7" />) (791) which is proved appropriate depending on the context and the specific goals of the application (333), complicating the assessment of AI models. A classic example of this type of dilemma from the algorithmic decision-making literature is that the COMPAS software used to predict criminal recidivism is a well-known case of an unfair AI model. By some measures, COMPAS was found to be biased against African Americans while by other measures it was not (788, 790).

When an AI model is unfair, the actions that it takes are biased, harming individuals or communities. Bias in AI refers to unjustified favouritism towards or against entities based on their inherent or acquired characteristics (788). Bias is tightly intertwined with representation and fairness. The effectiveness of data-driven algorithms depends on the quality of the data they utilise. However, datasets often fail to adequately represent minorities, which can be reflected and amplified in the AI trained on these datasets (227). Biases in AI systems can lead to social harm, including unequal resource allocation and discriminatory decisions against marginalised groups (792) posing significant challenges across various domains (793, 794, 795).

## 5.4.1 Mitigation of bias and discrimination works throughout the stages of general-purpose AI development and deployment

Researchers deploy a variety of methods to mitigate or remove bias and improve fairness in general-purpose AI systems (2*, 22*), including pre-processing, in-processing, and post-processing techniques (796, 797). Pre-processing techniques analyse and rectify data to remove inherent bias existing in datasets, while in-processing techniques design and employ learning algorithms to mitigate discrimination during the training phase of the system. Post-processing methods adjust general-purpose AI system outputs once deployed.

*   **Pre-processing bias mitigation** - This includes targeting flaws in training data like harmful information or under-representation of certain demographic groups. Two prevalent techniques are data augmentation and data alteration. Data augmentation involves adding samples for under-represented groups to increase their representation in the datasets, either by creating modified copies of existing data or generating synthetic data (798, 799, 800). Data alteration modifies dataset samples based on predefined rules such as adding, removing, or masking attributes like gender and race from the data corpus (227, 801).

<footnote n="6">⁶ Individual fairness emphasises similar treatment for similar individuals</footnote>

<footnote n="7">⁷ Group fairness, on the other hand, ensures forms of statistical parity (e.g. between positive outcomes, or errors) for members of different protected groups (e.g. gender or race)</footnote>
completely 'fair', while others think that from a practical perspective, near-complete fairness is achievable.

Fairness has no universally agreed-upon definition, and varies according to the cultural, social, and disciplinary contexts (333, 787, 788, 789, 790). Philosophically, fairness entails a deep ethical reflection on principles, values, and the distribution of resources and opportunities, while legal definitions may hinge on constitutional principles and case law. Achieving consensus on fairness proves to be elusive due to its multifaceted nature, necessitating engagement with diverse perspectives and context-specific factors.

The rise of AI applications brings algorithmic fairness to the forefront as a significant concern. Fairness in AI attempts to correct algorithmic bias in automated decision-making or content generation. AI fairness can be defined and measured in various ways (such as 'individual fairness'<fnref n="6" /> versus 'group fairness'<fnref n="7" />) (791) which is proved appropriate depending on the context and the specific goals of the application (333), complicating the assessment of AI models. A classic example of this type of dilemma from the algorithmic decision-making literature is that the COMPAS software used to predict criminal recidivism is a well-known case of an unfair AI model. By some measures, COMPAS was found to be biased against African Americans while by other measures it was not (788, 790).

When an AI model is unfair, the actions that it takes are biased, harming individuals or communities. Bias in AI refers to unjustified favouritism towards or against entities based on their inherent or acquired characteristics (788). Bias is tightly intertwined with representation and fairness. The effectiveness of data-driven algorithms depends on the quality of the data they utilise. However, datasets often fail to adequately represent minorities, which can be reflected and amplified in the AI trained on these datasets (227). Biases in AI systems can lead to social harm, including unequal resource allocation and discriminatory decisions against marginalised groups (792) posing significant challenges across various domains (793, 794, 795).

## 5.4.1 Mitigation of bias and discrimination works throughout the stages of general-purpose AI development and deployment

Researchers deploy a variety of methods to mitigate or remove bias and improve fairness in general-purpose AI systems (2*, 22*), including pre-processing, in-processing, and post-processing techniques (796, 797). Pre-processing techniques analyse and rectify data to remove inherent bias existing in datasets, while in-processing techniques design and employ learning algorithms to mitigate discrimination during the training phase of the system. Post-processing methods adjust general-purpose AI system outputs once deployed.

*   **Pre-processing bias mitigation** - This includes targeting flaws in training data like harmful information or under-representation of certain demographic groups. Two prevalent techniques are data augmentation and data alteration. Data augmentation involves adding samples for under-represented groups to increase their representation in the datasets, either by creating modified copies of existing data or generating synthetic data (798, 799, 800). Data alteration modifies dataset samples based on predefined rules such as adding, removing, or masking attributes like gender and race from the data corpus (227, 801).

<footnote n="6">⁶ Individual fairness emphasises similar treatment for similar individuals</footnote>

<footnote n="7">⁷ Group fairness, on the other hand, ensures forms of statistical parity (e.g. between positive outcomes, or errors) for members of different protected groups (e.g. gender or race)</footnote>
* In-processing bias mitigation - Data manipulation alone often fails to ensure fairness in general-purpose AI systems (802, 803). Even with data that perfectly represents the population distribution, undesirable stereotypes and prejudices that are present in society can manifest (803). An additional strategy in pre-processing techniques is to integrate fairness into general-purpose AI systems by designing and utilising specialised learning algorithms to teach the general-purpose Al model to generate unbiased content (804, 805, 806, 807, 808, 809, 810). This can be achieved either by humans giving the general-purpose Al model feedback about what kind of output is and is not desirable (2*, 22*, 702*, 811) or by teaching the general-purpose Al model to follow human instructions (3*, 812, 813, 814). General-purpose Al models can also teach each other fairness, transferring information from a less biased general-purpose Al model (the 'teacher') to a second general-purpose Al model (the 'student') (815). A conventional method to mitigate bias in general-purpose Al systems is to train the general-purpose Al model with both unbiased and biased data samples, so that the model learns what to do and what not to do (816, 817, 818). Making under-represented attributes more salient is another pragmatic strategy toward model un-biasing (819, 820*); yet it can potentially compromise privacy when trying to compensate for under-represented attributes (821*).

* Post-processing mitigation of discrimination - When one cannot modify the full model directly (822). Another approach is to manipulate only the input or output of the general-purpose Al system toward fairness. Modifying general-purpose Al model inputs through prompts can help mitigate discrimination in outputs (2*, 823). In many cases, an external module such as a bias/safety classifier is responsible for detecting the unfair or unsafe output of the system (812). The external module can then rank outputs (811) and require the general-purpose Al system to regenerate the output to produce correct, non-discriminatory, content. If the topic is highly sensitive, the general-purpose Al system may be trained to refuse to reply to users seeking to prompt biased and discriminatory outputs (820*, 822, 824). These methods focus on manipulating the outputs of general-purpose Al systems to be fairer. These techniques are usually employed to complement other approaches.

There is no single technique that can ensure fairness in every situation or outcome. Consequently, leading AI companies utilise a combination of these approaches to iteratively improve fairness in their general-purpose Al systems (20, 825*).

Importantly, increasing meaningful representation and participation can help to reduce the risk that under-represented groups are excluded. From a societal perspective, the AI alignment problem discussed in 5.2.1 *Aligning general-purpose Al systems with developer intentions* is ill-defined. It is not possible to make a general-purpose Al system represent the values of everyone in a diverse society where people sometimes disagree (239, 307, 332). Increased participation (826), representation (827), and dialogue (332) have been proposed as ways to reduce the risk that alignment to some people's interests will be harmful to others. However, some forms of participation can fail to be meaningful, and cannot fully solve challenges posed by disagreements between different people (331).

## 5.4.2 Is fairness in general-purpose Al systems achievable?

It is debated whether general-purpose Al systems can ever be completely 'fair'. There are arguments both for and against its feasibility. Mathematical results suggest that it may not be possible to satisfy all aspects of fairness simultaneously under reasonable assumptions (828, 829, 830, 831). This impossibility theorem of fairness is supported by results indicating the complexity of training unbiased general-purpose Al models (832, 833).

Many desirable properties involve trade-offs, such as the four-way trade-off among system fairness, accuracy, privacy, and efficiency (834). Studies suggest there is a trade-off between fairness and other values such as privacy (821*, 834) and predictive accuracy in general-purpose Al systems (829, 835, 836). A possible example is Google Gemini, which generated images of indigenous people and women of colour as US senators from the 1800s, and 'ethnically diverse' World War II-era German soldiers. These contents factually misrepresent the history, possibly as a result of an attempt to
* In-processing bias mitigation - Data manipulation alone often fails to ensure fairness in general-purpose Al systems (802, 803). Even with data that perfectly represents the population distribution, undesirable stereotypes and prejudices that are present in society can manifest (803). An additional strategy in pre-processing techniques is to integrate fairness into general-purpose Al systems by designing and utilising specialised learning algorithms to teach the general-purpose Al model to generate unbiased content (804, 805, 806, 807, 808, 809, 810). This can be achieved either by humans giving the general-purpose Al model feedback about what kind of output is and is not desirable (2*, 22*, 702*, 811) or by teaching the general-purpose Al model to follow human instructions (3*, 812, 813, 814). General-purpose Al models can also teach each other fairness, transferring information from a less biased general-purpose Al model (the 'teacher') to a second general-purpose Al model (the 'student') (815). A conventional method to mitigate bias in general-purpose Al systems is to train the general-purpose Al model with both unbiased and biased data samples, so that the model learns what to do and what not to do (816, 817, 818). Making under-represented attributes more salient is another pragmatic strategy toward model un-biasing (819, 820*); yet it can potentially compromise privacy when trying to compensate for under-represented attributes (821*).

* Post-processing mitigation of discrimination - When one cannot modify the full model directly (822). Another approach is to manipulate only the input or output of the general-purpose Al system toward fairness. Modifying general-purpose Al model inputs through prompts can help mitigate discrimination in outputs (2*, 823). In many cases, an external module such as a bias/safety classifier is responsible for detecting the unfair or unsafe output of the system (812). The external module can then rank outputs (811) and require the general-purpose Al system to regenerate the output to produce correct, non-discriminatory, content. If the topic is highly sensitive, the general-purpose Al system may be trained to refuse to reply to users seeking to prompt biased and discriminatory outputs (820*, 822, 824). These methods focus on manipulating the outputs of general-purpose Al systems to be fairer. These techniques are usually employed to complement other approaches.

There is no single technique that can ensure fairness in every situation or outcome. Consequently, leading Al companies utilise a combination of these approaches to iteratively improve fairness in their general-purpose Al systems (20, 825*).

Importantly, increasing meaningful representation and participation can help to reduce the risk that under-represented groups are excluded. From a societal perspective, the Al alignment problem discussed in 5.2.1 *Aligning general-purpose Al systems with developer intentions* is ill-defined. It is not possible to make a general-purpose Al system represent the values of everyone in a diverse society where people sometimes disagree (239, 307, 332). Increased participation (826), representation (827), and dialogue (332) have been proposed as ways to reduce the risk that alignment to some people's interests will be harmful to others. However, some forms of participation can fail to be meaningful, and cannot fully solve challenges posed by disagreements between different people (331).

## 5.4.2 Is fairness in general-purpose Al systems achievable?

It is debated whether general-purpose Al systems can ever be completely 'fair'. There are arguments both for and against its feasibility. Mathematical results suggest that it may not be possible to satisfy all aspects of fairness simultaneously under reasonable assumptions (828, 829, 830, 831). This impossibility theorem of fairness is supported by results indicating the complexity of training unbiased general-purpose Al models (832, 833).

Many desirable properties involve trade-offs, such as the four-way trade-off among system fairness, accuracy, privacy, and efficiency (834). Studies suggest there is a trade-off between fairness and other values such as privacy (821*, 834) and predictive accuracy in general-purpose Al systems (829, 835, 836). A possible example is Google Gemini, which generated images of indigenous people and women of colour as US senators from the 1800s, and 'ethnically diverse' World War II-era German soldiers. These contents factually misrepresent the history, possibly as a result of an attempt to
ensure racial diversity that failed to foresee and adjust to these specific use cases. To avoid prematurely prioritising specific aspects that inadvertently reflect the personal values of the stakeholders, technology can use quantitative and qualitative measures comprehensible to important technologists, helping them make well-informed decisions about trade-offs, which can then be enforced by the developed systems.

The counter argument is that, while theoretical limitations exist, practical solutions are attainable (837, 838). Some researchers suggest that fairness definitions can be reconciled with one another practically (839, 840), and it is possible to simultaneously satisfy multiple fairness criteria at least to a greater extent than they typically are (841). Empirical evidence challenges the idea that there is always a non-negligible trade-off between fairness and accuracy, indicating such trade-offs can often be resolved in practice. These studies suggest that reducing disparities in general-purpose AI system outputs may not necessarily entail significant drops in accuracy, or require complex methods (838, 839, 840).

Despite rigorous and comprehensive training and testing efforts, establishing general-purpose AI systems that are fair across all measures and across different cultural, social and scientific contexts remains challenging. No existing measure can entirely eliminate all potential risks of bias and unfairness which are inherent in the development of highly capable AI systems (837). Nevertheless, there exists the possibility of striving for continual refinement towards fairer systems.

### 5.4.3 Challenges in achieving fair general-purpose AI systems

Despite all the efforts to remove bias from general-purpose AI systems, major challenges have remained. Firstly, how fairness should be defined and measured is debated (802, 842). The line between useful and accurate world knowledge, and reinforcing harmful stereotypes, can be difficult to draw, and the perception of bias may vary depending on the situation (227, 796). Secondly, other aspects of ensuring general-purpose AI system safety can create or amplify bias: for instance, cleaning data to mitigate toxicity and privacy leakage can change the demographic distribution of the datasets, leading to more bias (843). Thirdly, some issues such as intersectional bias remain difficult to address (844); for instance, a general-purpose AI system might be fair to Asians and women separately, yet be biased toward Asian women. Finally, mitigation of bias requires ongoing effort throughout the development, deployment, and usage of the general-purpose AI systems. Bias in its various forms can emerge gradually, requiring specialised detection and mitigation techniques.

## 5.5 Privacy methods for general-purpose AI systems

### KEY INFORMATION

* General-purpose AI systems present a number of risks to people's privacy, such as loss of data confidentiality, transparency and control over how data is used, and new forms of privacy abuse.

* Privacy protection is an active area of research and development. However, existing technical tools struggle to scale to large general-purpose AI models, and can fail to provide users with meaningful control.

As described in [4.3.5. Risks to privacy](#), advanced general-purpose AI systems pose risks to people's privacy, such as loss of data confidentiality, transparency and control over how data is used, and new forms of privacy abuse. Existing technology and policy only partially address these threats.
ensure racial diversity that failed to foresee and adjust to these specific use cases. To avoid prematurely prioritising specific aspects that inadvertently reflect the personal values of the stakeholders, technology can use quantitative and qualitative measures comprehensible to important technologists, helping them make well-informed decisions about trade-offs, which can then be enforced by the developed systems.

The counter argument is that, while theoretical limitations exist, practical solutions are attainable (837, 838). Some researchers suggest that fairness definitions can be reconciled with one another practically (839, 840), and it is possible to simultaneously satisfy multiple fairness criteria at least to a greater extent than they typically are (841). Empirical evidence challenges the idea that there is always a non-negligible trade-off between fairness and accuracy, indicating such trade-offs can often be resolved in practice. These studies suggest that reducing disparities in general-purpose AI system outputs may not necessarily entail significant drops in accuracy, or require complex methods (838, 839, 840).

Despite rigorous and comprehensive training and testing efforts, establishing general-purpose AI systems that are fair across all measures and across different cultural, social and scientific contexts remains challenging. No existing measure can entirely eliminate all potential risks of bias and unfairness which are inherent in the development of highly capable AI systems (837). Nevertheless, there exists the possibility of striving for continual refinement towards fairer systems.

### 5.4.3 Challenges in achieving fair general-purpose AI systems

Despite all the efforts to remove bias from general-purpose AI systems, major challenges have remained. Firstly, how fairness should be defined and measured is debated (802, 842). The line between useful and accurate world knowledge, and reinforcing harmful stereotypes, can be difficult to draw, and the perception of bias may vary depending on the situation (227, 796). Secondly, other aspects of ensuring general-purpose AI system safety can create or amplify bias: for instance, cleaning data to mitigate toxicity and privacy leakage can change the demographic distribution of the datasets, leading to more bias (843). Thirdly, some issues such as intersectional bias remain difficult to address (844); for instance, a general-purpose AI system might be fair to Asians and women separately, yet be biased toward Asian women. Finally, mitigation of bias requires ongoing effort throughout the development, deployment, and usage of the general-purpose AI systems. Bias in its various forms can emerge gradually, requiring specialised detection and mitigation techniques.

## 5.5 Privacy methods for general-purpose AI systems

### KEY INFORMATION

* General-purpose AI systems present a number of risks to people's privacy, such as loss of data confidentiality, transparency and control over how data is used, and new forms of privacy abuse.

* Privacy protection is an active area of research and development. However, existing technical tools struggle to scale to large general-purpose AI models, and can fail to provide users with meaningful control.

As described in 4.3.5. Risks to privacy, advanced general-purpose AI systems pose risks to people's privacy, such as loss of data confidentiality, transparency and control over how data is used, and new forms of privacy abuse. Existing technology and policy only partially address these threats.
**Current privacy-enhancing technologies do not scale to large general-purpose Al Models.** While various privacy techniques can be applied to Al models to protect individual privacy while still allowing for useful insights to be derived from data (845, 846), these techniques can significantly impair model accuracy, are hard to scale to large models, and may not be suitable for all use cases, in particular for general-purpose Al models trained on text (847). For domains with highly sensitive data (e.g., medical or financial), it may be possible to attain strong privacy guarantees by adapting powerful general-purpose Al models that are first pre-trained on publicly available data from the internet (848, 849), but such techniques have rarely been applied in production, to date. Another solution is using synthetic data to avoid using sensitive data in general-purpose Al systems training pipelines. However, researchers have demonstrated that there is an important utility/privacy trade-off. If the synthetic data utility is high, then they may carry as much information as the original data and enable mostly the same attacks (850, 851, 852).

The confidentiality and data centralisation issues raised by general-purpose Al systems could, in principle, be addressed using secure computation solutions such as cryptographic approaches (853), federated learning (854), and hardware protections (855). Existing techniques, however, have not been scaled to the largest and most capable models being trained today. These solutions also all impose costs which might be prohibitive at scale, although researchers are trying to find ways to reduce these costs. Advances in hardware protections for accelerators, a class of specialised hardware designed to accelerate Al applications such as artificial neural networks and machine vision, could in future provide a practical avenue for training and running general-purpose Al models without accessing sensitive data.

**Measures to address the lack of data transparency and control in other areas could be applied to general-purpose Al systems.** Developing better mechanisms for individuals to control and trace their data would promote transparency and accountability in general-purpose Al systems. This includes providing user-friendly interfaces for managing data permissions, implementing secure data provenance systems to track how data is used and shared, and establishing clear processes for individuals to access, view, correct, and delete their data (856). The technological means to provide these controls exist and are successfully deployed in other areas (for example, user-controlled dashboards allowing users to decide how various websites or companies collect or use their personal data). It is possible that such and other approaches could be expanded to general-purpose Al systems. It may also be possible to redistribute the wealth derived from personal data in a more traceable and equitable manner, for example by using economic tools for data valuation (857). However, providing people with transparency and control over how their *public* data is used (i.e. information found readily on the Web) is much more challenging. While individual service providers such as social media platforms could prohibit their data from being used by external general-purpose Al systems, the control is ultimately not in the hands of the end-user and only covers a small portion of the data on the Web.

Another challenge is to provide meaningful controls for the use of *derived* data, or data that is not identified but allows inference about a person. Such cases will likely be common in Al systems that use personal data. More research is needed to explore ways to reduce the risks of unauthorised data use and sharing.

**Some forms of privacy abuse are hard to prevent through technical means.** In part, due to the lack of data transparency and control, new forms of privacy abuse stemming from general-purpose Al, such as non-consensual deepfakes or stalking, are hard to prevent via technical means. Some legal frameworks aim to hold creators and distributors accountable for malicious use (858), and to provide remedies for individuals whose privacy has been violated. Some recent regulations also call for Al systems to be developed and deployed in a manner that respects privacy principles, such as by enforcing data minimisation and purpose limitation (859, 860), yet how to achieve these properties, or the extent to which they are achievable is questionable (861).
**Current privacy-enhancing technologies do not scale to large general-purpose Al Models.** While various privacy techniques can be applied to Al models to protect individual privacy while still allowing for useful insights to be derived from data (845, 846), these techniques can significantly impair model accuracy, are hard to scale to large models, and may not be suitable for all use cases, in particular for general-purpose Al models trained on text (847). For domains with highly sensitive data (e.g., medical or financial), it may be possible to attain strong privacy guarantees by adapting powerful general-purpose Al models that are first pre-trained on publicly available data from the internet (848, 849), but such techniques have rarely been applied in production, to date. Another solution is using synthetic data to avoid using sensitive data in general-purpose Al systems training pipelines. However, researchers have demonstrated that there is an important utility/privacy trade-off. If the synthetic data utility is high, then they may carry as much information as the original data and enable mostly the same attacks (850, 851, 852).

The confidentiality and data centralisation issues raised by general-purpose Al systems could, in principle, be addressed using secure computation solutions such as cryptographic approaches (853), federated learning (854), and hardware protections (855). Existing techniques, however, have not been scaled to the largest and most capable models being trained today. These solutions also all impose costs which might be prohibitive at scale, although researchers are trying to find ways to reduce these costs. Advances in hardware protections for accelerators, a class of specialised hardware designed to accelerate Al applications such as artificial neural networks and machine vision, could in future provide a practical avenue for training and running general-purpose Al models without accessing sensitive data.

**Measures to address the lack of data transparency and control in other areas could be applied to general-purpose Al systems.** Developing better mechanisms for individuals to control and trace their data would promote transparency and accountability in general-purpose Al systems. This includes providing user-friendly interfaces for managing data permissions, implementing secure data provenance systems to track how data is used and shared, and establishing clear processes for individuals to access, view, correct, and delete their data (856). The technological means to provide these controls exist and are successfully deployed in other areas (for example, user-controlled dashboards allowing users to decide how various websites or companies collect or use their personal data). It is possible that such and other approaches could be expanded to general-purpose Al systems. It may also be possible to redistribute the wealth derived from personal data in a more traceable and equitable manner, for example by using economic tools for data valuation (857). However, providing people with transparency and control over how their *public* data is used (i.e. information found readily on the Web) is much more challenging. While individual service providers such as social media platforms could prohibit their data from being used by external general-purpose Al systems, the control is ultimately not in the hands of the end-user and only covers a small portion of the data on the Web.

Another challenge is to provide meaningful controls for the use of *derived* data, or data that is not identified but allows inference about a person. Such cases will likely be common in Al systems that use personal data. More research is needed to explore ways to reduce the risks of unauthorised data use and sharing.

**Some forms of privacy abuse are hard to prevent through technical means.** In part, due to the lack of data transparency and control, new forms of privacy abuse stemming from general-purpose Al, such as non-consensual deepfakes or stalking, are hard to prevent via technical means. Some legal frameworks aim to hold creators and distributors accountable for malicious use (858), and to provide remedies for individuals whose privacy has been violated. Some recent regulations also call for Al systems to be developed and deployed in a manner that respects privacy principles, such as by enforcing data minimisation and purpose limitation (859, 860), yet how to achieve these properties, or the extent to which they are achievable is questionable (861).