## Al chip production challenges and slowing GPU improvements may limit Al compute scaling

Over the past few years, the production of data-centre GPUs has been a bottleneck in scaling up compute for general-purpose Al systems, partly due to the limited capacity of semiconductor manufacturing plants and the constraints and priorities in the global semiconductor supply chain (142, 143). Al chip manufacturing depends on a complex, hard-to-scale supply chain, featuring sophisticated lithography, advanced packaging, specialised photoresists, and unique chemicals. Constructing new semiconductor fabrication plants ('fabس') is very expensive and typically takes three to five years (144, 145) making the industry slow to respond to market demand. These factors make projecting future supply complex, and a range of scenarios for chip availability have been proposed. While state-of-the-art GPU production is picking up substantially, supply chains for Al chips may not be able to adapt to demand. This may slow down the ambitions of frontier Al companies for further rapid growth, though Al companies might continue to scale in the near-term if they can acquire large fractions of the total number of GPUs produced.

Improved GPU performance has also contributed to the recent scaling of compute (67, 68). It is possible that over this decade, progress in individual GPU performance may slow due to physical limits on transistor size (146, 147) and energy efficiency (148). This would have only a limited impact on scaling because the primary driver of compute scaling has not been improved GPU performance, but the increase in the number of GPUs used. GPU price performance and energy efficiency for relevant computation have been improving roughly 30% annually, with an additional factor of 10x - 100x from the addition of tensor cores and the move to lower precision formats (67, 68). However, the total compute used in training has increased by approximately 4x per year since 2010 (17), outpacing the rate of hardware efficiency improvements. This suggests that increased spending, rather than gains in hardware efficiency, has been the primary driver of the growth in compute budgets for Al training.

### 2.4.3 Will algorithmic progress lead to rapid advancements?

There is continued rapid growth in the efficiency of training algorithms. Techniques and algorithms underpinning general-purpose Al models have consistently and robustly improved over time (see 2.3.1 *Recent trends in compute, data, and algorithms*). This includes incremental changes to key algorithms such as the architecture of transformers, improvements to how Al algorithms are implemented on hardware, a better understanding of how to scale models, better methods to process representations in a neural network, and other advances. These advances enable researchers and labs to produce ever-more capable general-purpose Al models without increasing their hardware budget, for example in vision (88*), reinforcement learning (90) and language modelling (89). The rate of improving algorithmic efficiency in language modelling shows no sign of slowing down (89), though there may be diminishing returns to progress in the near future.

**Post-training algorithms can be used to significantly improve general-purpose Al model capabilities at low cost, including via fine-tuning, tool use, and structured reasoning techniques.**

Recent years have seen significant advancements in techniques and approaches for enhancing the performance of general-purpose Al models after pre-training. Many post-training algorithms improved model performance on a given benchmark by more than using 5x the training compute, and in some cases more than 20x (25). Post-training algorithms can be applied to a given model for various use-cases with the aim of better serving end-users specific needs, at a much lower cost than developing the original model. This low cost means that a wide range of actors, including low-resource actors, could advance frontier general-purpose Al capabilities by developing better post-training algorithms. Governance processes need to take into account post-training algorithms.

Post-training algorithms include fine-tuning models for better performance, equipping them with the ability to leverage external tools, crafting prompts to guide their outputs, structuring their reasoning
processes for more coherent and logical responses, and selecting from multiple responses the most relevant and accurate candidate outputs. There is a rapidly growing body of work on post-training enhancements, to improve frontier LLMs performance broadly (149), and in specific domains such as code generation (150) and mathematics (151). Such innovations have the potential to further improve the performance of general-purpose AI systems over the next few years. However, if there is a slowdown in the scaling of resources going to training, due to the bottlenecks discussed above, this might in turn slow progress in finding better post-training algorithms.

**General-purpose AI systems might be deployed to automate and accelerate AI R&D.** Narrow AI systems have already been used to develop and improve algorithms (152, 153). Recent LLMs are used in areas related to AI R&D, particularly in programming (26), generating and optimising prompts (154, 155, 156, 157), replacing human fine-tuning data (158*), and selecting high-quality training data (159*). As the capabilities of general-purpose AI systems advance, it becomes harder to predict the effect on algorithmic progress and engineering in AI.
processes for more coherent and logical responses, and selecting from multiple responses the most relevant and accurate candidate outputs. There is a rapidly growing body of work on post-training enhancements, to improve frontier LLMs performance broadly (149), and in specific domains such as code generation (150) and mathematics (151). Such innovations have the potential to further improve the performance of general-purpose AI systems over the next few years. However, if there is a slowdown in the scaling of resources going to training, due to the bottlenecks discussed above, this might in turn slow progress in finding better post-training algorithms.

**General-purpose AI systems might be deployed to automate and accelerate AI R&D.** Narrow AI systems have already been used to develop and improve algorithms (152, 153). Recent LLMs are used in areas related to AI R&D, particularly in programming (26), generating and optimising prompts (154, 155, 156, 157), replacing human fine-tuning data (158*), and selecting high-quality training data (159*). As the capabilities of general-purpose AI systems advance, it becomes harder to predict the effect on algorithmic progress and engineering in AI.
# 3 Methodology to assess and understand general-purpose AI systems

## KEY INFORMATION

* General-purpose AI governance approaches assume that both AI developers and policymakers can understand and measure what general-purpose AI systems are capable of, and their potential impacts.

* Technical methods can help answer these questions but have limitations. Current approaches cannot provide strong assurances against large-scale general-purpose AI-related harms.

* Currently, developers still understand little about how their general-purpose AI models operate. Model explanation and interpretability techniques can improve researchers' and developers' understanding of how general-purpose AI systems operate, but this research is nascent.

* The capabilities of general-purpose AI are mainly assessed through testing the general-purpose AI on various inputs. These spot checks are helpful and necessary but do not provide quantitative guarantees. They often miss hazards, and overestimate or underestimate general-purpose AI capabilities, because test conditions differ from the real world. Many areas of concern are not fully amenable to the type of quantification that current evaluations rely on (for example, bias and misinformation).

* Independent actors can, in principle, audit general-purpose AI models or systems developed by a company. However, companies do not always provide independent auditors with the necessary level of 'white-box' access to models or information about data and methods used, which are needed for rigorous assessment. Several governments are beginning to build capacity for conducting technical evaluations and audits.

* It is difficult to assess the downstream societal impact of a general-purpose AI system because rigorous and comprehensive assessment methodologies have not yet been developed and because general-purpose AI has a wide range of possible real-world uses. Understanding the potential downstream societal impacts of general-purpose AI models and systems requires nuanced and multidisciplinary analysis. Increasing participation and representation of perspectives in the AI development and evaluation process is an ongoing technical and institutional challenge.

Modern general-purpose AI systems can contain multiple large-scale models and hundreds of billions of parameters, often deployed as general-purpose products. Consequently, it is difficult to anticipate how such general-purpose AI products may function in the many possible deployment scenarios, and even harder to appropriately characterise the downstream consequences of their deployment. Scientists are often surprised at the unexpected capabilities and impacts of general-purpose AI systems.

## 3.1 General-purpose AI assessments serve to evaluate model capabilities and impacts.

There are two broad reasons to assess general-purpose AI models and systems:
# 3 Methodology to assess and understand general-purpose AI systems

## KEY INFORMATION

* General-purpose AI governance approaches assume that both AI developers and policymakers can understand and measure what general-purpose AI systems are capable of, and their potential impacts.

* Technical methods can help answer these questions but have limitations. Current approaches cannot provide strong assurances against large-scale general-purpose AI-related harms.

* Currently, developers still understand little about how their general-purpose AI models operate. Model explanation and interpretability techniques can improve researchers' and developers' understanding of how general-purpose AI systems operate, but this research is nascent.

* The capabilities of general-purpose AI are mainly assessed through testing the general-purpose AI on various inputs. These spot checks are helpful and necessary but do not provide quantitative guarantees. They often miss hazards, and overestimate or underestimate general-purpose AI capabilities, because test conditions differ from the real world. Many areas of concern are not fully amenable to the type of quantification that current evaluations rely on (for example, bias and misinformation).

* Independent actors can, in principle, audit general-purpose AI models or systems developed by a company. However, companies do not always provide independent auditors with the necessary level of 'white-box' access to models or information about data and methods used, which are needed for rigorous assessment. Several governments are beginning to build capacity for conducting technical evaluations and audits.

* It is difficult to assess the downstream societal impact of a general-purpose AI system because rigorous and comprehensive assessment methodologies have not yet been developed and because general-purpose AI has a wide range of possible real-world uses. Understanding the potential downstream societal impacts of general-purpose AI models and systems requires nuanced and multidisciplinary analysis. Increasing participation and representation of perspectives in the AI development and evaluation process is an ongoing technical and institutional challenge.

Modern general-purpose AI systems can contain multiple large-scale models and hundreds of billions of parameters, often deployed as general-purpose products. Consequently, it is difficult to anticipate how such general-purpose AI products may function in the many possible deployment scenarios, and even harder to appropriately characterise the downstream consequences of their deployment. Scientists are often surprised at the unexpected capabilities and impacts of general-purpose AI systems.

## 3.1 General-purpose AI assessments serve to evaluate model capabilities and impacts.

There are two broad reasons to assess general-purpose AI models and systems:
1. **Determining general capabilities and limitations:** Model evaluations indicate the relationship between model design choices and model outcomes. This performance analysis helps researchers understand how well these systems meet our expectations in both controlled and natural settings. A more informed understanding of model capabilities helps to judge its suitability for use. Every evaluation comes with limitations and uncertainties, which must be documented to properly interpret its results.

2. **Assessing societal impact and downstream risks:** Forecasting and assessing a general-purpose Al system's broader impact can inform debate about questions related to deployment or governance. However, these assessments are a complex interdisciplinary challenge. Societal risk assessment can assess product safety, security vulnerabilities, and unwanted externalities such as labour and environmental impacts, among other concerns. This typically involves factors that can contribute to accidents during expected product use, as well as addressing unanticipated and malicious use.

## 3.2 Approaches for model performance analysis

Various stakeholders (i.e. Al developers, users, impacted population members, etc.) have expectations of how a general-purpose Al system should behave in terms of model capabilities and preventing negative downstream social impacts. Researchers have developed a variety of methods to compare model outcomes with these expectations (160). This model performance analysis is integral in understanding how a model performs and which limitations, benefits, or risks may arise in deployment.

### 3.2.1 Case studies

In many research papers, the evaluation of model capabilities is qualitative, relying on anecdotal demonstrations of model performance (161) and human judgments. For instance, early evaluation of image-generating models often relied on simply demonstrating a small number of examples (162, 163). When GPT-4 was newly released (99*), examples of model outputs on a curated set of tasks, beyond conventional benchmarks, were used to illustrate model performance. Now, several popular benchmarks rely on asking human raters to rate the responses of different models against one another (164, 165). Meanwhile, risks are sometimes gauged through 'uplift studies', where the aim is to test how much more competent a human is at accomplishing a potentially harmful task when they have access to a general-purpose Al system versus when they do not (166*).

### 3.2.2 Benchmarks

Most machine learning evaluations are completed on standardised benchmark measurements (167). For example, a relatively small set of image processing benchmarks involving classification (168, 169, 170, 171), segmentation (36*, 172), and question answering (173, 174) have been integral to Al vision research. Similarly, research on language modelling has been shaped by several publicly available benchmarks that are meant to measure general capabilities (95, 96, 165, 175, 176, 177*, 178, 179) and trustworthiness (180, 181, 182). More recently, benchmarks are being designed to measure the growing general-purpose Al capabilities at combining information from multiple modalities and using software tools like web browsers (183) (see also 4.4.1 Cross-cutting technical risk factors).

**Performance on benchmarks can be imperfect measures of downstream task performance.**

Benchmarks are inherently proxy measures for desired performance, and good scores on a benchmark do not always translate to desired performance in practice (184) because of various validity challenges. Internal validity challenges relate to the reliability of the benchmark measurement, i.e. how reliable the reported metric is over repeated executions in comparison to a strong baseline. For instance, internal validity issues arise if the benchmark does not contain enough examples to make statistically valid claims about model performance (185) or contains false labels (186, 187). External
1. **Determining general capabilities and limitations:** Model evaluations indicate the relationship between model design choices and model outcomes. This performance analysis helps researchers understand how well these systems meet our expectations in both controlled and natural settings. A more informed understanding of model capabilities helps to judge its suitability for use. Every evaluation comes with limitations and uncertainties, which must be documented to properly interpret its results.

2. **Assessing societal impact and downstream risks:** Forecasting and assessing a general-purpose AI system's broader impact can inform debate about questions related to deployment or governance. However, these assessments are a complex interdisciplinary challenge. Societal risk assessment can assess product safety, security vulnerabilities, and unwanted externalities such as labour and environmental impacts, among other concerns. This typically involves factors that can contribute to accidents during expected product use, as well as addressing unanticipated and malicious use.

## 3.2 Approaches for model performance analysis

Various stakeholders (i.e. AI developers, users, impacted population members, etc.) have expectations of how a general-purpose AI system should behave in terms of model capabilities and preventing negative downstream social impacts. Researchers have developed a variety of methods to compare model outcomes with these expectations (160). This model performance analysis is integral in understanding how a model performs and which limitations, benefits, or risks may arise in deployment.

### 3.2.1 Case studies

In many research papers, the evaluation of model capabilities is qualitative, relying on anecdotal demonstrations of model performance (161) and human judgments. For instance, early evaluation of image-generating models often relied on simply demonstrating a small number of examples (162, 163). When GPT-4 was newly released (99*), examples of model outputs on a curated set of tasks, beyond conventional benchmarks, were used to illustrate model performance. Now, several popular benchmarks rely on asking human raters to rate the responses of different models against one another (164, 165). Meanwhile, risks are sometimes gauged through 'uplift studies', where the aim is to test how much more competent a human is at accomplishing a potentially harmful task when they have access to a general-purpose AI system versus when they do not (166*).

### 3.2.2 Benchmarks

Most machine learning evaluations are completed on standardised benchmark measurements (167). For example, a relatively small set of image processing benchmarks involving classification (168, 169, 170, 171), segmentation (36*, 172), and question answering (173, 174) have been integral to AI vision research. Similarly, research on language modelling has been shaped by several publicly available benchmarks that are meant to measure general capabilities (95, 96, 165, 175, 176, 177*, 178, 179) and trustworthiness (180, 181, 182). More recently, benchmarks are being designed to measure the growing general-purpose AI capabilities at combining information from multiple modalities and using software tools like web browsers (183) (see also 4.4.1 Cross-cutting technical risk factors).

**Performance on benchmarks can be imperfect measures of downstream task performance.**

Benchmarks are inherently proxy measures for desired performance, and good scores on a benchmark do not always translate to desired performance in practice (184) because of various validity challenges. Internal validity challenges relate to the reliability of the benchmark measurement, i.e. how reliable the reported metric is over repeated executions in comparison to a strong baseline. For instance, internal validity issues arise if the benchmark does not contain enough examples to make statistically valid claims about model performance (185) or contains false labels (186, 187). External
validity refers to how well benchmark performance translates to real-world settings. Benchmarks themselves can be poorly constructed, inadequate, or incomplete representations of real-world tasks. For instance, limitations on how state-of-the-art models are prompted and combined with other systems can lead to their capabilities being underestimated (24, 188, 189). Currently, benchmarks are often not explicit about their scope of applicability. Benchmarks that lay claim to 'general' performance often disguise biases in cultural representation and annotator differences, contested notions of ground truth, and more (101, 167, 190, 191, 192). Also, as modern general-purpose AI models have been trained on large amounts of internet data, it can be difficult to disentangle novel capabilities from memorised ones (193, 194).

**Interpreting human performance on benchmarks can be difficult.** A critical aspect of intuitively understanding model performance is a comparison with human performance (195*). Human performance measures are often unreliable – for instance, the baseline for ‘human performance’ on the ImageNet benchmark consists of the annotations of a single graduate student (196). Human annotators of ‘ground truth’ are notoriously fickle (197), often disagreeing in meaningful ways (198) due to differences in cultural context, values, or expertise. Furthermore, there is a meaningful difference between human performance and human competence at a task – the latter often involves, for example, judgments of robustness and not just accuracy (199). Evaluations which intentionally diversify their annotator populations (200), allow for a multiplicity of ground truth labels (198), or properly consider the context of human performance claims (201) tend to be more trustworthy assessments of human-Al comparisons.

### 3.2.3 Red-teaming and adversarial attacks

Before deploying systems in real-world conditions, evaluators use adversarial attacks and red-teaming to identify worst-case behaviours, malicious use opportunities, and the system's potential to fail unexpectedly. In cybersecurity, an adversarial attack refers to a deliberate attempt to make a system fail. For example, attacks against language models can take the form of automatically generated attacks (202, 203, 204*, 205, 206) or manually generated attacks (204*, 207). These can include 'jailbreaking' attacks which subvert the models' safety restrictions (208, 209, 210, 211, 212).

A red team refers to a set of people who aim to find vulnerabilities in a system by attacking it. In contrast to benchmarks which are a fixed set of test cases, a key advantage of red-teaming is that it adapts the evaluation to the specific system being tested. Through interaction with a system, red-teamers can design custom tests for models. Researchers approach red teaming through various strategies and tools. Ojewale et al. (213) map out an ecosystem of tools leveraged in the AI accountability process including resources for 'harm discovery', such as 'bug bounty' platforms, incident databases (214), and more. These tools support the identification of potential vectors of harm and enable broader participation in harm discovery.

**However, evaluators can sometimes fail to represent public interests.** Red-teaming for state-of-the-art general-purpose AI systems is predominantly done by the organisations that develop them (2*, 22*, 204*). Academia, auditing networks, and dedicated evaluation organisations can also play a key role (215, 216, 217). As is the case with AI developers themselves, red team evaluators are not always representative of public interests or demographics and can display bias, or omit important considerations in their identification or assessment of AI-related harms (215). Best practices for red-teaming are not established yet (216).

**Various faults in general-purpose AI models have remained undetected during red teaming.** Red-teaming and adversarial attacks are useful for developing a better understanding of a model's worst-case performance and assessing performance at capabilities benchmarks don't adequately cover. However, they come with limitations. Previous work on benchmarking red-teaming and adversarial attack techniques has found that bugs often evade detection (218). A real-world example is jailbreaks for current state-of-the-art general-purpose AI chat systems (208, 209, 210, 211, 212), which seem to have evaded initial detection by developers who designed them (2*, 22*, 204*). Overall, red-teaming is just one of several assessment tools necessary for meaningful understanding of general-purpose AI
validity refers to how well benchmark performance translates to real-world settings. Benchmarks themselves can be poorly constructed, inadequate, or incomplete representations of real-world tasks. For instance, limitations on how state-of-the-art models are prompted and combined with other systems can lead to their capabilities being underestimated (24, 188, 189). Currently, benchmarks are often not explicit about their scope of applicability. Benchmarks that lay claim to 'general' performance often disguise biases in cultural representation and annotator differences, contested notions of ground truth, and more (101, 167, 190, 191, 192). Also, as modern general-purpose AI models have been trained on large amounts of internet data, it can be difficult to disentangle novel capabilities from memorised ones (193, 194).

**Interpreting human performance on benchmarks can be difficult.** A critical aspect of intuitively understanding model performance is a comparison with human performance (195*). Human performance measures are often unreliable – for instance, the baseline for ‘human performance’ on the ImageNet benchmark consists of the annotations of a single graduate student (196). Human annotators of ‘ground truth’ are notoriously fickle (197), often disagreeing in meaningful ways (198) due to differences in cultural context, values, or expertise. Furthermore, there is a meaningful difference between human performance and human competence at a task – the latter often involves, for example, judgments of robustness and not just accuracy (199). Evaluations which intentionally diversify their annotator populations (200), allow for a multiplicity of ground truth labels (198), or properly consider the context of human performance claims (201) tend to be more trustworthy assessments of human-AI comparisons.

### 3.2.3 Red-teaming and adversarial attacks

Before deploying systems in real-world conditions, evaluators use adversarial attacks and red-teaming to identify worst-case behaviours, malicious use opportunities, and the system's potential to fail unexpectedly. In cybersecurity, an adversarial attack refers to a deliberate attempt to make a system fail. For example, attacks against language models can take the form of automatically generated attacks (202, 203, 204*, 205, 206) or manually generated attacks (204*, 207). These can include 'jailbreaking' attacks which subvert the models' safety restrictions (208, 209, 210, 211, 212).

A red team refers to a set of people who aim to find vulnerabilities in a system by attacking it. In contrast to benchmarks which are a fixed set of test cases, a key advantage of red-teaming is that it adapts the evaluation to the specific system being tested. Through interaction with a system, red-teamers can design custom tests for models. Researchers approach red teaming through various strategies and tools. Ojewale et al. (213) map out an ecosystem of tools leveraged in the AI accountability process including resources for 'harm discovery', such as 'bug bounty' platforms, incident databases (214), and more. These tools support the identification of potential vectors of harm and enable broader participation in harm discovery.

**However, evaluators can sometimes fail to represent public interests.** Red-teaming for state-of-the-art general-purpose AI systems is predominantly done by the organisations that develop them (2*, 22*, 204*). Academia, auditing networks, and dedicated evaluation organisations can also play a key role (215, 216, 217). As is the case with AI developers themselves, red team evaluators are not always representative of public interests or demographics and can display bias, or omit important considerations in their identification or assessment of AI-related harms (215). Best practices for red-teaming are not established yet (216).

**Various faults in general-purpose AI models have remained undetected during red teaming.** Red-teaming and adversarial attacks are useful for developing a better understanding of a model's worst-case performance and assessing performance at capabilities benchmarks don't adequately cover. However, they come with limitations. Previous work on benchmarking red-teaming and adversarial attack techniques has found that bugs often evade detection (218). A real-world example is jailbreaks for current state-of-the-art general-purpose AI chat systems (208, 209, 210, 211, 212), which seem to have evaded initial detection by developers who designed them (2*, 22*, 204*). Overall, red-teaming is just one of several assessment tools necessary for meaningful understanding of general-purpose AI
capabilities (219). Red-teaming can also fail to catch downstream harms, which only arise when the Al system is more widely deployed in society. This is discussed further in 4.3. Systemic risks.

### 3.2.4 Auditing

Design choices throughout the general-purpose Al development process affect how the resulting system works. Auditing provides a mechanism to scrutinise and ensure accountability for those choices. Different groups of general-purpose Al auditors achieve differing degrees of success when it comes to the quality of the evidence gathered and the accountability outcomes achieved (215, 217). One survey of a range of approaches to Al auditing (217) suggests independently assessing deployed general-purpose Al systems across different dimensions (see Figure 1) to hold stakeholders accountable for the choices they make regarding the development of the general-purpose Al system, and its use.

Analysis of training data can reveal problematic content. The analysis of training data, otherwise known as 'data audits' (217), is one concrete approach to the analysis of key model design choices, and can reveal problematic content. In the machine learning development process, data is collected, curated, and annotated (190). Examining how these data engineering decisions could lead to indirect harm, and influence its outcomes is useful in understanding the model and its ultimate downstream impacts. For example, the analysis of text and image data from the internet used to train modern systems has identified copyrighted content (220, 221, 222), hateful speech and memes (223, 224, 225), malign stereotypes (224, 226, 227), sexually-explicit content (223, 226), and depictions of sexual violence (226) including child abuse material (228).

Furthermore, investigations into training datasets often reveal issues in terms of the demographic, geographic, and linguistic under-representation of certain populations in mainstream data sources (229, 230). Such data audits have provided the evidence necessary for legal challenges on copyright. For example, the New York Times lawsuit against OpenAl (231) heavily cites the data audit from Dodge et al. (232) and has led to the attempted redaction of some datasets deemed to contain inappropriate material (233, 234). However, modern general-purpose Al models are often trained on extremely large amounts of internet datasets, and these datasets are often not made public, so data provenance remains a systemic challenge (235, 236). As a result, it is challenging to systematically search for potentially harmful examples in training data at scale.

**The analysis of Al modelling and product choices can reveal trade-offs and indicate downstream risks.** Aside from training data, other methodological choices can contribute to specific problems. Audits that scrutinise how a model is developed are typically referred to as 'process audits'. For example, human feedback-based methods are state-of-the-art for training general-purpose Al models, but can contribute to problems such as sycophancy (237, 238) and producing non-diverse outputs (239, 240, 241). Similarly, engineering decisions such as 'model pruning' can result in inequitable impacts for certain test demographics (242). In the same vein, generative image model architecture choices have been shown to impact the performance of these systems in representing different races (243). It is challenging for researchers to analyse developers' methodology because the full development details of proprietary and even of open-source general-purpose Al models are rarely documented and disclosed (244).

Meanwhile, 'ecosystem audits' help to assess human-Al interactions. An increasing number of laboratory studies investigate how human users interact with general-purpose Al systems. These often take the form of controlled studies (245) where participants interact with models and the impact of modelling and user interaction settings on the decisions and behaviour of participants is measured directly. Such studies have revealed the tendency of users to trust certain presentations of model outputs over others (246, 247). However, since participants are recruited, it can be hard to design the study and recruit a wide enough range of participants to meaningfully reflect the full range of user impacts in practice. Several researchers have begun conducting natural and controlled experiments on the impact of Al use in real deployment settings. For instance, there have been studies on the impact of Al risk assessments on judges' bail decisions in Kentucky, USA (248), the use of automated
capabilities (219). Red-teaming can also fail to catch downstream harms, which only arise when the AI system is more widely deployed in society. This is discussed further in 4.3. Systemic risks.

### 3.2.4 Auditing

Design choices throughout the general-purpose AI development process affect how the resulting system works. Auditing provides a mechanism to scrutinise and ensure accountability for those choices. Different groups of general-purpose AI auditors achieve differing degrees of success when it comes to the quality of the evidence gathered and the accountability outcomes achieved (215, 217). One survey of a range of approaches to AI auditing (217) suggests independently assessing deployed general-purpose AI systems across different dimensions (see Figure 1) to hold stakeholders accountable for the choices they make regarding the development of the general-purpose AI system, and its use.

Analysis of training data can reveal problematic content. The analysis of training data, otherwise known as 'data audits' (217), is one concrete approach to the analysis of key model design choices, and can reveal problematic content. In the machine learning development process, data is collected, curated, and annotated (190). Examining how these data engineering decisions could lead to indirect harm, and influence its outcomes is useful in understanding the model and its ultimate downstream impacts. For example, the analysis of text and image data from the internet used to train modern systems has identified copyrighted content (220, 221, 222), hateful speech and memes (223, 224, 225), malign stereotypes (224, 226, 227), sexually-explicit content (223, 226), and depictions of sexual violence (226) including child abuse material (228).

Furthermore, investigations into training datasets often reveal issues in terms of the demographic, geographic, and linguistic under-representation of certain populations in mainstream data sources (229, 230). Such data audits have provided the evidence necessary for legal challenges on copyright. For example, the New York Times lawsuit against OpenAI (231) heavily cites the data audit from Dodge et al. (232) and has led to the attempted redaction of some datasets deemed to contain inappropriate material (233, 234). However, modern general-purpose AI models are often trained on extremely large amounts of internet datasets, and these datasets are often not made public, so data provenance remains a systemic challenge (235, 236). As a result, it is challenging to systematically search for potentially harmful examples in training data at scale.

**The analysis of AI modelling and product choices can reveal trade-offs and indicate downstream risks.** Aside from training data, other methodological choices can contribute to specific problems. Audits that scrutinise how a model is developed are typically referred to as 'process audits'. For example, human feedback-based methods are state-of-the-art for training general-purpose AI models, but can contribute to problems such as sycophancy (237, 238) and producing non-diverse outputs (239, 240, 241). Similarly, engineering decisions such as 'model pruning' can result in inequitable impacts for certain test demographics (242). In the same vein, generative image model architecture choices have been shown to impact the performance of these systems in representing different races (243). It is challenging for researchers to analyse developers' methodology because the full development details of proprietary and even of open-source general-purpose AI models are rarely documented and disclosed (244).

Meanwhile, 'ecosystem audits' help to assess human-AI interactions. An increasing number of laboratory studies investigate how human users interact with general-purpose AI systems. These often take the form of controlled studies (245) where participants interact with models and the impact of modelling and user interaction settings on the decisions and behaviour of participants is measured directly. Such studies have revealed the tendency of users to trust certain presentations of model outputs over others (246, 247). However, since participants are recruited, it can be hard to design the study and recruit a wide enough range of participants to meaningfully reflect the full range of user impacts in practice. Several researchers have begun conducting natural and controlled experiments on the impact of AI use in real deployment settings. For instance, there have been studies on the impact of AI risk assessments on judges' bail decisions in Kentucky, USA (248), the use of automated
hiring tools on hiring manager discretion (249), and the use of generative AI on middle manager performance (250). Qualitative interviews of stakeholders have proven effective at illustrating more systematic impacts such as some of the social implications of AI implementation (251), which can endure after the removal of the AI system (252).

Analysing AI systems in the real world, post-deployment, allows researchers to study them as components of larger societal systems. Post-market surveillance is quite common in other industries with audit ecosystems (253). Users often discover capabilities and failure modes that developers do not, and monitoring the real-world usage of a system can further scientific understanding. For example, jailbreaks against modern large language chat models were first studied following findings from ordinary users (254). The study of deepfakes in the real world has also helped to shape scientific research on studying and mitigating harms (255, 256).

## 3.3 Model transparency, explanations, and interpretations

In contrast to studying general-purpose AI model outputs, another common approach to assessing models is to study the inner mechanism through which models produce the outputs. This can help researchers contextualise assessments of model performance and deepens their understanding of model functionality. Studying *how* general-purpose AI models and systems operate internally is a popular topic of research with thousands of academic papers produced. Areas of research that aim to enhance transparency include documentation, third-party access mechanisms, black box analysis, explaining model actions, and interpreting the inner workings of models.

**Documentation templates record decisions made and facilitate transparency at an operational level.** Currently, one of the most practical ways to increase transparency about a general-purpose AI model is through documenting and communicating the engineering decisions that define the model. Several documentation solutions have been proposed to communicate such decisions to a broader range of internal and external stakeholders. Some of these efforts, such as the development of Model Cards (257) have been successful. One recent study revealed "a widespread uptake of model cards within the AI community" (258). There are documentation templates available for communication on dataset practices (259, 260, 261), broader system features (262, 263), and broader procedural decision-making (264).

**Model explanation and interpretability techniques can improve researchers' understanding of how general-purpose AI systems operate internally.** There are several tools that allow for the external scrutiny of general-purpose AI systems, enabling access for external actors to query general-purpose AI systems directly, or otherwise gain visibility into model details (213). One prominent technical approach to this involves studying how a model's output can be explained as a consequence of a given input (265, 266, 267, 268). These explanations can play a unique role in supporting accountability, by helping to determine liability, in cases where humans may be wrongfully harmed or discriminated against by automated AI systems (269, 270, 271). Another approach used to study the computations in neural networks has involved interpreting the role of parameters (272), neurons (273, 274, 275), subnetworks (276, 277), or layer representations (278, 279, 280, 281) inside of AI systems. Interpretations of models have sometimes aided researchers in finding vulnerabilities. Examples have involved red-teaming (207), identifying internal representations of spurious features (282), brittle feature representations (283, 284, 285, 286), and limitations of factual recall in transformers (287).

**It is challenging to understand how general-purpose AI systems operate internally and to use this understanding effectively.** A persistent problem is that, in the absence of an objective standard to compare to, it is difficult to ensure that an interpretation of how a general-purpose AI system operates is correct. A number of 'interpretability illusions', in which an interpretability technique suggests a misleading interpretation for how a model works, have been documented (288*, 289). In addition, some research has also critically examined how algorithmic transparency tools can be maliciously used to construct false dichotomies, obscure, and mislead (290). To rigorously evaluate
hiring tools on hiring manager discretion (249), and the use of generative AI on middle manager performance (250). Qualitative interviews of stakeholders have proven effective at illustrating more systematic impacts such as some of the social implications of AI implementation (251), which can endure after the removal of the AI system (252).

Analysing AI systems in the real world, post-deployment, allows researchers to study them as components of larger societal systems. Post-market surveillance is quite common in other industries with audit ecosystems (253). Users often discover capabilities and failure modes that developers do not, and monitoring the real-world usage of a system can further scientific understanding. For example, jailbreaks against modern large language chat models were first studied following findings from ordinary users (254). The study of deepfakes in the real world has also helped to shape scientific research on studying and mitigating harms (255, 256).

## 3.3 Model transparency, explanations, and interpretations

In contrast to studying general-purpose AI model outputs, another common approach to assessing models is to study the inner mechanism through which models produce the outputs. This can help researchers contextualise assessments of model performance and deepens their understanding of model functionality. Studying *how* general-purpose AI models and systems operate internally is a popular topic of research with thousands of academic papers produced. Areas of research that aim to enhance transparency include documentation, third-party access mechanisms, black box analysis, explaining model actions, and interpreting the inner workings of models.

**Documentation templates record decisions made and facilitate transparency at an operational level.** Currently, one of the most practical ways to increase transparency about a general-purpose AI model is through documenting and communicating the engineering decisions that define the model. Several documentation solutions have been proposed to communicate such decisions to a broader range of internal and external stakeholders. Some of these efforts, such as the development of Model Cards (257) have been successful. One recent study revealed "a widespread uptake of model cards within the AI community" (258). There are documentation templates available for communication on dataset practices (259, 260, 261), broader system features (262, 263), and broader procedural decision-making (264).

**Model explanation and interpretability techniques can improve researchers' understanding of how general-purpose AI systems operate internally.** There are several tools that allow for the external scrutiny of general-purpose AI systems, enabling access for external actors to query general-purpose AI systems directly, or otherwise gain visibility into model details (213). One prominent technical approach to this involves studying how a model's output can be explained as a consequence of a given input (265, 266, 267, 268). These explanations can play a unique role in supporting accountability, by helping to determine liability, in cases where humans may be wrongfully harmed or discriminated against by automated AI systems (269, 270, 271). Another approach used to study the computations in neural networks has involved interpreting the role of parameters (272), neurons (273, 274, 275), subnetworks (276, 277), or layer representations (278, 279, 280, 281) inside of AI systems. Interpretations of models have sometimes aided researchers in finding vulnerabilities. Examples have involved red-teaming (207), identifying internal representations of spurious features (282), brittle feature representations (283, 284, 285, 286), and limitations of factual recall in transformers (287).

**It is challenging to understand how general-purpose AI systems operate internally and to use this understanding effectively.** A persistent problem is that, in the absence of an objective standard to compare to, it is difficult to ensure that an interpretation of how a general-purpose AI system operates is correct. A number of 'interpretability illusions', in which an interpretability technique suggests a misleading interpretation for how a model works, have been documented (288*, 289). In addition, some research has also critically examined how algorithmic transparency tools can be maliciously used to construct false dichotomies, obscure, and mislead (290). To rigorously evaluate
interpretability techniques, the interpretations that they generate need to be demonstrably useful for some downstream task (291, 292, 293, 294*, 295), yet AI interpretability tools are not consistently competitive with simpler techniques for many tasks yet (296). In particular, different techniques to explain model actions often disagree with one another, and fail at sanity checks or downstream uses (218, 297, 298, 299). Nonetheless, interpretability has sometimes improved practical diagnostics and understanding, especially with recent progress in the field (300). Further progress may enable better applications. However, high-level interpretations of general-purpose AI systems cannot be used to make formal guarantees about their behaviour with current models and methods. The potential for current explainability and interpretability techniques to practically aid in rigorous model assessments is debated.

## 3.4 Challenges with studying general-purpose AI systems

It is extremely difficult to conduct thorough evaluations and generate strong assurances of general-purpose AI capabilities and risks. Modern general-purpose AI systems are the results of complex, decentralised projects involving data collection, training runs, system integration, and deployment applications. Meanwhile, there are a very large number of real-world uses for general-purpose AI systems. This complexity makes it difficult for any single actor to understand the entire process.

**The quality of evaluations depends on levels of access and transparency.** Different techniques for evaluating AI systems require different types of access. For example, evaluating a model's performance on test data typically only requires the ability to query the target model and analyse its outputs. This is typically referred to as 'black-box' access. The ability to query a black-box system is useful but many types of evaluation techniques depend on greater levels of access. Historically, AI researchers have benefitted from open-source methods, models, and data. Today, however, companies are increasingly keeping state-of-the-art general-purpose AI systems private (244). Lacking 'white-box' access (access to model parameters) makes it challenging for researchers to perform adversarial attacks, model interpretations, and fine-tuning (300, 301). Meanwhile, lacking 'outside-the-box' access to information about how a system was designed, including data, documentation, techniques, implementation details, and organisational details makes it difficult to perform evaluations of the development process (226, 227, 257, 300, 302, 303). Meanwhile, the third-party audit ecosystem is nascent but growing. Various AI audit tools, some of which are open-source, allow external users to query and access details of the model (213). Several studies have advocated for legal 'safe harbours' (304) or government-mediated access regimes (253) to enable independent red-teaming and audit efforts. Methods for structured access have been proposed that do not require making the code and weights public (305) but make it possible for independent researchers and auditors to perform their analysis with full access to the model in a secured environment designed to avoid leaks.

**Thoroughly assessing downstream societal impacts requires nuanced analysis, interdisciplinarity, and inclusion.** Although understanding the overall societal impact is the ultimate goal of many AI evaluations, many fall short of this goal. Firstly, there are always differences between the settings in which researchers study AI systems and the ever-changing real-world settings in which they will be deployed. Secondly, evaluating AI impacts on society is a complex socio-technical problem (306, 307). For example, while it is known that large language models exhibit significant cross-lingual differences in safety (308, 309), capabilities (310, 311*), and tendencies (312, 313*), it is challenging for researchers to thoroughly evaluate language models across languages. Furthermore, an over-reliance on simplified technical proxies when dealing with ethical concepts like 'fairness' and 'equity', can be misleading or exclude under-represented stakeholders (314, 315). Evaluations of the broader impacts of general-purpose AI are also highly multifaceted, requiring interdisciplinarity and representation of multiple stakeholders who may hold very different perspectives (316, 317, 318*). Modelling the effects of AI deployment in society, in advance of deployment, is inherently complex and difficult to anchor in quantitative analysis. While there has been progress on societal impact
interpretability techniques, the interpretations that they generate need to be demonstrably useful for some downstream task (291, 292, 293, 294*, 295), yet AI interpretability tools are not consistently competitive with simpler techniques for many tasks yet (296). In particular, different techniques to explain model actions often disagree with one another, and fail at sanity checks or downstream uses (218, 297, 298, 299). Nonetheless, interpretability has sometimes improved practical diagnostics and understanding, especially with recent progress in the field (300). Further progress may enable better applications. However, high-level interpretations of general-purpose AI systems cannot be used to make formal guarantees about their behaviour with current models and methods. The potential for current explainability and interpretability techniques to practically aid in rigorous model assessments is debated.

## 3.4 Challenges with studying general-purpose AI systems

It is extremely difficult to conduct thorough evaluations and generate strong assurances of general-purpose AI capabilities and risks. Modern general-purpose AI systems are the results of complex, decentralised projects involving data collection, training runs, system integration, and deployment applications. Meanwhile, there are a very large number of real-world uses for general-purpose AI systems. This complexity makes it difficult for any single actor to understand the entire process.

**The quality of evaluations depends on levels of access and transparency.** Different techniques for evaluating AI systems require different types of access. For example, evaluating a model's performance on test data typically only requires the ability to query the target model and analyse its outputs. This is typically referred to as 'black-box' access. The ability to query a black-box system is useful but many types of evaluation techniques depend on greater levels of access. Historically, AI researchers have benefitted from open-source methods, models, and data. Today, however, companies are increasingly keeping state-of-the-art general-purpose AI systems private (244). Lacking 'white-box' access (access to model parameters) makes it challenging for researchers to perform adversarial attacks, model interpretations, and fine-tuning (300, 301). Meanwhile, lacking 'outside-the-box' access to information about how a system was designed, including data, documentation, techniques, implementation details, and organisational details makes it difficult to perform evaluations of the development process (226, 227, 257, 300, 302, 303). Meanwhile, the third-party audit ecosystem is nascent but growing. Various AI audit tools, some of which are open-source, allow external users to query and access details of the model (213). Several studies have advocated for legal 'safe harbours' (304) or government-mediated access regimes (253) to enable independent red-teaming and audit efforts. Methods for structured access have been proposed that do not require making the code and weights public (305) but make it possible for independent researchers and auditors to perform their analysis with full access to the model in a secured environment designed to avoid leaks.

**Thoroughly assessing downstream societal impacts requires nuanced analysis, interdisciplinarity, and inclusion.** Although understanding the overall societal impact is the ultimate goal of many AI evaluations, many fall short of this goal. Firstly, there are always differences between the settings in which researchers study AI systems and the ever-changing real-world settings in which they will be deployed. Secondly, evaluating AI impacts on society is a complex socio-technical problem (306, 307). For example, while it is known that large language models exhibit significant cross-lingual differences in safety (308, 309), capabilities (310, 311*), and tendencies (312, 313*), it is challenging for researchers to thoroughly evaluate language models across languages. Furthermore, an over-reliance on simplified technical proxies when dealing with ethical concepts like 'fairness' and 'equity', can be misleading or exclude under-represented stakeholders (314, 315). Evaluations of the broader impacts of general-purpose AI are also highly multifaceted, requiring interdisciplinarity and representation of multiple stakeholders who may hold very different perspectives (316, 317, 318*). Modelling the effects of AI deployment in society, in advance of deployment, is inherently complex and difficult to anchor in quantitative analysis. While there has been progress on societal impact
evaluation for general-purpose Al (319, 320, 321, 322*, 323*), implementation remains challenging due
to the need to balance the interests of multiple stakeholders (324) and resourcing challenges (325)
that often make this work difficult to do in practice.

**Increasing participation and representation of perspectives in the AI development and evaluation process is an ongoing technical and institutional challenge.** Specifying relevant goals of evaluation is highly influenced by who is at the table and how the discussion is organised, meaning it is easy to miss or mis-define areas of concern. Broadening the range of those participating in the audit process also broadens the range of experiences incorporated into the process of discovering and characterising current or anticipated harms (215). Expanding participation has been a focus of the machine learning community in recent years, highlighting the need to incorporate and engage a broader range of perspectives and stakeholders into the process of AI model design, development, evaluation, and governance (326, 327, 328). Multiple strategies have been proposed from facilitating a broader notion of impacts during the implementation of impact assessments (329) to enable a more inclusive range of human feedback (330). However, seeking out and incorporating more voices is a nuanced endeavour, requiring sensitivity and respect for participating parties to minimise the potential for exploitation (331). Crucially, the challenge of increasing participation also involves the necessary negotiation of hard choices between incompatible values or priorities (332, 333, 334).

**While transparency is critical for assessing AI systems, it is difficult to achieve in practice.**
Transparency efforts are not always meaningful interventions toward accountability (290). Bhatt et al. (335) surveyed dozens of corporate implementations of explainability techniques and found that many explainability efforts are meaningfully used during the model development process but rarely live up to the policy aspirations of providing end-user transparency or justification. Furthermore, recent surveys of proprietary, open-source explainability and interpretability tooling reveal that such tools are rarely adequately validated to support claims, and are vulnerable themselves to manipulation and robustness challenges (213, 336, 337). Similarly, the logistics of operationalising documentation practice in corporate settings can be fraught with internal politics (338).
evaluation for general-purpose Al (319, 320, 321, 322*, 323*), implementation remains challenging due
to the need to balance the interests of multiple stakeholders (324) and resourcing challenges (325)
that often make this work difficult to do in practice.

**Increasing participation and representation of perspectives in the AI development and evaluation process is an ongoing technical and institutional challenge.** Specifying relevant goals of evaluation is highly influenced by who is at the table and how the discussion is organised, meaning it is easy to miss or mis-define areas of concern. Broadening the range of those participating in the audit process also broadens the range of experiences incorporated into the process of discovering and characterising current or anticipated harms (215). Expanding participation has been a focus of the machine learning community in recent years, highlighting the need to incorporate and engage a broader range of perspectives and stakeholders into the process of AI model design, development, evaluation, and governance (326, 327, 328). Multiple strategies have been proposed from facilitating a broader notion of impacts during the implementation of impact assessments (329) to enable a more inclusive range of human feedback (330). However, seeking out and incorporating more voices is a nuanced endeavour, requiring sensitivity and respect for participating parties to minimise the potential for exploitation (331). Crucially, the challenge of increasing participation also involves the necessary negotiation of hard choices between incompatible values or priorities (332, 333, 334).

**While transparency is critical for assessing AI systems, it is difficult to achieve in practice.**
Transparency efforts are not always meaningful interventions toward accountability (290). Bhatt et al. (335) surveyed dozens of corporate implementations of explainability techniques and found that many explainability efforts are meaningfully used during the model development process but rarely live up to the policy aspirations of providing end-user transparency or justification. Furthermore, recent surveys of proprietary, open-source explainability and interpretability tooling reveal that such tools are rarely adequately validated to support claims, and are vulnerable themselves to manipulation and robustness challenges (213, 336, 337). Similarly, the logistics of operationalising documentation practice in corporate settings can be fraught with internal politics (338).
4 Risks

The development and deployment of general-purpose Al gives rise to several risks, which are discussed in this section. This report distinguishes between 'risks' and 'cross-cutting risk factors'. For the purposes of this report 'risk' means the combination of the probability of an occurrence of harm and the severity of that harm (339). 'Cross-cutting risk factor' refers to a condition that contributes to not one but several risks.

4.1 Malicious use risks

As general-purpose Al covers a broad set of knowledge areas, it can be repurposed for malicious ends, potentially causing widespread harm. This section discusses some of the major risks of malicious use, but there are others and new risks may continue to emerge. While the risks discussed in this section range widely in terms of how well-evidenced they are, and in some cases, there is evidence suggesting that they may currently not be serious risks at all, we include them to provide a comprehensive overview of the malicious use risks associated with general-purpose Al systems.

4.1.1 Harm to individuals through fake content

KEY INFORMATION

* General-purpose Al systems can be used to increase the scale and sophistication of scams and fraud, for example through general-purpose Al-enhanced 'phishing' attacks.

* General-purpose Al can be used to generate fake compromising content featuring individuals without their consent, posing threats to individual privacy and reputation.

General-purpose Al can amplify the risk of frauds and scams, increasing both their volume and their sophistication. Their volume can be increased because general-purpose Al facilitates the generation of scam content at greater speeds and scale than previously possible. Their sophistication can be increased because general-purpose Al facilitates the creation of more convincing and personalised scam content at scale (340, 341). General-purpose Al language models can be used to design and deploy 'phishing' attacks in which attackers deceive people into sharing passwords or other sensitive information (342). This can include spear-phishing, a type of phishing campaign that is personalised to the target, and business email compromise, a type of cybercrime where the malicious user tries to trick someone into sending money or sharing confidential information. Research has found that between January to February 2023, there was a 135% increase in 'novel social engineering attacks' in a sample of email accounts (343*), which is thought to correspond to the widespread adoption of ChatGPT.

General-purpose Al language models have the potential to dramatically scale the reach of fraudsters by automating conversations with potential victims to find promising targets (340). General-purpose Al could also contribute to targeted identity theft, and generating fake identities which could be used for illegal purposes. For example, a few years ago, research had discussed potential risks from Al 'voice cloning' - where an Al system analyses the tone, pitch, and other characteristics of a human voice to create synthetic copies - and how such technology might be used by fraudsters to pretend to be their victim's friends or a trusted authority (344). General-purpose Al systems can also help criminals to evade security software by correcting language errors and improving the fluency of messages that might otherwise be caught by spam filters.
4 Risks

The development and deployment of general-purpose Al gives rise to several risks, which are discussed in this section. This report distinguishes between 'risks' and 'cross-cutting risk factors'. For the purposes of this report 'risk' means the combination of the probability of an occurrence of harm and the severity of that harm (339). 'Cross-cutting risk factor' refers to a condition that contributes to not one but several risks.

4.1 Malicious use risks

As general-purpose Al covers a broad set of knowledge areas, it can be repurposed for malicious ends, potentially causing widespread harm. This section discusses some of the major risks of malicious use, but there are others and new risks may continue to emerge. While the risks discussed in this section range widely in terms of how well-evidenced they are, and in some cases, there is evidence suggesting that they may currently not be serious risks at all, we include them to provide a comprehensive overview of the malicious use risks associated with general-purpose Al systems.

4.1.1 Harm to individuals through fake content

KEY INFORMATION

* General-purpose Al systems can be used to increase the scale and sophistication of scams and fraud, for example through general-purpose Al-enhanced 'phishing' attacks.

* General-purpose Al can be used to generate fake compromising content featuring individuals without their consent, posing threats to individual privacy and reputation.

General-purpose Al can amplify the risk of frauds and scams, increasing both their volume and their sophistication. Their volume can be increased because general-purpose Al facilitates the generation of scam content at greater speeds and scale than previously possible. Their sophistication can be increased because general-purpose Al facilitates the creation of more convincing and personalised scam content at scale (340, 341). General-purpose Al language models can be used to design and deploy 'phishing' attacks in which attackers deceive people into sharing passwords or other sensitive information (342). This can include spear-phishing, a type of phishing campaign that is personalised to the target, and business email compromise, a type of cybercrime where the malicious user tries to trick someone into sending money or sharing confidential information. Research has found that between January to February 2023, there was a 135% increase in 'novel social engineering attacks' in a sample of email accounts (343*), which is thought to correspond to the widespread adoption of ChatGPT.

General-purpose Al language models have the potential to dramatically scale the reach of fraudsters by automating conversations with potential victims to find promising targets (340). General-purpose Al could also contribute to targeted identity theft, and generating fake identities which could be used for illegal purposes. For example, a few years ago, research had discussed potential risks from Al 'voice cloning' - where an Al system analyses the tone, pitch, and other characteristics of a human voice to create synthetic copies - and how such technology might be used by fraudsters to pretend to be their victim's friends or a trusted authority (344). General-purpose Al systems can also help criminals to evade security software by correcting language errors and improving the fluency of messages that might otherwise be caught by spam filters.
Recent data indicates that Al-enabled fraud, particularly using deepfakes, is growing globally (345, 346). Detecting the use of general-purpose Al systems to generate content used to commit fraud can be difficult, and institutions may be reluctant to disclose the challenges they are facing with Al-powered fraud (347).

General-purpose Al-generated fake content can also be employed to harm individuals by featuring them in that content against their consent, thereby violating their right to privacy and damaging their reputation or dignity. This can happen in the form of fake content featuring any compromising or reputationally damaging activity, but has received particular attention in cases of deepfake pornography, where general-purpose Al is used to create pornographic audiovisual content of individuals without their consent. This may include the creation of child sexual abuse material (CSAM) and other forms of intimate images used to abuse, for example, former domestic partners or for blackmail (348, 349, 350).

## 4.1.2 Disinformation and manipulation of public opinion

### KEY INFORMATION

* General-purpose Al makes it possible to generate and disseminate disinformation at an unprecedented scale and with a high degree of sophistication, which could have serious implications for political processes. However, it is debated how impactful political disinformation campaigns generally are.

* It can be difficult to detect disinformation generated by general-purpose Al because the outputs are increasingly realistic. Technical countermeasures, like watermarking content, are useful but can usually be circumvented by moderately sophisticated actors.

Al, particularly general-purpose Al, can be maliciously used for disinformation (351), which for the purpose of this report refers to false information that was generated or spread with the deliberate intent to mislead or deceive. General-purpose Al-generated text can be indistinguishable from genuine human-generated material (352, 353), and may already be disseminated at scale on social media (354). In addition, general-purpose Al systems can be used to not only generate text but also fully synthetic or misleadingly altered images, audio, and video content (355). Humans often find such content indistinguishable from genuine examples, and generating such content is both relatively simple and extremely cheap (356). An example of this is images of human faces that were altered, or completely generated, using general-purpose Al or narrower types of Al systems (357). Such 'Deepfake' images and videos are thought to have been deployed in several national elections over recent months to defame political opponents, with potentially significant impact, but there is currently not much scientific evidence about the impact of such campaigns.

**General-purpose Al tools might be used to persuade and manipulate people, which could have serious implications for political processes.** General-purpose Al systems can be used to generate highly persuasive content at scale. This could, for example, be used in a commercial setting for advertising, or during an election campaign to influence public opinion (358). Recent work has measured the persuasiveness of general-purpose Al-generated political messages and found that it can somewhat sway the opinion of the people reading these messages (359, 360). In addition, general-purpose Al can be used to tailor persuasive content to specific individuals or demographics (known as 'microtargeting'), for example, by using information scraped from social media. However, there is currently only limited evidence that micro-targeted messages are more persuasive than generic general-purpose Al-produced content (361, 362), which is in line with emerging scepticism about the effectiveness of microtargeting in general (363).
Recent data indicates that Al-enabled fraud, particularly using deepfakes, is growing globally (345, 346). Detecting the use of general-purpose Al systems to generate content used to commit fraud can be difficult, and institutions may be reluctant to disclose the challenges they are facing with Al-powered fraud (347).

General-purpose Al-generated fake content can also be employed to harm individuals by featuring them in that content against their consent, thereby violating their right to privacy and damaging their reputation or dignity. This can happen in the form of fake content featuring any compromising or reputationally damaging activity, but has received particular attention in cases of deepfake pornography, where general-purpose Al is used to create pornographic audiovisual content of individuals without their consent. This may include the creation of child sexual abuse material (CSAM) and other forms of intimate images used to abuse, for example, former domestic partners or for blackmail (348, 349, 350).

## 4.1.2 Disinformation and manipulation of public opinion

### KEY INFORMATION

* General-purpose Al makes it possible to generate and disseminate disinformation at an unprecedented scale and with a high degree of sophistication, which could have serious implications for political processes. However, it is debated how impactful political disinformation campaigns generally are.

* It can be difficult to detect disinformation generated by general-purpose Al because the outputs are increasingly realistic. Technical countermeasures, like watermarking content, are useful but can usually be circumvented by moderately sophisticated actors.

Al, particularly general-purpose Al, can be maliciously used for disinformation (351), which for the purpose of this report refers to false information that was generated or spread with the deliberate intent to mislead or deceive. General-purpose Al-generated text can be indistinguishable from genuine human-generated material (352, 353), and may already be disseminated at scale on social media (354). In addition, general-purpose Al systems can be used to not only generate text but also fully synthetic or misleadingly altered images, audio, and video content (355). Humans often find such content indistinguishable from genuine examples, and generating such content is both relatively simple and extremely cheap (356). An example of this is images of human faces that were altered, or completely generated, using general-purpose Al or narrower types of Al systems (357). Such 'Deepfake' images and videos are thought to have been deployed in several national elections over recent months to defame political opponents, with potentially significant impact, but there is currently not much scientific evidence about the impact of such campaigns.

**General-purpose Al tools might be used to persuade and manipulate people, which could have serious implications for political processes.** General-purpose Al systems can be used to generate highly persuasive content at scale. This could, for example, be used in a commercial setting for advertising, or during an election campaign to influence public opinion (358). Recent work has measured the persuasiveness of general-purpose Al-generated political messages and found that it can somewhat sway the opinion of the people reading these messages (359, 360). In addition, general-purpose Al can be used to tailor persuasive content to specific individuals or demographics (known as 'microtargeting'), for example, by using information scraped from social media. However, there is currently only limited evidence that micro-targeted messages are more persuasive than generic general-purpose Al-produced content (361, 362), which is in line with emerging scepticism about the effectiveness of microtargeting in general (363).
An underexplored frontier is the use of conversational general-purpose Al systems to persuade users over multiple turns of dialogue. In one study, human-human or human-general-purpose Al pairs debated over a cycle of opinion-and-rebuttal, and general-purpose Al systems were found to be as persuasive as humans (364). Another study in which general-purpose Al systems attempted to dissuade humans of their belief in conspiracy theories over three conversational turns found reductions in reported belief in those theories of 15-20% that endured for up to two months (365). These results raise the possibility that in conversational settings, general-purpose Al systems could be used to very powerful persuasive effect. As general-purpose Al systems grow in capability, it might become easier to maliciously use them for deceptive or manipulative means, possibly even with higher effectiveness than skilled humans, to encourage users to take actions that are against their own best interests (366, 367*). In doing so, they may utilise new manipulation tactics against which humans are not prepared because our defences against manipulation have been developed through the influencing attempts of other humans (368).

**The overall impact of disinformation campaigns in general as well as the impact of widespread dissemination of general-purpose Al-generated media are still not well understood.** Despite indications of potentially serious risks to public discourse, and the integrity of the information ecosystem posed by general-purpose Al, there are caveats. First, there is a lack of evidence about the effectiveness of large-scale disinformation campaigns in general (whether using general-purpose Al or not). Second, some experts have argued that the main bottleneck for actors trying to have a large-scale impact with fake content is not generating that content, but distributing it at scale (369). Similarly, some research suggests that 'cheapfakes' (less sophisticated methods of manipulating audiovisual content that are not dependent on general-purpose Al use), might be as harmful as more sophisticated deepfakes (370). If true, this would support the hypothesis that the quality of fake content is currently less decisive for the success of a disinformation campaign than challenges around distributing that content to many users. Social media platforms like Meta or X employ various techniques such as human content moderation and labelling for reducing the reach of content likely to be disinformation, including general-purpose Al-generated disinformation. On the other hand, research has shown for years that social media algorithms often prioritise engagement and virality over the accuracy or authenticity of content, which could aid the rapid spread of Al-generated disinformation (371, 372).

In general, as general-purpose Al capabilities grow and are increasingly used for generating and spreading messages at scale, be they accurate, intentionally false, or unintentionally false, people might come to trust any information less, which could pose serious problems for public deliberation. Malicious actors could exploit such a generalised loss of trust by denying the truth of real, unfavourable evidence, claiming it is Al-generated, a phenomenon coined as the 'liars' dividend' (373).

**Potential measures to identify general-purpose Al-generated content, such as watermarking, can be helpful but are easy to circumvent for moderately sophisticated actors.** Researchers have employed various methods attempting to identify potential Al authorship (374, 375). *Content analysis* techniques explore statistical properties of text, such as unusual character frequencies or inconsistent sentence length distributions, which may deviate from patterns typically observed in human writing (376, 377, 378). Linguistic analysis techniques examine stylistic elements like sentiment or named entity recognition to uncover inconsistencies or unnatural language patterns indicative of Al generation (379, 380). Readability scores can also be used to examine where general-purpose Al-generated text might score unusually high or low on metrics like the Flesch Reading Ease Score compared to human-written content (381). Al researchers have also proposed other disinformation detection approaches, such as watermarking, in which an invisible signature identifies digital content as generated or altered by Al (382). However, current techniques are relatively easy to circumvent for moderately skilled actors (374), and perfect defences against the removal of watermarks are probably impossible (383). Still, these safeguards may deter relatively unsophisticated threat actors (384). [Technical approaches to mitigate risks](#) provides a more in-depth discussion of watermarking techniques.
An underexplored frontier is the use of conversational general-purpose Al systems to persuade users over multiple turns of dialogue. In one study, human-human or human-general-purpose Al pairs debated over a cycle of opinion-and-rebuttal, and general-purpose Al systems were found to be as persuasive as humans (364). Another study in which general-purpose Al systems attempted to dissuade humans of their belief in conspiracy theories over three conversational turns found reductions in reported belief in those theories of 15-20% that endured for up to two months (365). These results raise the possibility that in conversational settings, general-purpose Al systems could be used to very powerful persuasive effect. As general-purpose Al systems grow in capability, it might become easier to maliciously use them for deceptive or manipulative means, possibly even with higher effectiveness than skilled humans, to encourage users to take actions that are against their own best interests (366, 367*). In doing so, they may utilise new manipulation tactics against which humans are not prepared because our defences against manipulation have been developed through the influencing attempts of other humans (368).

**The overall impact of disinformation campaigns in general as well as the impact of widespread dissemination of general-purpose Al-generated media are still not well understood.** Despite indications of potentially serious risks to public discourse, and the integrity of the information ecosystem posed by general-purpose Al, there are caveats. First, there is a lack of evidence about the effectiveness of large-scale disinformation campaigns in general (whether using general-purpose Al or not). Second, some experts have argued that the main bottleneck for actors trying to have a large-scale impact with fake content is not generating that content, but distributing it at scale (369). Similarly, some research suggests that 'cheapfakes' (less sophisticated methods of manipulating audiovisual content that are not dependent on general-purpose Al use), might be as harmful as more sophisticated deepfakes (370). If true, this would support the hypothesis that the quality of fake content is currently less decisive for the success of a disinformation campaign than challenges around distributing that content to many users. Social media platforms like Meta or X employ various techniques such as human content moderation and labelling for reducing the reach of content likely to be disinformation, including general-purpose Al-generated disinformation. On the other hand, research has shown for years that social media algorithms often prioritise engagement and virality over the accuracy or authenticity of content, which could aid the rapid spread of Al-generated disinformation (371, 372).

In general, as general-purpose Al capabilities grow and are increasingly used for generating and spreading messages at scale, be they accurate, intentionally false, or unintentionally false, people might come to trust any information less, which could pose serious problems for public deliberation. Malicious actors could exploit such a generalised loss of trust by denying the truth of real, unfavourable evidence, claiming it is Al-generated, a phenomenon coined as the 'liars' dividend' (373).

**Potential measures to identify general-purpose Al-generated content, such as watermarking, can be helpful but are easy to circumvent for moderately sophisticated actors.** Researchers have employed various methods attempting to identify potential Al authorship (374, 375). *Content analysis* techniques explore statistical properties of text, such as unusual character frequencies or inconsistent sentence length distributions, which may deviate from patterns typically observed in human writing (376, 377, 378). Linguistic analysis techniques examine stylistic elements like sentiment or named entity recognition to uncover inconsistencies or unnatural language patterns indicative of Al generation (379, 380). Readability scores can also be used to examine where general-purpose Al-generated text might score unusually high or low on metrics like the Flesch Reading Ease Score compared to human-written content (381). Al researchers have also proposed other disinformation detection approaches, such as watermarking, in which an invisible signature identifies digital content as generated or altered by Al (382). However, current techniques are relatively easy to circumvent for moderately skilled actors (374), and perfect defences against the removal of watermarks are probably impossible (383). Still, these safeguards may deter relatively unsophisticated threat actors (384). **5. Technical approaches to mitigate risks** provides a more in-depth discussion of watermarking techniques.
4.1.3 Cyber offence

KEY INFORMATION

* General-purpose Al systems could uplift the cyber expertise of individuals, making it easier for malicious users to conduct effective cyber-attacks, as well as providing a tool that can be used in cyber defence. General-purpose Al systems can be used to automate and scale some types of cyber operations, such as social engineering attacks.
* There is no substantial evidence yet suggesting that general-purpose Al can automate sophisticated cybersecurity tasks which could tip the balance between cyber attackers and defenders in favour of the attackers.

General-purpose Al systems can exacerbate existing cybersecurity risks in several ways. Firstly, they may lower the barrier to entry of more sophisticated cyber attacks, so the number of people capable of such attacks might increase. Secondly, general-purpose Al systems could be used to scale offensive cyber operations, through increasing levels of automation and efficiency.

Moreover, general-purpose Al can inadvertently leave systems vulnerable to traditional cybersecurity attacks. For example, general-purpose Al systems are used pervasively as coding assistants and can inadvertently introduce software vulnerabilities (385, 386*, 387).

**General-purpose Al systems reduce the cost, technical know-how, and expertise needed to conduct cyber-attacks.** Offensive cyber operations include designing and spreading malicious software as well as discovering and exploiting vulnerabilities in critical systems. They can lead to significant security breaches, for example in critical national infrastructure (CNI), and pose a threat to public safety and security. Given the labour-intensive nature of these operations, advanced general-purpose Al that automates certain aspects of the process, reducing the number of experts needed and lowering the required level of expertise, could be useful for attackers.

So far, current general-purpose Al systems have been shown to be capable of autonomously carrying out basic cybersecurity challenges and narrow cyber tasks such as hacking a highly insecure website (388, 389*, 390, 391). However, despite consistent research efforts, existing models appear to not be able to carry out multi-step cybersecurity tasks that require longer horizon planning (367*, 391, 392).

Given that LLMs can already process and manipulate some cybersecurity concepts (393), planning could unlock more sophisticated cyber capabilities such as independently navigating complex digital environments, identifying and exploiting vulnerabilities at scale, and executing long-horizon strategies, without the need for direct human guidance.

<figure><img src="image_11.png" /><figcaption>Figure 6. An AI-enabled pipeline for automating simple attacks against websites (from (391)). A novice hacker supplies the general-purpose Al model with security information sources and textbooks, and prompts it to interact with their target website. Through trial and error, the general-purpose Al model identifies an attack that works on this website and returns the details to the hacker.</figcaption></figure>
4.1.3 Cyber offence

KEY INFORMATION

* General-purpose Al systems could uplift the cyber expertise of individuals, making it easier for malicious users to conduct effective cyber-attacks, as well as providing a tool that can be used in cyber defence. General-purpose Al systems can be used to automate and scale some types of cyber operations, such as social engineering attacks.
* There is no substantial evidence yet suggesting that general-purpose Al can automate sophisticated cybersecurity tasks which could tip the balance between cyber attackers and defenders in favour of the attackers.

General-purpose Al systems can exacerbate existing cybersecurity risks in several ways. Firstly, they may lower the barrier to entry of more sophisticated cyber attacks, so the number of people capable of such attacks might increase. Secondly, general-purpose Al systems could be used to scale offensive cyber operations, through increasing levels of automation and efficiency.

Moreover, general-purpose Al can inadvertently leave systems vulnerable to traditional cybersecurity attacks. For example, general-purpose Al systems are used pervasively as coding assistants and can inadvertently introduce software vulnerabilities (385, 386*, 387).

**General-purpose Al systems reduce the cost, technical know-how, and expertise needed to conduct cyber-attacks.** Offensive cyber operations include designing and spreading malicious software as well as discovering and exploiting vulnerabilities in critical systems. They can lead to significant security breaches, for example in critical national infrastructure (CNI), and pose a threat to public safety and security. Given the labour-intensive nature of these operations, advanced general-purpose Al that automates certain aspects of the process, reducing the number of experts needed and lowering the required level of expertise, could be useful for attackers.

So far, current general-purpose Al systems have been shown to be capable of autonomously carrying out basic cybersecurity challenges and narrow cyber tasks such as hacking a highly insecure website (388, 389*, 390, 391). However, despite consistent research efforts, existing models appear to not be able to carry out multi-step cybersecurity tasks that require longer horizon planning (367*, 391, 392).

Given that LLMs can already process and manipulate some cybersecurity concepts (393), planning could unlock more sophisticated cyber capabilities such as independently navigating complex digital environments, identifying and exploiting vulnerabilities at scale, and executing long-horizon strategies, without the need for direct human guidance.

<figure><img src="image_11.png" /><figcaption>Figure 6. An AI-enabled pipeline for automating simple attacks against websites (from (391)). A novice hacker supplies the general-purpose Al model with security information sources and textbooks, and prompts it to interact with their target website. Through trial and error, the general-purpose Al model identifies an attack that works on this website and returns the details to the hacker.</figcaption></figure>
**General-purpose Al systems could improve defensive cyber capabilities if applied across relevant areas.** Despite improved offensive capabilities, adversaries will not inevitably succeed, as advancements in general-purpose Al systems will also enhance defensive capabilities (394, 395). For instance, general-purpose Al systems can significantly cut down the time humans spend identifying and fixing vulnerabilities (396*, 397). However, similar to their attacking capabilities, the cyber-defence benefits of existing models also have limitations (367*, 398, 399, 400*, 401). Given the labour and time required to implement security patches across the digital infrastructure, attackers still have the opportunity to achieve successful breaches in systems that have not yet been fixed.

As general-purpose Al systems improve and become more widely used, the dynamics between attackers and defenders can be influenced by organisational factors such as resource availability and expertise levels. Major cyber defence competitions, such as the (402*), alongside high-quality datasets and benchmarks (403), can drive the development of more sophisticated and resilient cybersecurity measures.

### 4.1.4 Dual use science risks

**KEY INFORMATION:**

* General-purpose Al systems could accelerate advances in a range of scientific endeavours, from training new scientists to enabling faster research workflows. While these capabilities could have numerous beneficial applications, some experts have expressed concern that they could be used for malicious purposes, especially if further capabilities are developed soon before appropriate countermeasures are put in place.

* General-purpose Al systems for biological uses do not present a clear current threat, and future threats are hard to assess and rule out. In the biology domain, current general-purpose Al systems demonstrate growing capabilities but the limited studies available do not provide clear evidence that current systems can 'uplift' malicious actors to obtain biological pathogens better than using the internet. There is insufficient publicly available research to assess if near-term advances will provide this uplift, for example through trouble-shooting hands-on laboratory work.

* Due to insufficient scientific work, this interim report does not assess the risks of malicious use leading to chemical, radiological, and nuclear risks.

There are two avenues by which general-purpose Al systems could, speculatively, facilitate malicious use in the life sciences: firstly by providing increased access to information and expertise relevant to malicious use, and secondly by increasing the ceiling of capabilities, which may enable the development of more harmful versions of existing threats or, eventually, lead to novel threats (404, 405).

**Current capabilities**

**Increased access to information** - Advanced general-purpose Al models can provide scientific knowledge, step-by-step experimental protocols, and guidance for troubleshooting experiments, all of which could potentially be exploited for malicious purposes (405, 406, 407, 408). However, information relevant to biological threat creation is already broadly accessible, given its 'dual-use' nature (409, 410). The capabilities of current general-purpose Al systems to 'uplift' a malicious actor's ability to maliciously use biology, by increasing their ability to access information relative to existing resources such as the internet, is unclear. Few empirical studies have assessed if current general-
**General-purpose Al systems could improve defensive cyber capabilities if applied across relevant areas.** Despite improved offensive capabilities, adversaries will not inevitably succeed, as advancements in general-purpose Al systems will also enhance defensive capabilities (394, 395). For instance, general-purpose Al systems can significantly cut down the time humans spend identifying and fixing vulnerabilities (396*, 397). However, similar to their attacking capabilities, the cyber-defence benefits of existing models also have limitations (367*, 398, 399, 400*, 401). Given the labour and time required to implement security patches across the digital infrastructure, attackers still have the opportunity to achieve successful breaches in systems that have not yet been fixed.

As general-purpose Al systems improve and become more widely used, the dynamics between attackers and defenders can be influenced by organisational factors such as resource availability and expertise levels. Major cyber defence competitions, such as the (402*), alongside high-quality datasets and benchmarks (403), can drive the development of more sophisticated and resilient cybersecurity measures.

### 4.1.4 Dual use science risks

**KEY INFORMATION:**

* General-purpose Al systems could accelerate advances in a range of scientific endeavours, from training new scientists to enabling faster research workflows. While these capabilities could have numerous beneficial applications, some experts have expressed concern that they could be used for malicious purposes, especially if further capabilities are developed soon before appropriate countermeasures are put in place.

* General-purpose Al systems for biological uses do not present a clear current threat, and future threats are hard to assess and rule out. In the biology domain, current general-purpose Al systems demonstrate growing capabilities but the limited studies available do not provide clear evidence that current systems can 'uplift' malicious actors to obtain biological pathogens better than using the internet. There is insufficient publicly available research to assess if near-term advances will provide this uplift, for example through trouble-shooting hands-on laboratory work.

* Due to insufficient scientific work, this interim report does not assess the risks of malicious use leading to chemical, radiological, and nuclear risks.

There are two avenues by which general-purpose Al systems could, speculatively, facilitate malicious use in the life sciences: firstly by providing increased access to information and expertise relevant to malicious use, and secondly by increasing the ceiling of capabilities, which may enable the development of more harmful versions of existing threats or, eventually, lead to novel threats (404, 405).

**Current capabilities**

**Increased access to information** - Advanced general-purpose Al models can provide scientific knowledge, step-by-step experimental protocols, and guidance for troubleshooting experiments, all of which could potentially be exploited for malicious purposes (405, 406, 407, 408). However, information relevant to biological threat creation is already broadly accessible, given its 'dual-use' nature (409, 410). The capabilities of current general-purpose Al systems to 'uplift' a malicious actor's ability to maliciously use biology, by increasing their ability to access information relative to existing resources such as the internet, is unclear. Few empirical studies have assessed if current general-
purpose Al models provide an uplift and current evidence is mixed (166*, 411). Methodologies to
quantify uplift are also nascent and face limitations.<fnref n="4" />

**Increased access to hands-on expertise** - Access to information alone is insufficient to enable biological malicious use. Malicious actors must also be able to successfully synthesise, weaponise, and deliver the biological agent. Historically, those processes have required specialised expertise and hands-on skills for practical laboratory tasks (412, 413). Experts theorise that bioengineering combined with general-purpose Al advances may have lowered these barriers (414). However, existing studies have not evaluated whether general-purpose Al systems uplift malicious actors with practical laboratory research tasks. Existing advanced general-purpose Al models have some ability to design and troubleshoot laboratory experiments (407). These capabilities can be enhanced by equipping LLMs with the ability to access tools such as web search and specialised computational tools (415). When connected to laboratory robots or cloud labs, LLMs have been experimentally used to directly instruct these platforms to carry out experiments (189). However further empirical work is required to determine whether these existing general-purpose Al systems' capabilities 'uplift' actors with practical laboratory research tasks, relative to the internet's capabilities.

**Increasing the ceiling of capabilities** - While this report focuses on general-purpose Al systems (as defined in 1. Introduction), dual-use science risks are also impacted by the existence of narrow Al tools and the ability for general-purpose Al systems to interact with those narrow Al tools. Narrow Al biological tools can already redesign existing proteins to enhance existing functionality (416) and confer new biological functions (417), as well as generate novel proteins (418, 419). Analogous design capabilities have been demonstrated in other scientific domains, including chemistry (420). The capabilities of narrow tools themselves have dual-use implications: for instance, their use to predict viral mutations likely to demonstrate immune evasion (421), or generate plausible novel toxic molecules (422). Narrow Al tools are also often widely available, which makes implementing meaningful safeguards challenging (43, 419, 423*). In addition, general-purpose Al systems are able to direct laboratory robots using language instructions, and make use of specialised chemical computational tools (189, 415, 424).

Projected future capabilities

The malicious use potential of future general-purpose Al systems could be influenced by a range of projected advances. These include: advances in model capabilities, the integration of general-purpose Al systems with narrow Al tools, and the integration of general-purpose Al with automated laboratory equipment. Though there is some evidence of these projected advances being realised, it remains highly uncertain whether these capabilities will uplift users' abilities relative to existing resources.

**Advances in general-purpose Al model capabilities** - Future general-purpose Al systems will, plausibly, feature even greater domain-specific knowledge, reasoning abilities, and the capacity to formulate complex plans. Experts disagree on how much general-purpose Al systems could enable troubleshooting practical laboratory tasks, compared to using internet search. There is agreement that more effective, real-time troubleshooting for practical laboratory tasks would be enabled by 'multimodal' general-purpose Al systems that can incorporate information from images and videos (405, 407, 425). However, this possibility has not been tested yet. Additionally, some researchers argue that general-purpose Al systems trained on biological data may eventually outperform narrower tools designed for specific biological applications (426), but general-purpose Al systems are yet to demonstrate such capabilities, and the future performance and scalability of general-purpose Al systems in the biological domain remain unclear.

<footnote n="4">⁴ For instance, there is uncertainty about the use of statistical significance as an effective measure of measuring model risk, and it is challenging to operationalise the threshold at which increased information access would actually be dangerous.</footnote>
purpose Al models provide an uplift and current evidence is mixed (166*, 411). Methodologies to quantify uplift are also nascent and face limitations.<fnref n="4" />

**Increased access to hands-on expertise** - Access to information alone is insufficient to enable biological malicious use. Malicious actors must also be able to successfully synthesise, weaponise, and deliver the biological agent. Historically, those processes have required specialised expertise and hands-on skills for practical laboratory tasks (412, 413). Experts theorise that bioengineering combined with general-purpose Al advances may have lowered these barriers (414). However, existing studies have not evaluated whether general-purpose Al systems uplift malicious actors with practical laboratory research tasks. Existing advanced general-purpose Al models have some ability to design and troubleshoot laboratory experiments (407). These capabilities can be enhanced by equipping LLMs with the ability to access tools such as web search and specialised computational tools (415). When connected to laboratory robots or cloud labs, LLMs have been experimentally used to directly instruct these platforms to carry out experiments (189). However further empirical work is required to determine whether these existing general-purpose Al systems' capabilities 'uplift' actors with practical laboratory research tasks, relative to the internet's capabilities.

**Increasing the ceiling of capabilities** - While this report focuses on general-purpose Al systems (as defined in 1. Introduction), dual-use science risks are also impacted by the existence of narrow Al tools and the ability for general-purpose Al systems to interact with those narrow Al tools. Narrow Al biological tools can already redesign existing proteins to enhance existing functionality (416) and confer new biological functions (417), as well as generate novel proteins (418, 419). Analogous design capabilities have been demonstrated in other scientific domains, including chemistry (420). The capabilities of narrow tools themselves have dual-use implications: for instance, their use to predict viral mutations likely to demonstrate immune evasion (421), or generate plausible novel toxic molecules (422). Narrow Al tools are also often widely available, which makes implementing meaningful safeguards challenging (43, 419, 423*). In addition, general-purpose Al systems are able to direct laboratory robots using language instructions, and make use of specialised chemical computational tools (189, 415, 424).

## Projected future capabilities

The malicious use potential of future general-purpose Al systems could be influenced by a range of projected advances. These include: advances in model capabilities, the integration of general-purpose Al systems with narrow Al tools, and the integration of general-purpose Al with automated laboratory equipment. Though there is some evidence of these projected advances being realised, it remains highly uncertain whether these capabilities will uplift users' abilities relative to existing resources.

**Advances in general-purpose Al model capabilities** - Future general-purpose Al systems will, plausibly, feature even greater domain-specific knowledge, reasoning abilities, and the capacity to formulate complex plans. Experts disagree on how much general-purpose Al systems could enable troubleshooting practical laboratory tasks, compared to using internet search. There is agreement that more effective, real-time troubleshooting for practical laboratory tasks would be enabled by 'multimodal' general-purpose Al systems that can incorporate information from images and videos (405, 407, 425). However, this possibility has not been tested yet. Additionally, some researchers argue that general-purpose Al systems trained on biological data may eventually outperform narrower tools designed for specific biological applications (426), but general-purpose Al systems are yet to demonstrate such capabilities, and the future performance and scalability of general-purpose Al systems in the biological domain remain unclear.

<footnote n="4">⁴ For instance, there is uncertainty about the use of statistical significance as an effective measure of measuring model risk, and it is challenging to operationalise the threshold at which increased information access would actually be dangerous.</footnote>
**Integration with narrow tools** – Though general-purpose Al systems can make use of narrow Al tools, this type of integration has so far been limited (189, 415, 427*). It is uncertain how increasing the integration of general-purpose Al systems with narrow Al tools will affect the risk of malicious use. Today, narrow Al tools require specialist expertise to use effectively for scientific research (428). Though the ability to direct these tools using natural language will make specialised tasks more accessible, leveraging the outputs will likely still require technical expertise until substantial advances are made with general-purpose Al. Existing tools also have limited outputs and require extensive laboratory validation, and the extent to which these limitations will be overcome is unclear.

**Autonomous science capabilities** – While advanced general-purpose Al systems are already enabling some autonomous science capabilities, it is not clear what the near-term implications are for potential biological malicious use. While aspects of chemistry workflows such as chemical synthesis are already automatable (189, 415), experts are uncertain if these advances will transfer well to biology workflows, due to challenges automating work involving living systems (407). Furthermore, the high costs of automated laboratories are likely to make large-scale automation largely inaccessible to all but the most advanced malicious actors, although commercial cloud labs could partially offset this.

**In summary,** the degree to which current state-of-the-art general-purpose Al systems enhance the capabilities of malicious actors to use the life sciences over existing resources, such as the internet, remains unclear. Though some empirical work has assessed this uplift with respect to information access and biological threats (166*, 429), additional studies evaluating a broader range of tasks and scientific domains are needed to provide greater insight into this question. It has not yet been investigated whether general-purpose Al systems could also strengthen defences against dual-use science hazards.

## 4.2 Risks from malfunctions

### 4.2.1 Risks from product functionality issues

#### KEY INFORMATION

* Product functionality issues occur when there is confusion or misinformation about what a general-purpose Al model or system is capable of. This can lead to unrealistic expectations and overreliance on general-purpose Al systems, potentially causing harm if a system fails to deliver on expected capabilities.

* These functionality misconceptions may arise from technical difficulties in assessing an Al model's true capabilities on its own, or predicting its performance when part of a larger system. Misleading claims in advertising and communications can also contribute to these misconceptions.

Risks may arise where general-purpose Al models and systems fail to comply with general tenets of product safety and product functionality. As with many products, risks from general-purpose Al-based products occur because of misunderstandings of functionality and inadequate guidance for appropriate and safe use. In that respect, general-purpose Al-based products may be no different (430).

Product functionality issues, and the risks they pose may be clustered by potential failure modes (see Table 1).

*Impossible tasks* arise from instances of an attempt to accomplish goals with a general-purpose Al system that goes beyond the general-purpose Al system's capability. It can be hard to say definitively what constitutes an impossible task in a modern setting. Historically, large language models have not been able to consider events or developments that occurred after the end of their training. However,
**Integration with narrow tools** - Though general-purpose Al systems can make use of narrow Al tools, this type of integration has so far been limited (189, 415, 427*). It is uncertain how increasing the integration of general-purpose Al systems with narrow Al tools will affect the risk of malicious use. Today, narrow Al tools require specialist expertise to use effectively for scientific research (428). Though the ability to direct these tools using natural language will make specialised tasks more accessible, leveraging the outputs will likely still require technical expertise until substantial advances are made with general-purpose Al. Existing tools also have limited outputs and require extensive laboratory validation, and the extent to which these limitations will be overcome is unclear.

**Autonomous science capabilities** - While advanced general-purpose Al systems are already enabling some autonomous science capabilities, it is not clear what the near-term implications are for potential biological malicious use. While aspects of chemistry workflows such as chemical synthesis are already automatable (189, 415), experts are uncertain if these advances will transfer well to biology workflows, due to challenges automating work involving living systems (407). Furthermore, the high costs of automated laboratories are likely to make large-scale automation largely inaccessible to all but the most advanced malicious actors, although commercial cloud labs could partially offset this.

**In summary,** the degree to which current state-of-the-art general-purpose Al systems enhance the capabilities of malicious actors to use the life sciences over existing resources, such as the internet, remains unclear. Though some empirical work has assessed this uplift with respect to information access and biological threats (166*, 429), additional studies evaluating a broader range of tasks and scientific domains are needed to provide greater insight into this question. It has not yet been investigated whether general-purpose Al systems could also strengthen defences against dual-use science hazards.

## 4.2 Risks from malfunctions

### 4.2.1 Risks from product functionality issues

#### KEY INFORMATION

* Product functionality issues occur when there is confusion or misinformation about what a general-purpose Al model or system is capable of. This can lead to unrealistic expectations and overreliance on general-purpose Al systems, potentially causing harm if a system fails to deliver on expected capabilities.

* These functionality misconceptions may arise from technical difficulties in assessing an Al model's true capabilities on its own, or predicting its performance when part of a larger system. Misleading claims in advertising and communications can also contribute to these misconceptions.

Risks may arise where general-purpose Al models and systems fail to comply with general tenets of product safety and product functionality. As with many products, risks from general-purpose Al-based products occur because of misunderstandings of functionality and inadequate guidance for appropriate and safe use. In that respect, general-purpose Al-based products may be no different (430).

Product functionality issues, and the risks they pose may be clustered by potential failure modes (see Table 1).

*Impossible tasks* arise from instances of an attempt to accomplish goals with a general-purpose Al system that goes beyond the general-purpose Al system's capability. It can be hard to say definitively what constitutes an impossible task in a modern setting. Historically, large language models have not been able to consider events or developments that occurred after the end of their training. However,
enabling AI products to retrieve information from databases has improved their ability to consider what happened after their training – although models still perform worse on tests that require novel information (431). Another potentially impossible task may be tasks requiring data that is inherently inaccessible -- such as information that does not exist in the format of computable media, or data not available for training due to legal or security reasons.

Impossible tasks pose risks because often, salient types of failure-- including many of the engineering failures, post-deployment failures and communication failures (see Table 1) -- might be the by-product of mismeasurements, misapprehensions or miscommunication around what a model can do, and the misinformed deployments that result.

For instance, the GPT-4 model achieved results of "passing a simulated bar exam with a score around the top 10% of test takers" and being in the 88th percentile of LSAT test takers (2*). Confidence in this result even led some lawyers to adopt the technology for their professional use (432). Under different circumstances, such as changes to test-taking settings or when comparing to first-time bar examinees who passed the exam, the model achieved substantially lower percentile results (433). Those who were attempting to make use of the model in actual legal practice encountered these inadequacies, facing severe professional consequences for the errors produced by these models (i.e. inaccurate legal citations, inappropriate format and phrasing, etc.) (434). Similar misapprehensions regarding model performance are thought to apply in the medical context (435), where real world use and re-evaluations reveal complexity to the claims of these models containing reliable clinical knowledge (436) or passing medical tests such as the MCAT (2*) or USMLE (437). More generally, some deployed large language models struggle under some linguistic circumstances: They might, for instance, have trouble navigating negations and consequently fail to distinguish between advising for and against a course of action – though some research suggests these issues are addressed by general capability gains (438, 439).

Some shortcomings are only revealed after deployment. Although many thorough evaluations have examined large language model use for code generation (440*), including in relevant real-world tasks (441), instances of real-world deployment of large language models for coding suggest that the use of these models could lead to the potential introduction of critical overlooked bugs (442), as well as confusing or misleading edits (443) that could be especially impactful when guiding engineering programmers, particularly in applications that automate parts of the workflow (444).

<table><caption>Table 1: Taxonomy of AI functionality issues, reproduced with permission (430).</caption><thead><tr><td>Failure Modes</td><td>Categories</td></tr></thead><tbody><tr><td>Impossible Tasks</td><td>Conceptually Possible<br>Practically Impossible</td></tr><tr><td>Engineering Failures</td><td>Design Failures<br>Implementation Failures<br>Missing Safety Features</td></tr><tr><td>Post-Deployment Failures</td><td>Robustness Issues<br>Failure under Adversarial Attacks<br>Unanticipated Interactions</td></tr><tr><td>Communication Failures</td><td>Falsified or Overstated Capabilities<br>Misrepresented Capabilities</td></tr></tbody></table>

Functionality misapprehensions arise from different underlying issues. Firstly, as noted in [3]. Methodology to assess and understand general-purpose AI systems, there are technical difficulties in designing and representative evaluations of performance for general-purpose AI systems, making definite statements about functionality difficult. Secondly, functionality issues might only manifest or
enabling AI products to retrieve information from databases has improved their ability to consider what happened after their training – although models still perform worse on tests that require novel information (431). Another potentially impossible task may be tasks requiring data that is inherently inaccessible -- such as information that does not exist in the format of computable media, or data not available for training due to legal or security reasons.

Impossible tasks pose risks because often, salient types of failure-- including many of the engineering failures, post-deployment failures and communication failures (see Table 1) -- might be the by-product of mismeasurements, misapprehensions or miscommunication around what a model can do, and the misinformed deployments that result.

For instance, the GPT-4 model achieved results of "passing a simulated bar exam with a score around the top 10% of test takers" and being in the 88th percentile of LSAT test takers (2*). Confidence in this result even led some lawyers to adopt the technology for their professional use (432). Under different circumstances, such as changes to test-taking settings or when comparing to first-time bar examinees who passed the exam, the model achieved substantially lower percentile results (433). Those who were attempting to make use of the model in actual legal practice encountered these inadequacies, facing severe professional consequences for the errors produced by these models (i.e. inaccurate legal citations, inappropriate format and phrasing, etc.) (434). Similar misapprehensions regarding model performance are thought to apply in the medical context (435), where real world use and re-evaluations reveal complexity to the claims of these models containing reliable clinical knowledge (436) or passing medical tests such as the MCAT (2*) or USMLE (437). More generally, some deployed large language models struggle under some linguistic circumstances: They might, for instance, have trouble navigating negations and consequently fail to distinguish between advising for and against a course of action – though some research suggests these issues are addressed by general capability gains (438, 439).

Some shortcomings are only revealed after deployment. Although many thorough evaluations have examined large language model use for code generation (440*), including in relevant real-world tasks (441), instances of real-world deployment of large language models for coding suggest that the use of these models could lead to the potential introduction of critical overlooked bugs (442), as well as confusing or misleading edits (443) that could be especially impactful when guiding engineering programmers, particularly in applications that automate parts of the workflow (444).

<table><caption>Table 1: Taxonomy of AI functionality issues, reproduced with permission (430).</caption><thead><tr><td>Failure Modes</td><td>Categories</td></tr></thead><tbody><tr><td>Impossible Tasks</td><td>Conceptually Possible<br>Practically Impossible</td></tr><tr><td>Engineering Failures</td><td>Design Failures<br>Implementation Failures<br>Missing Safety Features</td></tr><tr><td>Post-Deployment Failures</td><td>Robustness Issues<br>Failure under Adversarial Attacks<br>Unanticipated Interactions</td></tr><tr><td>Communication Failures</td><td>Falsified or Overstated Capabilities<br>Misrepresented Capabilities</td></tr></tbody></table>

Functionality misapprehensions arise from different underlying issues. Firstly, as noted in [3]. Methodology to assess and understand general-purpose AI systems, there are technical difficulties in designing and representative evaluations of performance for general-purpose AI systems, making definite statements about functionality difficult. Secondly, functionality issues might only manifest or
manifest differently in a real-world realistic context, rendering even informative model evaluation insufficient for robust statements about general-purpose Al system and product functionality. Thirdly, failures can result not just from inadequate evaluation, but a lack of appropriate communication to product users around the product's limitations and the potential consequences. Misleading advertising, as it occurs in many markets, could become a substantial source of risks from functionality in general-purpose Al (445).

In general, for many machine learning based products, it can be unclear exactly which context of deployment is well represented in the data and suitable for the model. However, more general purpose Al tools specifically are more difficult to vet for deployment readiness than lower-capability or narrower Al systems: With General Purpose Al, it can be difficult to clearly define and restrict potential use cases that may not be suitable or may be premature, although substantial progress on restricting use cases is feasible.

## 4.2.2 Risks from bias and underrepresentation

### KEY INFORMATION

* The outputs and impacts of general-purpose Al systems can be biased with respect to various aspects of human identity, including race, gender, culture, age, and disability. This creates risks in high-stakes domains such as healthcare, job recruitment, and financial lending.

* General-purpose Al systems are primarily trained on language and image datasets that disproportionately represent English-speaking and Western cultures, increasing the potential for harm to individuals not represented well by this data.

Harmful bias and underrepresentation in Al systems have been challenges since well before the increased attention to general-purpose Al. They remain an issue with general-purpose Al, and will likely be a major challenge with general-purpose Al systems for the foreseeable future. Decisions by an Al might be biased if their decision-making is skewed based on protected characteristics, such as gender, race, etc. They might hence be discriminatory when this bias informs decisions to the disadvantage of members of these protected groups; thereby creating harm to fairness. This section discusses present and future risks resulting from bias and underrepresentation risks in Al. Because of the rich history of research in this space, this section explores research both on narrow Al and general-purpose Al.

Al systems can demonstrate bias as a result of skewed training data, choices made during model development, or the premature deployment of flawed systems. Despite extensive research, reliable methods to fully mitigate any discrimination remain elusive. There are particular concerns over the tendency of advanced general-purpose Al systems to replicate and amplify bias present within their training data (446). This poses a significant risk of discrimination in high-impact applications such as job recruitment, financial lending, and healthcare (447). In these areas biased decisions resulting from general-purpose Al systems outputs can have profoundly negative consequences for individuals, potentially limiting employment prospects (448, 449), hindering upward financial mobility, and restricting access to essential healthcare services (450, 451).

**There are several well-documented cases of Al systems displaying discriminatory behaviour based on race, gender, age, and disability status, causing substantial harm.** Given increasingly widespread adoption of Al systems across various sectors, such behaviour can perpetuate various types of bias, including race, gender, age, and disability. This can cause serious harm if these systems are entrusted with increasingly high-stakes decisions which can have severe consequences for individuals. Racial bias in Al systems has been shown to be present in commercially available facial recognition algorithms (452) and has caused ineffectiveness in predicting recidivism outcomes for
manifest differently in a real-world realistic context, rendering even informative model evaluation
insufficient for robust statements about general-purpose Al system and product functionality. Thirdly,
failures can result not just from inadequate evaluation, but a lack of appropriate communication to
product users around the product's limitations and the potential consequences. Misleading
advertising, as it occurs in many markets, could become a substantial source of risks from
functionality in general-purpose Al (445).

In general, for many machine learning based products, it can be unclear exactly which context of
deployment is well represented in the data and suitable for the model. However, more general purpose
Al tools specifically are more difficult to vet for deployment readiness than lower-capability or
narrower Al systems: With General Purpose Al, it can be difficult to clearly define and restrict potential
use cases that may not be suitable or may be premature, although substantial progress on restricting
use cases is feasible.

4.2.2 Risks from bias and underrepresentation

KEY INFORMATION

* The outputs and impacts of general-purpose Al systems can be biased with respect to various aspects of human identity, including race, gender, culture, age, and disability. This creates risks in high-stakes domains such as healthcare, job recruitment, and financial lending.

* General-purpose Al systems are primarily trained on language and image datasets that disproportionately represent English-speaking and Western cultures, increasing the potential for harm to individuals not represented well by this data.

Harmful bias and underrepresentation in Al systems have been challenges since well before the increased attention to general-purpose Al. They remain an issue with general-purpose Al, and will likely be a major challenge with general-purpose Al systems for the foreseeable future. Decisions by an Al might be biased if their decision-making is skewed based on protected characteristics, such as gender, race, etc. They might hence be discriminatory when this bias informs decisions to the disadvantage of members of these protected groups; thereby creating harm to fairness. This section discusses present and future risks resulting from bias and underrepresentation risks in Al. Because of the rich history of research in this space, this section explores research both on narrow Al and general-purpose Al.

Al systems can demonstrate bias as a result of skewed training data, choices made during model
development, or the premature deployment of flawed systems. Despite extensive research, reliable
methods to fully mitigate any discrimination remain elusive. There are particular concerns over the
tendency of advanced general-purpose Al systems to replicate and amplify bias present within their
training data (446). This poses a significant risk of discrimination in high-impact applications such as
job recruitment, financial lending, and healthcare (447). In these areas biased decisions resulting from
general-purpose Al systems outputs can have profoundly negative consequences for individuals,
potentially limiting employment prospects (448, 449), hindering upward financial mobility, and
restricting access to essential healthcare services (450, 451).

There are several well-documented cases of Al systems displaying discriminatory behaviour
based on race, gender, age, and disability status, causing substantial harm. Given increasingly
widespread adoption of Al systems across various sectors, such behaviour can perpetuate various
types of bias, including race, gender, age, and disability. This can cause serious harm if these systems
are entrusted with increasingly high-stakes decisions which can have severe consequences for
individuals. Racial bias in Al systems has been shown to be present in commercially available facial
recognition algorithms (452) and has caused ineffectiveness in predicting recidivism outcomes for
defendants of colour, underestimation of the needs of patients from marginalised racial and ethnic backgrounds (453, 454), and the perpetuation of inappropriate race-based medicine in responses from text-generation models (435, 450).

Gender biases in the outputs of AI systems are another key concern. Research has uncovered sexist, misogynistic, and gender-stereotyping content being produced from general-purpose AI (455, 456) and male-dominated results from gender-neutral internet searches using narrow AI algorithms (457). Age bias is also a key issue: some AI systems have exhibited bias against older job seekers (458), and age bias appears in some outputs from sentiment analysis models (459). One reason for this could be biases in training data. For example, LLM-powered HR screening tools may be trained on resumes skewed towards younger workers that may inadvertently discount the experiences and skill sets of older applicants. Similarly, healthcare allocation algorithms developed by health insurance companies may disadvantage older individuals based on age-related health risks, even if these individuals are healthy. Lending algorithms may not appropriately handle older adults' financial circumstances, particularly regarding social security income, which could impact approval outcomes (460).

Research has also shown that AI systems and tools can discriminate against users with disabilities, for example by disproportionately denying coverage claims from disabled individuals with complex medical needs (461), reproducing societal stereotypes about disabilities (462), and inaccurately classifying sentiments about people with disabilities (463). Despite growing research on sign language recognition (464), AI systems have limited automated transcription abilities for sign language speakers (143), and limited diversity in sign language datasets may also exacerbate disability bias from advanced general-purpose AI systems, as the majority of sign language datasets represent American Sign Language. Recent work that, for instance, has developed datasets for six African sign languages (465) is a step, albeit a modest one, toward achieving more equitable inclusion of sign language dialects.

AI systems exhibit a tendency towards intersectional biases. Intersectionality describes how individuals who share more than one marginalised characteristic may experience compounded bias or discrimination (for example, a low-income woman of colour). Intersectional bias exacerbates existing social inequalities and can limit individuals' access to vital resources and opportunities. While there is an emerging area of research focused on developing methods for detecting intersectional bias within AI models (466, 467, 468), there has been much less progress on mitigating potential impacts (469).

**Bias in general-purpose AI system outputs may result from a lack of representation in training datasets, leading to biased outputs.** Different groups and different cultures are unequally represented at various stages of the AI lifecycle – from the input data to the outputs of language and image generation models. Many of the issues associated with bias in Al demonstrate the harms of limited representation in training datasets, which are overwhelmingly likely to be in English (236). This has led to major disparities in the safety and reliability of modern general-purpose AI systems in different societies (309, 310, 313*). Datasets used to train Al models have also been shown to under-represent various demographic markers such as age, race, gender, and disability status (470, 471). Al language models predominantly rely on digitised books and online text in their training, which fails to reflect oral traditions and non-digitised cultures. The inherent historical biases embedded in these sources, coupled with potential inequalities in the data collection process, can perpetuate systemic injustices (472*) and lead AI systems to reflect dominant cultures, languages, and worldviews, at the detriment of marginalised groups such as indigenous communities (196, 230, 239, 473, 474).

**Issues of bias and representation remain an unsolved problem.** Despite considerable attention in the literature and economic incentives for companies to avoid reputational damage from biased AI products, mitigating bias remains an unsolved challenge. While developers may attempt to explicitly address bias during model fine-tuning, AI models can still pick up on implicit associations between features (475) or perpetuate significant biases and stereotypes even when prompts do not contain explicit demographic identifiers (476). While methods such as Reinforcement Learning from Human Feedback (RLHF) aim to align model decision-making with human preferences, these methods could inadvertently introduce biases based on the diversity and representativeness of the humans providing feedback (303). RLHF has been shown to lead to more politically biased models (239, 477) and
defendants of colour, underestimation of the needs of patients from marginalised racial and ethnic backgrounds (453, 454), and the perpetuation of inappropriate race-based medicine in responses from text-generation models (435, 450).

Gender biases in the outputs of AI systems are another key concern. Research has uncovered sexist, misogynistic, and gender-stereotyping content being produced from general-purpose AI (455, 456) and male-dominated results from gender-neutral internet searches using narrow AI algorithms (457). Age bias is also a key issue: some AI systems have exhibited bias against older job seekers (458), and age bias appears in some outputs from sentiment analysis models (459). One reason for this could be biases in training data. For example, LLM-powered HR screening tools may be trained on resumes skewed towards younger workers that may inadvertently discount the experiences and skill sets of older applicants. Similarly, healthcare allocation algorithms developed by health insurance companies may disadvantage older individuals based on age-related health risks, even if these individuals are healthy. Lending algorithms may not appropriately handle older adults' financial circumstances, particularly regarding social security income, which could impact approval outcomes (460).

Research has also shown that AI systems and tools can discriminate against users with disabilities, for example by disproportionately denying coverage claims from disabled individuals with complex medical needs (461), reproducing societal stereotypes about disabilities (462), and inaccurately classifying sentiments about people with disabilities (463). Despite growing research on sign language recognition (464), AI systems have limited automated transcription abilities for sign language speakers (143), and limited diversity in sign language datasets may also exacerbate disability bias from advanced general-purpose AI systems, as the majority of sign language datasets represent American Sign Language. Recent work that, for instance, has developed datasets for six African sign languages (465) is a step, albeit a modest one, toward achieving more equitable inclusion of sign language dialects.

AI systems exhibit a tendency towards intersectional biases. Intersectionality describes how individuals who share more than one marginalised characteristic may experience compounded bias or discrimination (for example, a low-income woman of colour). Intersectional bias exacerbates existing social inequalities and can limit individuals' access to vital resources and opportunities. While there is an emerging area of research focused on developing methods for detecting intersectional bias within AI models (466, 467, 468), there has been much less progress on mitigating potential impacts (469).

**Bias in general-purpose AI system outputs may result from a lack of representation in training datasets, leading to biased outputs.** Different groups and different cultures are unequally represented at various stages of the AI lifecycle – from the input data to the outputs of language and image generation models. Many of the issues associated with bias in Al demonstrate the harms of limited representation in training datasets, which are overwhelmingly likely to be in English (236). This has led to major disparities in the safety and reliability of modern general-purpose AI systems in different societies (309, 310, 313*). Datasets used to train Al models have also been shown to under-represent various demographic markers such as age, race, gender, and disability status (470, 471). Al language models predominantly rely on digitised books and online text in their training, which fails to reflect oral traditions and non-digitised cultures. The inherent historical biases embedded in these sources, coupled with potential inequalities in the data collection process, can perpetuate systemic injustices (472*) and lead AI systems to reflect dominant cultures, languages, and worldviews, at the detriment of marginalised groups such as indigenous communities (196, 230, 239, 473, 474).

**Issues of bias and representation remain an unsolved problem.** Despite considerable attention in the literature and economic incentives for companies to avoid reputational damage from biased AI products, mitigating bias remains an unsolved challenge. While developers may attempt to explicitly address bias during model fine-tuning, AI models can still pick up on implicit associations between features (475) or perpetuate significant biases and stereotypes even when prompts do not contain explicit demographic identifiers (476). While methods such as Reinforcement Learning from Human Feedback (RLHF) aim to align model decision-making with human preferences, these methods could inadvertently introduce biases based on the diversity and representativeness of the humans providing feedback (303). RLHF has been shown to lead to more politically biased models (239, 477) and
incorporate user beliefs over factual information (238). In addition, rater feedback is often inconsistent (478, 479). Understanding issues of representation within datasets, models, and evaluation methods such as RLHF is crucial, as skewed representation can lead to biased outputs in AI models. More research is needed to address these issues.

### 4.2.3 Loss of control

#### KEY INFORMATION

* Ongoing AI research is seeking to develop more capable 'general-purpose AI agents', that is, general-purpose AI systems that can autonomously interact with the world, plan ahead, and pursue goals.

* 'Loss of control' scenarios are potential future scenarios in which society can no longer meaningfully constrain some advanced general-purpose AI agents, even if it becomes clear they are causing harm. These scenarios are hypothesised to arise through a combination of social and technical factors, such as pressures to delegate decisions to general-purpose AI systems, and limitations of existing techniques used to influence the behaviours of general-purpose AI systems.

* There is broad agreement among AI experts that currently known general-purpose AI systems pose no significant loss of control risk, due to their limited capabilities.

* Some experts believe that loss of control scenarios are implausible, while others believe they are likely, and some consider them as low-likelihood risks that deserve consideration due to their high severity.

* This expert disagreement is difficult to resolve, since there is not yet an agreed-upon methodology for assessing the likelihood of loss of control, or when the relevant AI capabilities might be developed.

* If the risk of loss of control will in fact be large, then resolving this risk could require making fundamental progress on certain technical problems in AI safety. It is unclear if this progress would require many years of preparatory work.

AI companies and researchers are increasingly interested in developing general-purpose AI 'agents' (sometimes also referred to as 'autonomous general-purpose AI systems'). General-purpose AI agents are systems that can autonomously interact with the world, plan ahead, and pursue goals. Although general-purpose AI agents are beginning to be developed, they still demonstrate only very limited capabilities (26, 178, 480*). Various researchers and AI labs ultimately hope to create general-purpose AI agents that can operate and accomplish long-term tasks with little or no human oversight or intervention.

Autonomous general-purpose AI systems, if fully realised, could be useful in many sectors. However, some researchers worry about risks from their malicious use, or from accidents and unintended consequences of their deployment (481*, 482).

Some researchers have also expressed concern about society's ability to exercise reliable oversight and control over autonomous general-purpose AI systems. For decades, concerns about a potential loss of control have been raised by computer scientists looking ahead toward these kinds of AI systems, including AI pioneers such as Alan Turing (483), I. J. Good (484), and Norbert Wiener (485). These concerns have gained more prominence recently (486), partly because a subset of researchers now believe that sufficiently advanced general-purpose AI agents could be developed sooner than previously thought (127, 487, 488).
incorporate user beliefs over factual information (238). In addition, rater feedback is often inconsistent (478, 479). Understanding issues of representation within datasets, models, and evaluation methods such as RLHF is crucial, as skewed representation can lead to biased outputs in AI models. More research is needed to address these issues.

### 4.2.3 Loss of control

#### KEY INFORMATION

* Ongoing AI research is seeking to develop more capable 'general-purpose AI agents', that is, general-purpose AI systems that can autonomously interact with the world, plan ahead, and pursue goals.

* 'Loss of control' scenarios are potential future scenarios in which society can no longer meaningfully constrain some advanced general-purpose AI agents, even if it becomes clear they are causing harm. These scenarios are hypothesised to arise through a combination of social and technical factors, such as pressures to delegate decisions to general-purpose AI systems, and limitations of existing techniques used to influence the behaviours of general-purpose AI systems.

* There is broad agreement among AI experts that currently known general-purpose AI systems pose no significant loss of control risk, due to their limited capabilities.

* Some experts believe that loss of control scenarios are implausible, while others believe they are likely, and some consider them as low-likelihood risks that deserve consideration due to their high severity.

* This expert disagreement is difficult to resolve, since there is not yet an agreed-upon methodology for assessing the likelihood of loss of control, or when the relevant AI capabilities might be developed.

* If the risk of loss of control will in fact be large, then resolving this risk could require making fundamental progress on certain technical problems in AI safety. It is unclear if this progress would require many years of preparatory work.

AI companies and researchers are increasingly interested in developing general-purpose AI 'agents' (sometimes also referred to as 'autonomous general-purpose AI systems'). General-purpose AI agents are systems that can autonomously interact with the world, plan ahead, and pursue goals. Although general-purpose AI agents are beginning to be developed, they still demonstrate only very limited capabilities (26, 178, 480*). Various researchers and AI labs ultimately hope to create general-purpose AI agents that can operate and accomplish long-term tasks with little or no human oversight or intervention.

Autonomous general-purpose AI systems, if fully realised, could be useful in many sectors. However, some researchers worry about risks from their malicious use, or from accidents and unintended consequences of their deployment (481*, 482).

Some researchers have also expressed concern about society's ability to exercise reliable oversight and control over autonomous general-purpose AI systems. For decades, concerns about a potential loss of control have been raised by computer scientists looking ahead toward these kinds of AI systems, including AI pioneers such as Alan Turing (483), I. J. Good (484), and Norbert Wiener (485). These concerns have gained more prominence recently (486), partly because a subset of researchers now believe that sufficiently advanced general-purpose AI agents could be developed sooner than previously thought (127, 487, 488).
The plausibility of such risks remains highly contentious. This section seeks to clarify the nature of the proposed risk, outline the main arguments and evidence that currently inform researchers' views about its likelihood, and summarise current expert opinion.

## Risk factors

An Al system is considered 'controllable' when its behaviours can be meaningfully determined or constrained by humans. While a lack of control is not intrinsically harmful, it significantly increases the risks of various harms. Current general-purpose Al systems are generally considered to be controllable, but, if autonomous general-purpose Al systems are fully developed, then risk of the loss of control may grow considerably.

In 4.4. Cross-cutting risk factors, this report discusses technical and societal risk factors for dangerous Al systems. In this section, we expand on how these factors may affect loss of control risk. Overall, the risk of losing control over currently known general-purpose Al systems appears negligible. The degree to which future, potentially much more capable, autonomous general-purpose Al agents can be controlled remains unclear.

**It is not yet known whether it will be easier or harder in future to ensure that highly capable Al systems pursue the objectives their developers intend.** One reason for this is that Al systems can 'game' their objectives by achieving them in unintended and potentially harmful ways. For example, widely deployed general-purpose Al language models adjust their stated view to better match their user's view, regardless of truth, and consequently gain more positive feedback (237, 238). There are observations that objective-gaming can become more prevalent as system capability increases, because more capable systems can find more ways to achieve a given objective (489, 490). Furthermore, more capable systems have more ways to coherently 'generalise' to behave differently in situations that differ from their training setting including potentially harmful ways – although this is only a hypothesised concern to date (489, 490). However, some recent more capable general-purpose Al systems have been more controllable due to better tools for training and oversight (19, 20, 491), and are less likely to generalise incoherently beyond their training data (492). It is therefore unclear if general-purpose Al systems will become more or less controllable as more advanced systems are created. For a discussion of how controllable present general-purpose Al systems are, see 5.2 Training more trustworthy models and 4.4.1 Cross-cutting technical risk factors.

**Some mathematical findings suggest that future general-purpose Al agents may use strategies that hinder human control, but as yet it is unclear how well these findings will apply to real-world general-purpose Al systems.** Some mathematical models of idealised goal-directed Al agents have found that, with sufficiently advanced planning capabilities, many such Al agents would hinder human attempts to interfere with their goal pursuit (493, 494, 495*, 496). Similar mathematical findings suggest that many such Al agents could have a tendency to 'seek power' by accumulating resources, interfering with oversight processes, and avoiding being deactivated, because these actions help them achieve their given goals (493, 494, 495*, 497*, 498, 499).

However, drawing real-world implications from this mathematical research is not straightforward. For instance, most research assumes that an Al system is trained using a randomly chosen goal (493, 494, 495*, 497*, 498, 499), but in practice, Al developers can substantially influence which goals are potentially encoded in general-purpose Al models (see below and 5.2 Training more trustworthy models). Further, philosophical research suggests that not all of the above behaviours are implied by pursuing a random goal (500). Currently, one mathematical finding and early-stage empirical findings support, although weakly as of this writing, that such behaviours may be found under more realistic circumstances (181, 237, 490, 497*). There have also been case studies where, without being prompted to do so, general-purpose Al systems and other Al systems learned to systematically induce false beliefs in others because this was useful for achieving a goal (366, 501). If observed more widely, such behaviour is also relevant to near-term applications of general-purpose Al chatbots and agents.

**If people entrust general-purpose Al systems with increasingly critical responsibilities, then this could increase the risk of loss of control.** A range of social and economic forces would influence the
The plausibility of such risks remains highly contentious. This section seeks to clarify the nature of the proposed risk, outline the main arguments and evidence that currently inform researchers' views about its likelihood, and summarise current expert opinion.

## Risk factors

An AI system is considered 'controllable' when its behaviours can be meaningfully determined or constrained by humans. While a lack of control is not intrinsically harmful, it significantly increases the risks of various harms. Current general-purpose AI systems are generally considered to be controllable, but, if autonomous general-purpose AI systems are fully developed, then risk of the loss of control may grow considerably.

In 4.4. Cross-cutting risk factors, this report discusses technical and societal risk factors for dangerous AI systems. In this section, we expand on how these factors may affect loss of control risk. Overall, the risk of losing control over currently known general-purpose AI systems appears negligible. The degree to which future, potentially much more capable, autonomous general-purpose AI agents can be controlled remains unclear.

**It is not yet known whether it will be easier or harder in future to ensure that highly capable AI systems pursue the objectives their developers intend.** One reason for this is that AI systems can 'game' their objectives by achieving them in unintended and potentially harmful ways. For example, widely deployed general-purpose AI language models adjust their stated view to better match their user's view, regardless of truth, and consequently gain more positive feedback (237, 238). There are observations that objective-gaming can become more prevalent as system capability increases, because more capable systems can find more ways to achieve a given objective (489, 490).

Furthermore, more capable systems have more ways to coherently 'generalise' to behave differently in situations that differ from their training setting including potentially harmful ways – although this is only a hypothesised concern to date (489, 490). However, some recent more capable general-purpose AI systems have been more controllable due to better tools for training and oversight (19, 20, 491), and are less likely to generalise incoherently beyond their training data (492). It is therefore unclear if general-purpose AI systems will become more or less controllable as more advanced systems are created. For a discussion of how controllable present general-purpose AI systems are, see 5.2 Training more trustworthy models and 4.4.1 Cross-cutting technical risk factors.

**Some mathematical findings suggest that future general-purpose AI agents may use strategies that hinder human control, but as yet it is unclear how well these findings will apply to real-world general-purpose AI systems.** Some mathematical models of idealised goal-directed AI agents have found that, with sufficiently advanced planning capabilities, many such AI agents would hinder human attempts to interfere with their goal pursuit (493, 494, 495*, 496). Similar mathematical findings suggest that many such AI agents could have a tendency to 'seek power' by accumulating resources, interfering with oversight processes, and avoiding being deactivated, because these actions help them achieve their given goals (493, 494, 495*, 497*, 498, 499).

However, drawing real-world implications from this mathematical research is not straightforward. For instance, most research assumes that an AI system is trained using a randomly chosen goal (493, 494, 495*, 497*, 498, 499), but in practice, AI developers can substantially influence which goals are potentially encoded in general-purpose AI models (see below and 5.2 Training more trustworthy models). Further, philosophical research suggests that not all of the above behaviours are implied by pursuing a random goal (500). Currently, one mathematical finding and early-stage empirical findings support, although weakly as of this writing, that such behaviours may be found under more realistic circumstances (181, 237, 490, 497*). There have also been case studies where, without being prompted to do so, general-purpose AI systems and other AI systems learned to systematically induce false beliefs in others because this was useful for achieving a goal (366, 501). If observed more widely, such behaviour is also relevant to near-term applications of general-purpose AI chatbots and agents.

**If people entrust general-purpose AI systems with increasingly critical responsibilities, then this could increase the risk of loss of control.** A range of social and economic forces would influence the
interaction between human and autonomous agents in such scenarios. For example, economic pressures may favour general-purpose AI-enabled automation in the absence of intervention, despite potentially negative consequences (502), and human over-reliance on general-purpose AI agents would make it harder to exercise oversight (481*). Using general-purpose AI agents to automate decision-making in government, military, or judicial applications might elevate concerns over AI's influence on important societal decisions (503, 504, 505, 506). As a more extreme case, some actors have stated an interest in purposefully developing uncontrolled AI agents (507).

**Certain specific capabilities could disproportionately increase the risk of loss of control.** These capabilities – which are currently limited – include identifying and exploiting software vulnerabilities, persuasion, automating AI research and development, and capabilities needed to autonomously replicate and adapt (367*, 508*, 509). The relevant sections in this report discuss how capable current general-purpose AI systems are in some of these areas (4.1.3 Cyber offence, 4.1.2 Disinformation and manipulation of public opinion, 4.1.4 Dual use science risks). Particularly relevant are agent capabilities, which increase the ability for general-purpose AI systems to operate autonomously, such as planning and using memory. These are discussed in 4.4.1 Cross-cutting technical risk factors.

## Consequences of loss of control

An irreversible loss of control of some general-purpose AI systems is not necessarily catastrophic. As an analogy, computer viruses have long been able to proliferate near-irreversibly and in large numbers (510) without causing the internet to collapse. Some researchers have explored hypothetical scenarios where highly advanced future general-purpose AI agents acting autonomously might cause catastrophic harm to humans, especially if humans present obstacles to achieving the agents' given goals (127, 511). The mechanisms of harm are often imagined to stem from capabilities such as the ones listed in the previous paragraph and in 4.1.4 Dual use science risks. However, these scenarios remain hypothetical as they are not exhibited by current general-purpose AI systems.

## AI researchers have differing views on loss of control risks

AI experts debate the likelihood of losing control over future general-purpose AI systems. Key questions include whether sufficiently capable general-purpose AI agents will be developed in the medium-term, and whether technical safety and governance solutions can be developed in time to keep them adequately controlled. As there is limited research that assesses the risk of loss of control, expert opinion can provide alternative guidance, but it cannot replace research.

**A subset of researchers suggest that risks of loss of control deserves consideration. However, the overall likelihood of extreme control failures remains highly contentious.** Some researchers have argued that there has been little progress in developing the types of AI systems that others fear could pose risks of loss of control (129, 512). Nevertheless, broad hypothetical scenarios for how society might lose control over AI systems have been proposed (507, 511) and these scenarios have been emphasised by some leading researchers (127). For example, several hundred AI researchers have signed a statement declaring that “Mitigating the risk of extinction from AI should be a global priority” (486), though without explicitly referring to loss of control. The actual risk remains highly contentious as there is only limited research assessing it, and the opinions of scientists cannot replace this research.
interaction between human and autonomous agents in such scenarios. For example, economic pressures may favour general-purpose AI-enabled automation in the absence of intervention, despite potentially negative consequences (502), and human over-reliance on general-purpose AI agents would make it harder to exercise oversight (481*). Using general-purpose AI agents to automate decision-making in government, military, or judicial applications might elevate concerns over AI's influence on important societal decisions (503, 504, 505, 506). As a more extreme case, some actors have stated an interest in purposefully developing uncontrolled AI agents (507).

**Certain specific capabilities could disproportionately increase the risk of loss of control.** These capabilities – which are currently limited – include identifying and exploiting software vulnerabilities, persuasion, automating AI research and development, and capabilities needed to autonomously replicate and adapt (367*, 508*, 509). The relevant sections in this report discuss how capable current general-purpose AI systems are in some of these areas (4.1.3 Cyber offence, 4.1.2 Disinformation and manipulation of public opinion, 4.1.4 Dual use science risks). Particularly relevant are agent capabilities, which increase the ability for general-purpose AI systems to operate autonomously, such as planning and using memory. These are discussed in 4.4.1 Cross-cutting technical risk factors.

## Consequences of loss of control

An irreversible loss of control of some general-purpose AI systems is not necessarily catastrophic. As an analogy, computer viruses have long been able to proliferate near-irreversibly and in large numbers (510) without causing the internet to collapse. Some researchers have explored hypothetical scenarios where highly advanced future general-purpose AI agents acting autonomously might cause catastrophic harm to humans, especially if humans present obstacles to achieving the agents' given goals (127, 511). The mechanisms of harm are often imagined to stem from capabilities such as the ones listed in the previous paragraph and in 4.1.4 Dual use science risks. However, these scenarios remain hypothetical as they are not exhibited by current general-purpose AI systems.

## AI researchers have differing views on loss of control risks

AI experts debate the likelihood of losing control over future general-purpose AI systems. Key questions include whether sufficiently capable general-purpose AI agents will be developed in the medium-term, and whether technical safety and governance solutions can be developed in time to keep them adequately controlled. As there is limited research that assesses the risk of loss of control, expert opinion can provide alternative guidance, but it cannot replace research.

**A subset of researchers suggest that risks of loss of control deserves consideration. However, the overall likelihood of extreme control failures remains highly contentious.** Some researchers have argued that there has been little progress in developing the types of AI systems that others fear could pose risks of loss of control (129, 512). Nevertheless, broad hypothetical scenarios for how society might lose control over AI systems have been proposed (507, 511) and these scenarios have been emphasised by some leading researchers (127). For example, several hundred AI researchers have signed a statement declaring that "Mitigating the risk of extinction from AI should be a global priority" (486), though without explicitly referring to loss of control. The actual risk remains highly contentious as there is only limited research assessing it, and the opinions of scientists cannot replace this research.
## 4.3 Systemic risks

### 4.3.1 Labour market risks

#### KEY INFORMATION

* Unlike previous waves of automation, general-purpose AI has the potential to automate a very broad range of tasks, which could have a significant effect on the labour market.

* This could mean many people could lose their current jobs. However, many economists expect that potential job losses as a result of automation could be offset, partly or completely, by the creation of new jobs and by increased demand in non-automated sectors.

* Labour market frictions, such as the time needed for workers to learn new skills or relocate for new jobs, could cause unemployment in the short run even if overall labour demand remained unchanged.

* The expected impact of general-purpose AI on wages is ambiguous. It is likely to simultaneously increase wages in some sectors by augmenting productivity and creating new opportunities, and decrease wages in other sectors where automation reduces labour demand faster than new tasks are created.

#### AI is likely to transform a range of jobs and could displace workers

Economists expect general-purpose AI to impact the workforce by automating tasks, augmenting worker productivity and earnings, changing the skills needed for various occupations, and displacing workers from certain occupations (513, 514, 515). Economists hold a wide range of views about the magnitude and timing of these effects, with some expecting widespread economic transformation in the next ten years, while others do not think a step-change in AI-related automation and productivity growth is imminent (516). The uncertainty about future general-purpose AI progress contributes to this uncertainty about the labour market effects of general-purpose AI.

Previous waves of automation by computing have primarily affected 'routine' tasks which could be easily codified and programmed into computers (517). General-purpose AI, in contrast, has the potential to perform a wide array of tasks that are typically carried out by humans, including complex problem-solving and decision-making. An extensive body of literature has studied the exposure of jobs to automation and AI generally, without a particular focus on general-purpose AI (518). In comparison to this research, the exploration of likely labour market impacts of general-purpose AI is at a very early stage. In advanced economies, it is estimated that, due to the prevalence of cognitive-task-oriented jobs, 60% of current jobs could be affected by the introduction of general-purpose AI systems such as today's LLMs (519). This means that general-purpose AI has the potential to either automate substantial portions of the work or to complement and significantly alter how the work is done. In emerging economies, the share of jobs thought to be potentially affected by general-purpose AI systems is lower, but still substantial, at 40% (519).

Recent empirical studies have begun to demonstrate the impact of current-generation general-purpose AI systems on various industries, notably in knowledge work:

* General-purpose AI systems have been shown to increase productivity, quality, and speed in strategy consulting (520), improve performance in customer support (250) and improve computer programming (521*).

* Machine translation and language models have been shown to substitute for human workers in tasks like simple large-scale language translation (522) and some writing/coding-related occupations (523, 524), leading to reduced demand for their services.
## 4.3 Systemic risks

### 4.3.1 Labour market risks

#### KEY INFORMATION

* Unlike previous waves of automation, general-purpose AI has the potential to automate a very broad range of tasks, which could have a significant effect on the labour market.

* This could mean many people could lose their current jobs. However, many economists expect that potential job losses as a result of automation could be offset, partly or completely, by the creation of new jobs and by increased demand in non-automated sectors.

* Labour market frictions, such as the time needed for workers to learn new skills or relocate for new jobs, could cause unemployment in the short run even if overall labour demand remained unchanged.

* The expected impact of general-purpose AI on wages is ambiguous. It is likely to simultaneously increase wages in some sectors by augmenting productivity and creating new opportunities, and decrease wages in other sectors where automation reduces labour demand faster than new tasks are created.

#### AI is likely to transform a range of jobs and could displace workers

Economists expect general-purpose AI to impact the workforce by automating tasks, augmenting worker productivity and earnings, changing the skills needed for various occupations, and displacing workers from certain occupations (513, 514, 515). Economists hold a wide range of views about the magnitude and timing of these effects, with some expecting widespread economic transformation in the next ten years, while others do not think a step-change in AI-related automation and productivity growth is imminent (516). The uncertainty about future general-purpose AI progress contributes to this uncertainty about the labour market effects of general-purpose AI.

Previous waves of automation by computing have primarily affected 'routine' tasks which could be easily codified and programmed into computers (517). General-purpose AI, in contrast, has the potential to perform a wide array of tasks that are typically carried out by humans, including complex problem-solving and decision-making. An extensive body of literature has studied the exposure of jobs to automation and AI generally, without a particular focus on general-purpose AI (518). In comparison to this research, the exploration of likely labour market impacts of general-purpose AI is at a very early stage. In advanced economies, it is estimated that, due to the prevalence of cognitive-task-oriented jobs, 60% of current jobs could be affected by the introduction of general-purpose AI systems such as today's LLMs (519). This means that general-purpose AI has the potential to either automate substantial portions of the work or to complement and significantly alter how the work is done. In emerging economies, the share of jobs thought to be potentially affected by general-purpose AI systems is lower, but still substantial, at 40% (519).

Recent empirical studies have begun to demonstrate the impact of current-generation general-purpose AI systems on various industries, notably in knowledge work:

* General-purpose AI systems have been shown to increase productivity, quality, and speed in strategy consulting (520), improve performance in customer support (250) and improve computer programming (521*).

* Machine translation and language models have been shown to substitute for human workers in tasks like simple large-scale language translation (522) and some writing/coding-related occupations (523, 524), leading to reduced demand for their services.
* The introduction and mass adoption of ChatGPT has resulted in a decrease in employment and earnings for freelancers offering writing, editing, and proofreading services (523).

A key question is whether there will be significant job losses as general-purpose AI systems become more advanced and more widespread. Some economists consider it likely that job losses will be offset by increased demand for labour in existing occupations and the creation of new kinds of jobs (516, 517, 525). This would be in line with the impact of previous waves of automation: For instance, more than 60% of employment in 2018 was in jobs with titles that did not exist in 1940 (526). Other economists emphasise that the future effects of general-purpose AI systems on job markets is very difficult to predict (519, 527) general-purpose AI could lead to a substantial reduction of the value of human labour compared with capital (528).

Even if overall demand for labour in the economy is not reduced, the process of displacement can create unemployment if the labour market fails to match workers with new employment opportunities quickly enough. This can happen because of various labour market frictions (529), such as:

* When workers have to learn new skills to switch occupations, it takes time for them to complete the necessary training or education

* The imperfect mobility of labour, which limits workers' ability to relocate for new job opportunities

* Skill mismatches between the requirements of new jobs and the existing skills of displaced workers

These factors influence how quickly and whether workers displaced by automation can move to new roles. Some workers may therefore be temporarily unemployed even though there may be job vacancies in the economy as a whole. General-purpose AI could lead to concentrated job losses, while productivity gains are likely to be more spread out throughout the economy, potentially leading to a difficult transition for some workers unless support is made available.

Some economists consider it plausible or even likely that the rate of job displacement due to general-purpose AI systems enabling automation could outpace the creation of new job opportunities, especially if the development of general-purpose AI systems is focused on substituting human labour rather than augmenting it (502, 530). However, there are few precise quantitative predictions, and the existing research is consistent with a wide range of possible outcomes.

AI researchers disagree on the pace of future general-purpose AI advancements, but there is some support for the possibility of extremely rapid advancements, including the possibility of general-purpose AI systems matching or surpassing the abilities of human experts in almost any cognitive task. (see 2.4.3. Will algorithmic progress lead to rapid advancements?). This latter scenario has only been considered in little economic research. That research assumes that AI systems can perform virtually all knowledge tasks more cost effectively than humans and indicates that such a scenario could lead to a collapse in wages across many sectors, severe unemployment and a dramatic reduction in labour force participation (531, 532). However, even in such a scenario some demand for human labour may persist due to consumer preferences and ethical or control-related reasons (531, 532). Among economists, extreme scenarios involving a dramatic decline in labour demand due to mass general-purpose AI automation are currently considered relatively fringe.

It is very difficult to forecast how and when general-purpose AI systems might affect various labour markets. Firstly, there is considerable uncertainty about how quickly general-purpose AI technology will advance and what its future capabilities may be. Secondly, even for a given level of technological sophistication, the extent to which general-purpose AI systems enable automation and how this kind of automation could affect labour markets is unclear. One reason for this is that in many sectors there can be considerable lags in deploying and adopting general-purpose AI systems at scale. For instance, if employees lack the necessary skills to effectively employ general-purpose AI assistance, this can slow down adoption. Overall, many economists expect modest macroeconomic impacts of general-purpose AI systems over the next decade (533), while some economists expect substantial labour market and macroeconomic effects in the next five to ten years (534). These forecasts often indicate high uncertainty due to the unknown pace of future advancements in general-purpose AI capabilities (see 2.4 Capability progress in coming years).
* The introduction and mass adoption of ChatGPT has resulted in a decrease in employment and earnings for freelancers offering writing, editing, and proofreading services (523).

A key question is whether there will be significant job losses as general-purpose Al systems become more advanced and more widespread. Some economists consider it likely that job losses will be offset by increased demand for labour in existing occupations and the creation of new kinds of jobs (516, 517, 525). This would be in line with the impact of previous waves of automation: For instance, more than 60% of employment in 2018 was in jobs with titles that did not exist in 1940 (526). Other economists emphasise that the future effects of general-purpose Al systems on job markets is very difficult to predict (519, 527) general-purpose Al could lead to a substantial reduction of the value of human labour compared with capital (528).

Even if overall demand for labour in the economy is not reduced, the process of displacement can create unemployment if the labour market fails to match workers with new employment opportunities quickly enough. This can happen because of various labour market frictions (529), such as:

* When workers have to learn new skills to switch occupations, it takes time for them to complete the necessary training or education

* The imperfect mobility of labour, which limits workers' ability to relocate for new job opportunities

* Skill mismatches between the requirements of new jobs and the existing skills of displaced workers

These factors influence how quickly and whether workers displaced by automation can move to new roles. Some workers may therefore be temporarily unemployed even though there may be job vacancies in the economy as a whole. General-purpose Al could lead to concentrated job losses, while productivity gains are likely to be more spread out throughout the economy, potentially leading to a difficult transition for some workers unless support is made available.

Some economists consider it plausible or even likely that the rate of job displacement due to general-purpose Al systems enabling automation could outpace the creation of new job opportunities, especially if the development of general-purpose Al systems is focused on substituting human labour rather than augmenting it (502, 530). However, there are few precise quantitative predictions, and the existing research is consistent with a wide range of possible outcomes.

Al researchers disagree on the pace of future general-purpose Al advancements, but there is some support for the possibility of extremely rapid advancements, including the possibility of general-purpose Al systems matching or surpassing the abilities of human experts in almost any cognitive task. (see 2.4.3. Will algorithmic progress lead to rapid advancements?). This latter scenario has only been considered in little economic research. That research assumes that Al systems can perform virtually all knowledge tasks more cost effectively than humans and indicates that such a scenario could lead to a collapse in wages across many sectors, severe unemployment and a dramatic reduction in labour force participation (531, 532). However, even in such a scenario some demand for human labour may persist due to consumer preferences and ethical or control-related reasons (531, 532). Among economists, extreme scenarios involving a dramatic decline in labour demand due to mass general-purpose Al automation are currently considered relatively fringe.

It is very difficult to forecast how and when general-purpose Al systems might affect various labour markets. Firstly, there is considerable uncertainty about how quickly general-purpose Al technology will advance and what its future capabilities may be. Secondly, even for a given level of technological sophistication, the extent to which general-purpose Al systems enable automation and how this kind of automation could affect labour markets is unclear. One reason for this is that in many sectors there can be considerable lags in deploying and adopting general-purpose Al systems at scale. For instance, if employees lack the necessary skills to effectively employ general-purpose Al assistance, this can slow down adoption. Overall, many economists expect modest macroeconomic impacts of general-purpose Al systems over the next decade (533), while some economists expect substantial labour market and macroeconomic effects in the next five to ten years (534). These forecasts often indicate high uncertainty due to the unknown pace of future advancements in general-purpose Al capabilities (see 2.4 Capability progress in coming years).
**The impact on wages of general-purpose AI systems leading to automation is uncertain and the evidence is mixed.** General-purpose AI automation could increase wages in some sectors by:

* Directly augmenting human productivity with general-purpose AI tools (502, 535)

* Leading to an overall growth of the economy, which lead to more investment that enhances the productivity of workers in non-automated sectors (536)

* Increasing output and boosting the demand for labour in tasks that are not yet automated, or creating new tasks and occupations (517, 537)

* Improving technology broadly through accelerating R&D (538)

If general-purpose AI systems complement human labour, then as the capabilities of general-purpose AI systems advance, wage growth could accelerate alongside economic growth, potentially much faster than historically (539). Recent large-scale surveys of the use of current general-purpose AI systems support this view. For example, a recent survey of a total of 5,334 workers and 2,053 firms in the manufacturing and financial sectors across seven OECD countries found that around 80% of workers who use AI said that AI had improved their performance at work (540).

However, if in the future automation reduces labour demand faster than new jobs are created, then it is possible that wages and the share of income going to human workers may substantially decline (541). It is also possible that general-purpose AI systems might have different economic effects at different points in time: the use of general-purpose AI systems might initially boost wages, but as more and more tasks are automated, increasing competition for the remaining jobs could push wages down (532). There is currently no clear consensus among experts regarding the net effects of general-purpose AI automation on average wages: the effects are likely to differ over time and across occupations, depending on many factors including social acceptance of the technology and organisational decision-making, as well as government policies.

## General-purpose AI could increase income inequality

General-purpose AI could increase income inequality both within and between countries. Historically, the automation of routine jobs has likely increased wage inequality within countries by displacing workers from the types of jobs in which they held a comparative advantage (517). Similarly, general-purpose AI could systematically compete with some tasks that human knowledge workers currently hold a comparative advantage in, potentially depressing wages if they cannot easily find work elsewhere (542, 543). At the same time, general-purpose AI could improve the productivity of high-income occupations, and so high-wage earners could see disproportionately larger increases in labour income, thereby amplifying labour income inequality. One simulation suggests that widespread adoption of AI could increase wage inequality between high and low-income occupations by 10% within a decade of adoption in advanced economies (519).

Automation could also exacerbate inequality by reducing labour's share of income, which would boost the relative incomes of wealthier capital owners (528). This would be part of an ongoing trend: globally, the share of income from labour has fallen by roughly six percentage points between 1980 to 2022, with similar patterns seen in the United States, Asia and Europe (544). Further automation could result in a continuation of this trend. Typically, 10% of earners earn the majority of capital income (545, 546). Hence, the greater productivity of capital would therefore act as a boon for high-earners. This dynamic could be particularly pronounced if general-purpose AI aids the creation of 'superstar' firms with strong market power, as they would capture an outsized share of economic profits (519).

Finally, general-purpose AI technology could exacerbate global inequality if it is primarily adopted by advanced economies (also see 4.3.2. Global AI divide). These countries have a higher share of cognitive task-oriented jobs exposed to the potential impacts of general-purpose AI, stronger digital infrastructure, skilled workforces, and more developed innovation ecosystems. This positions them to capture AI productivity gains more rapidly than emerging markets and developing economies, potentially leading to divergent income growth trajectories and a widening gap between high- and low-income countries (519, 547).
**The impact on wages of general-purpose AI systems leading to automation is uncertain and the evidence is mixed.** General-purpose AI automation could increase wages in some sectors by:

* Directly augmenting human productivity with general-purpose AI tools (502, 535)

* Leading to an overall growth of the economy, which lead to more investment that enhances the productivity of workers in non-automated sectors (536)

* Increasing output and boosting the demand for labour in tasks that are not yet automated, or creating new tasks and occupations (517, 537)

* Improving technology broadly through accelerating R&D (538)

If general-purpose AI systems complement human labour, then as the capabilities of general-purpose AI systems advance, wage growth could accelerate alongside economic growth, potentially much faster than historically (539). Recent large-scale surveys of the use of current general-purpose AI systems support this view. For example, a recent survey of a total of 5,334 workers and 2,053 firms in the manufacturing and financial sectors across seven OECD countries found that around 80% of workers who use AI said that AI had improved their performance at work (540).

However, if in the future automation reduces labour demand faster than new jobs are created, then it is possible that wages and the share of income going to human workers may substantially decline (541). It is also possible that general-purpose AI systems might have different economic effects at different points in time: the use of general-purpose AI systems might initially boost wages, but as more and more tasks are automated, increasing competition for the remaining jobs could push wages down (532). There is currently no clear consensus among experts regarding the net effects of general-purpose AI automation on average wages: the effects are likely to differ over time and across occupations, depending on many factors including social acceptance of the technology and organisational decision-making, as well as government policies.

## General-purpose AI could increase income inequality

General-purpose AI could increase income inequality both within and between countries. Historically, the automation of routine jobs has likely increased wage inequality within countries by displacing workers from the types of jobs in which they held a comparative advantage (517). Similarly, general-purpose AI could systematically compete with some tasks that human knowledge workers currently hold a comparative advantage in, potentially depressing wages if they cannot easily find work elsewhere (542, 543). At the same time, general-purpose AI could improve the productivity of high-income occupations, and so high-wage earners could see disproportionately larger increases in labour income, thereby amplifying labour income inequality. One simulation suggests that widespread adoption of AI could increase wage inequality between high and low-income occupations by 10% within a decade of adoption in advanced economies (519).

Automation could also exacerbate inequality by reducing labour's share of income, which would boost the relative incomes of wealthier capital owners (528). This would be part of an ongoing trend: globally, the share of income from labour has fallen by roughly six percentage points between 1980 to 2022, with similar patterns seen in the United States, Asia and Europe (544). Further automation could result in a continuation of this trend. Typically, 10% of earners earn the majority of capital income (545, 546). Hence, the greater productivity of capital would therefore act as a boon for high-earners. This dynamic could be particularly pronounced if general-purpose AI aids the creation of 'superstar' firms with strong market power, as they would capture an outsized share of economic profits (519).

Finally, general-purpose AI technology could exacerbate global inequality if it is primarily adopted by advanced economies (also see 4.3.2. Global AI divide). These countries have a higher share of cognitive task-oriented jobs exposed to the potential impacts of general-purpose AI, stronger digital infrastructure, skilled workforces, and more developed innovation ecosystems. This positions them to capture AI productivity gains more rapidly than emerging markets and developing economies, potentially leading to divergent income growth trajectories and a widening gap between high- and low-income countries (519, 547).
4.3.2 Global AI divide

KEY INFORMATION

* General-purpose Al research and development is currently concentrated in a few Western countries and China. This 'Al Divide' is multicausal, but in part related to limited access to computing power in low-income countries.
* Access to large and expensive quantities of computing power has become a prerequisite for developing advanced general-purpose Al. This has led to a growing dominance of large technology companies in general-purpose Al development.
* The Al R&D divide often overlaps with existing global socioeconomic disparities, potentially exacerbating them.

There is a well-documented concentration of Al research and development, including research on potential societal impacts of Al, in Western countries and China (316, 548, 549). This global 'Al Divide' could become even larger for general-purpose Al specifically because of the high costs associated with general-purpose Al development. Some countries face substantial barriers to benefiting from general-purpose Al development and deployment, including lower digital skills literacy, limited access to computing resources, infrastructure challenges, and economic dependence on entities in higher-income countries (519, 550). Because general-purpose Al system development is so dominated by a few companies, particularly those based in the US, there are concerns that prominent general-purpose Al systems which are used worldwide primarily reflect the values, cultures and goals of large Western corporations. In addition, the recent trend towards aiming to develop ever-larger, more powerful general-purpose Al models could also exacerbate global supply chain inequalities (551), place demands on energy usage, and lead to harmful climate effects which also worsen global inequalities (552, 553). The global general-purpose Al divide could also be harmful if biased or inequitable general-purpose Al systems are deployed globally.

**Disparities in the concentration of skilled talent and the steep financial costs of developing and sustaining general-purpose Al systems could align the Al divide with existing global socioeconomic disparities.** The United States has the largest percentage of elite Al researchers, contains a majority of the institutions who conduct top-tier research, and is the top destination for Al talent globally (554). However, countries leading in Al development also experience issues with the distribution of skilled Al talent, which is rapidly shifting towards industry. For example, 70 percent of graduates of North American universities with Al PhDs end up getting a job in private industry compared with 21% of graduates two decades ago (555).

In April 2023, OpenAI's Al systems were reportedly estimated to incur $700k/day in inference costs (77), a cost that is widely inaccessible for the vast majority of academic institutions and companies and even more so for those based in the Global South (556, 557). Low-resource regions also experience challenges with access to data given the high costs of collection, labelling, and storage. The lower availability of skilled talent to leverage these datasets for model development purposes could further contribute to the Al divide. Infrastructure concerns are a major factor that prohibit equitable access to the resources needed to train and implement general-purpose Al due to issues such as inadequate access to broadband internet (558, 559), power blackouts and insufficient access to electricity (560, 561).

**Academic institutions in China and the United States lead in general-purpose Al research production, but industry is increasingly influential.** China currently publishes the most research on Al, as measured by the total volume of articles in journals, conferences, and online repositories. Geographically, the development of significant machine learning models is concentrated in nations such as the US, Canada, the UK, and China, with at least one coming from Africa. US industry currently
4.3.2 Global AI divide

KEY INFORMATION

* General-purpose Al research and development is currently concentrated in a few Western countries and China. This 'Al Divide' is multicausal, but in part related to limited access to computing power in low-income countries.
* Access to large and expensive quantities of computing power has become a prerequisite for developing advanced general-purpose Al. This has led to a growing dominance of large technology companies in general-purpose Al development.
* The Al R&D divide often overlaps with existing global socioeconomic disparities, potentially exacerbating them.

There is a well-documented concentration of Al research and development, including research on potential societal impacts of Al, in Western countries and China (316, 548, 549). This global 'Al Divide' could become even larger for general-purpose Al specifically because of the high costs associated with general-purpose Al development. Some countries face substantial barriers to benefiting from general-purpose Al development and deployment, including lower digital skills literacy, limited access to computing resources, infrastructure challenges, and economic dependence on entities in higher-income countries (519, 550). Because general-purpose Al system development is so dominated by a few companies, particularly those based in the US, there are concerns that prominent general-purpose Al systems which are used worldwide primarily reflect the values, cultures and goals of large Western corporations. In addition, the recent trend towards aiming to develop ever-larger, more powerful general-purpose Al models could also exacerbate global supply chain inequalities (551), place demands on energy usage, and lead to harmful climate effects which also worsen global inequalities (552, 553). The global general-purpose Al divide could also be harmful if biased or inequitable general-purpose Al systems are deployed globally.

**Disparities in the concentration of skilled talent and the steep financial costs of developing and sustaining general-purpose Al systems could align the Al divide with existing global socioeconomic disparities.** The United States has the largest percentage of elite Al researchers, contains a majority of the institutions who conduct top-tier research, and is the top destination for Al talent globally (554). However, countries leading in Al development also experience issues with the distribution of skilled Al talent, which is rapidly shifting towards industry. For example, 70 percent of graduates of North American universities with Al PhDs end up getting a job in private industry compared with 21% of graduates two decades ago (555).

In April 2023, OpenAl's Al systems were reportedly estimated to incur $700k/day in inference costs (77), a cost that is widely inaccessible for the vast majority of academic institutions and companies and even more so for those based in the Global South (556, 557). Low-resource regions also experience challenges with access to data given the high costs of collection, labelling, and storage. The lower availability of skilled talent to leverage these datasets for model development purposes could further contribute to the Al divide. Infrastructure concerns are a major factor that prohibit equitable access to the resources needed to train and implement general-purpose Al due to issues such as inadequate access to broadband internet (558, 559), power blackouts and insufficient access to electricity (560, 561).

**Academic institutions in China and the United States lead in general-purpose Al research production, but industry is increasingly influential.** China currently publishes the most research on Al, as measured by the total volume of articles in journals, conferences, and online repositories. Geographically, the development of significant machine learning models is concentrated in nations such as the US, Canada, the UK, and China, with at least one coming from Africa. US industry currently