Artificial
IntelligenceIndex Report2023

Artificial IntelligenceIndex Report 2023

Introduction to the
AI Index Report 2023
Welcome to the sixth edition of the AI Index Report! This year, the report introduces more original data than any
previous edition, including a new chapter on AI public opinion, a more thorough technical performance chapter,original analysis about large language and multimodal models, detailed trends in global AI legislation records,
a study of the environmental impact of AI systems, and more.
The AI Index Report tracks, collates, distills, and visualizes data related to artificial intelligence. Our mission isto provide unbiased, rigorously vetted, broadly sourced data in order for policymakers, researchers, executives,
journalists, and the general public to develop a more thorough and nuanced understanding of the complex field of
AI. The report aims to be the world’s most credible and authoritative source for data and insights about AI.

From the Co-Directors
AI has moved into its era of deployment; throughout 2022 and the beginning of 2023, new large-scale AI models
have been released every month. These models, such as ChatGPT, Stable Diffusion, Whisper, and DALL-E 2, arecapable of an increasingly broad range of tasks, from text manipulation and analysis, to image generation, tounprecedentedly good speech recognition. These systems demonstrate capabilities in question answering and thegeneration of text, image, and code unimagined a decade ago, and they outperform the state of the art on many
benchmarks, old and new. However, they are prone to hallucination, routinely biased, and can be tricked into
serving nefarious aims, highlighting the complicated ethical challenges associated with their deployment.
Although 2022 was the first year in a decade where private AI investment decreased, AI is still a topic of greatinterest to policymakers, industry leaders, researchers, and the public. Policymakers are talking about AI morethan ever before. Industry leaders that have integrated AI into their businesses are seeing tangible cost andrevenue benefits. The number of AI publications and collaborations continues to increase. And the public is
forming sharper opinions about AI and which elements they like or dislike.
AI will continue to improve and, as such, become a greater part of all our lives. Given the increased presence of
this technology and its potential for massive disruption, we should all begin thinking more critically about how
exactly we want AI to be developed and deployed. We should also ask questions about who is deploying it—as
our analysis shows, AI is increasingly defined by the actions of a small set of private sector actors, rather than abroader range of societal actors. This year’s AI Index paints a picture of where we are so far with AI, in order tohighlight what might await us in the future.
Jack Clark and Ray Perrault

Artificial IntelligenceIndex Report 2023

Top Ten Takeaways

1

Industry races ahead of academia.

Until 2014, most significant machine learningmodels were released by academia. Since then,industry has taken over. In 2022, there were 32
significant industry-produced machine learningmodels compared to just three produced byacademia. Building state-of-the-art AI systems
increasingly requires large amounts of data, computer
power, and money—resources that industry actors
inherently possess in greater amounts compared to
nonprofits and academia.

2

Performance saturation on
traditional benchmarks.

AI continued to post state-of-the-art results, but
year-over-year improvement on many benchmarks
continues to be marginal. Moreover, the speed atwhich benchmark saturation is being reached isincreasing. However, new, more comprehensive
benchmarking suites such as BIG-bench and HELM
are being released.

3

AI is both helping andharming the environment.

New research suggests that AI systems can haveserious environmental impacts. According to
Luccioni et al., 2022, BLOOM’s training run
emitted 25 times more carbon than a single air
traveler on a one-way trip from New York to
San Francisco. Still, new reinforcement learningmodels like BCOOLER show that AI systems
can be used to optimize energy usage.

4

The world’s best new scientist … AI?

5

The number of incidents concerning
the misuse of AI is rapidly rising.

AI models are starting to rapidly acceleratescientific progress and in 2022 were used to aidhydrogen fusion, improve the efficiency of matrix
manipulation, and generate new antibodies.

According to the AIAAIC database, which tracks
incidents related to the ethical misuse of AI, thenumber of AI incidents and controversies has
increased 26 times since 2012. Some notable incidents
in 2022 included a deepfake video of Ukrainian
President Volodymyr Zelenskyy surrendering andU.S. prisons using call-monitoring technology on theirinmates. This growth is evidence of both greater use of
AI technologies and awareness of misuse possibilities.

6

The demand for AI-related
professional skills is increasing acrossvirtually every American industrial sector.

Across every sector in the United States for which
there is data (with the exception of agriculture,
forestry, fishing, and hunting), the number of AIrelated job postings has increased on average from
1.7% in 2021 to 1.9% in 2022. Employers in the UnitedStates are increasingly looking for workers with AIrelated skills.

Artificial IntelligenceIndex Report 2023

Top Ten Takeaways (cont’d)

7

For the first time in the last decade,year-over-year private investment
in AI decreased.
Global AI private investment was $91.9 billion in
2022, which represented a 26.7% decrease since2021. The total number of AI-related funding events
as well as the number of newly funded AI companies
likewise decreased. Still, during the last decade as awhole, AI investment has significantly increased. In
2022 the amount of private investment in AI was 18
times greater than it was in 2013.

8

While the proportion of companies
adopting AI has plateaued, thecompanies that have adopted AI
continue to pull ahead.
The proportion of companies adopting AI in 2022
has more than doubled since 2017, though it has
plateaued in recent years between 50% and 60%,
according to the results of McKinsey’s annual
research survey. Organizations that have adopted
AI report realizing meaningful cost decreases andrevenue increases.

9

Policymaker interest in AI
is on the rise.

An AI Index analysis of the legislative records of 127
countries shows that the number of bills containing“artificial intelligence” that were passed into law
grew from just 1 in 2016 to 37 in 2022. An analysisof the parliamentary records on AI in 81 countries
likewise shows that mentions of AI in global
legislative proceedings have increased nearly
6.5 times since 2016.

10

Chinese citizens are among those
who feel the most positively about
AI products and services. Americans …
not so much.

In a 2022 IPSOS survey, 78% of Chinese respondents(the highest proportion of surveyed countries) agreed
with the statement that products and services using
AI have more benefits than drawbacks. After Chinese
respondents, those from Saudi Arabia (76%) and India(71%) felt the most positive about AI products. Only
35% of sampled Americans (among the lowest of
surveyed countries) agreed that products and services
using AI had more benefits than drawbacks.

Artificial IntelligenceIndex Report 2023

Steering Committee
Co-directors
Jack Clark
Anthropic, OECD

Raymond Perrault
SRI International

Members
Erik Brynjolfsson
Stanford University

Katrina Ligett
Hebrew University

John Etchemendy
Stanford University

Terah Lyons

Juan Carlos Niebles
Stanford University,
Salesforce
Vanessa Parli
Stanford University

James Manyika
Google,University of Oxford

Yoav Shoham(Founding Director)
Stanford University,
AI21 Labs
Russell Wald
Stanford University

Staff and Researchers
Research Manager and Editor in Chief

Research Associate

Nestor Maslej
Stanford University

Loredana Fattorini
Stanford University

Affiliated Researchers
Elif Kiesow Cortez
Stanford Law School
Research Fellow

Helen Ngo
Hugging Face

Robi Rahman
Data Scientist

Alexandra RomeFreelance Researcher

Graduate Researcher
Han Bai
Stanford University

Undergraduate Researchers
Vania
Chow
Stanford
University

Siddhartha
Javvaji
Stanford
University

Mena
HassanStanford
University

Naima
Patel
Stanford
University

SukrutOak
Stanford
University

Stone
Yang
Stanford
University

Lucy
Zimmerman
Stanford
University

Elizabeth
Zhu
Stanford
University

Artificial IntelligenceIndex Report 2023

How to Cite This Report
Nestor Maslej, Loredana Fattorini, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons,James Manyika, Helen Ngo, Juan Carlos Niebles, Vanessa Parli, Yoav Shoham, Russell Wald, Jack Clark,
and Raymond Perrault, “The AI Index 2023 Annual Report,” AI Index Steering Committee,
Institute for Human-Centered AI, Stanford University, Stanford, CA, April 2023.
The AI Index 2023 Annual Report by Stanford University is licensed under
Attribution-NoDerivatives 4.0 International.

Public Data and Tools
The AI Index 2023 Report is supplemented by raw data and an interactive tool.
We invite each reader to use the data and the tool in a way most relevant to their work and interests.
Raw data and charts: The public data andhigh-resolution images of all the charts
in the report are available on Google Drive.

Global AI Vibrancy Tool: Compare up to
30 countries across 21 indicators. The Global AI
Vibrancy tool will be updated in the latter half of 2023.

AI Index and Stanford HAI
The AI Index is an independent initiative at the
Stanford Institute for Human-Centered Artificial Intelligence (HAI).

The AI Index was conceived within the One Hundred Year Study on AI (AI100).

We welcome feedback and new ideas for next year.
Contact us at AI-Index-Report@stanford.edu.

Artificial IntelligenceIndex Report 2023

Supporting Partners

Analytics and Research Partners

Artificial IntelligenceIndex Report 2023

Contributors
We want to acknowledge the following individuals by chapter and section for their contributions of data,
analysis, advice, and expert commentary included in the AI Index 2023 Report:

Research and Development
Sara Abdulla, Catherine Aiken, Luis Aranda, Peter Cihon, Jack Clark, Loredana Fattorini, Nestor Maslej,
Besher Massri, Vanessa Parli, Naima Patel, Ray Perrault, Robi Rahman, Alexandra Rome, Kevin Xu

Technical Performance
Jack Clark, Loredana Fattorini, Siddhartha Javvaji, Katrina Ligett, Nestor Maslej, Juan Carlos Niebles,
Sukrut Oak, Vanessa Parli, Ray Perrault, Robi Rahman, Alexandra Rome, Yoav Shoham, Elizabeth Zhu

Technical AI Ethics
Jack Clark, Loredana Fattorini, Katrina Ligett, Nestor Maslej, Helen Ngo, Sukrut Oak, Vanessa Parli,
Ray Perrault, Alexandra Rome, Elizabeth Zhu, Lucy Zimmerman

EconomySusanne Bieller, Erik Brynjolfsson, Vania Chow, Jack Clark, Natalia Dorogi, Murat Erer, Loredana Fattorini,
Akash Kaura, James Manyika, Nestor Maslej, Layla O’Kane, Vanessa Parli, Ray Perrault, Brittany Presten,
Alexandra Rome, Nicole Seredenko, Bledi Taska, Bill Valle, Casey Weston

Education
Han Bai, Betsy Bizot, Jack Clark, John Etchemendy, Loredana Fattorini, Katrina Ligett, Nestor Maslej,
Vanessa Parli, Ray Perrault, Sean Roberts, Alexandra Rome

Policy and Governance
Meghan Anand, Han Bai, Vania Chow, Jack Clark, Elif Kiesow Cortez, Rebecca DeCrescenzo, Loredana Fattorini,
Taehwa Hong, Joe Hsu, Kai Kato, Terah Lyons, Nestor Maslej, Alistair Murray, Vanessa Parli, Ray Perrault, Alexandra Rome,Sarah Smedley, Russell Wald, Brian Williams, Catherina Xu, Stone Yang, Katie Yoon, Daniel Zhang

Diversity
Han Bai, Betsy Bizot, Jack Clark, Loredana Fattorini, Nezihe Merve Gürel, Mena Hassan, Katrina Ligett,
Nestor Maslej, Vanessa Parli, Ray Perrault, Sean Roberts, Alexandra Rome, Sarah Tan, Lucy Zimmerman

Public Opinion
Jack Clark, Loredana Fattorini, Mena Hassan, Nestor Maslej, Vanessa Parli, Ray Perrault,
Alexandra Rome, Nicole Seredenko, Bill Valle, Lucy Zimmerman

Conference Attendance
Terri Auricchio (ICML), Lee Campbell (ICLR), Cassio de Campos (UAI), Meredith Ellison (AAAI), Nicole Finn (CVPR),
Vasant Gajanan (AAAI), Katja Hofmann (ICLR), Gerhard Lakemeyer (KR), Seth Lazar (FAccT), Shugen Ma (IROS),
Becky Obbema (NeurIPS), Vesna Sabljakovic-Fritz (IJCAI), Csaba Szepesvari (ICML), Matthew Taylor (AAMAS),
Sylvie Thiebaux (ICAPS), Pradeep Varakantham (ICAPS)

Artificial IntelligenceIndex Report 2023

We thank the following organizations and individuals who provided
data for inclusion in the AI Index 2023 Report:

Organizations
Code.org
Sean Roberts

Lightcast
Layla O’Kane, Bledi Taska

Center for Security andEmerging Technology,
Georgetown University
Sara Abdulla, Catherine Aiken

LinkedIn
Murat Erer, Akash Kaura,
Casey Weston

Computing Research
Association
Betsy Bizot
GitHubPeter Cihon, Kevin Xu
GoviniRebecca DeCrescenzo,
Joe Hsu, Sarah Smedley

McKinsey & Company
Natalia Dorogi, Brittany Presten
NetBase Quid
Nicole Seredenko, Bill Valle
OECD.AI Policy Observatory
Luis Aranda, Besher Massri
Women in Machine Learning
Nezihe Merve Gürel, Sarah Tan

We also would like to thank Jeanina Casusi, Nancy King, Shana Lynch, Jonathan Mindes,
Michi Turner, and Madeleine Wright for their help in preparing this report, and Joe Hinman andSantanu Mukherjee for their help in maintaining the AI Index website.

Artificial IntelligenceIndex Report 2023

Table of Contents
Report Highlights		 11Chapter 1

Research and Development		 20Chapter 2

Technical Performance		 69Chapter 3

Technical AI Ethics

125

Chapter 4

The Economy

168

Chapter 5

Education

234

Chapter 6

Policy and Governance

263

Chapter 7

Diversity

296

Chapter 8

Public Opinion

319

Appendix		ACCESS THE PUBLIC DATA

344

Artificial IntelligenceIndex Report 2023

Report Highlights
Chapter 1: Research and Development
The United States and China had the greatest number of cross-country collaborations in AI
publications from 2010 to 2021, although the pace of collaboration has slowed. The number of AI
research collaborations between the United States and China increased roughly 4 times since 2010,
and was 2.5 times greater than the collaboration totals of the next nearest country pair, the United
Kingdom and China. However the total number of U.S.-China collaborations only increased by 2.1%
from 2020 to 2021, the smallest year-over-year growth rate since 2010.
AI research is on the rise, across the board. The total number of AI publications has more thandoubled since 2010. The specific AI topics that continue dominating research include patternrecognition, machine learning, and computer vision.
China continues to lead in total AI journal, conference, and repository publications.
The United States is still ahead in terms of AI conference and repository citations, but thoseleads are slowly eroding. Still, the majority of the world’s large language and multimodal models(54% in 2022) are produced by American institutions.
Industry races ahead of academia. Until 2014, most significant machine learning models werereleased by academia. Since then, industry has taken over. In 2022, there were 32 significant
industry-produced machine learning models compared to just three produced by academia.
Building state-of-the-art AI systems increasingly requires large amounts of data, computer power,
and money—resources that industry actors inherently possess in greater amounts compared to
nonprofits and academia.
Large language models are getting bigger and more expensive. GPT-2, released in 2019,considered by many to be the first large language model, had 1.5 billion parameters and cost an
estimated $50,000 USD to train. PaLM, one of the flagship large language models launched in 2022,
had 540 billion parameters and cost an estimated $8 million USD—PaLM was around 360 times
larger than GPT-2 and cost 160 times more. It’s not just PaLM: Across the board, large language andmultimodal models are becoming larger and pricier.

Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
Performance saturation on traditional benchmarks. AI continued to post state-of-the-art results,
but year-over-year improvement on many benchmarks continues to be marginal. Moreover,the speed at which benchmark saturation is being reached is increasing. However, new, morecomprehensive benchmarking suites such as BIG-bench and HELM are being released.
Generative AI breaks into the public consciousness. 2022 saw the release of text-to-image
models like DALL-E 2 and Stable Diffusion, text-to-video systems like Make-A-Video, and chatbots
like ChatGPT. Still, these systems can be prone to hallucination, confidently outputting incoherent or
untrue responses, making it hard to rely on them for critical applications.
AI systems become more flexible. Traditionally AI systems have performed well on narrow tasks
but have struggled across broader tasks. Recently released models challenge that trend; BEiT-3,
PaLI, and Gato, among others, are single AI systems increasingly capable of navigating multiple tasks(for example, vision, language).
Capable language models still struggle with reasoning. Language models continued to improvetheir generative capabilities, but new research suggests that they still struggle with complex
planning tasks.
AI is both helping and harming the environment. New research suggests that AI systems can haveserious environmental impacts. According to Luccioni et al., 2022, BLOOM’s training run emitted 25
times more carbon than a single air traveler on a one-way trip from New York to San Francisco. Still,
new reinforcement learning models like BCOOLER show that AI systems can be used to optimize
energy usage.
The world’s best new scientist … AI? AI models are starting to rapidly accelerate scientificprogress and in 2022 were used to aid hydrogen fusion, improve the efficiency of matrix
manipulation, and generate new antibodies.
AI starts to build better AI. Nvidia used an AI reinforcement learning agent to improve the designof the chips that power AI systems. Similarly, Google recently used one of its language models,
PaLM, to suggest ways to improve the very same model. Self-improving AI learning will accelerate
AI progress.

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
The effects of model scale on bias and toxicity are confounded by training data and mitigation
methods. In the past year, several institutions have built their own large models trained on
proprietary data—and while large models are still toxic and biased, new evidence suggests thatthese issues can be somewhat mitigated after training larger models with instruction-tuning.
Generative models have arrived and so have their ethical problems. In 2022, generative models
became part of the zeitgeist. These models are capable but also come with ethical challenges. Textto-image generators are routinely biased along gender dimensions, and chatbots like ChatGPT canbe tricked into serving nefarious aims.
The number of incidents concerning the misuse of AI is rapidly rising. According to the AIAAIC
database, which tracks incidents related to the ethical misuse of AI, the number of AI incidents
and controversies has increased 26 times since 2012. Some notable incidents in 2022 included a
deepfake video of Ukrainian President Volodymyr Zelenskyy surrendering and U.S. prisons using
call-monitoring technology on their inmates. This growth is evidence of both greater use of AI
technologies and awareness of misuse possibilities.
Fairer models may not be less biased. Extensive analysis of language models suggests that while there
is a clear correlation between performance and fairness, fairness and bias can be at odds: Language
models which perform better on certain fairness benchmarks tend to have worse gender bias.
Interest in AI ethics continues to skyrocket. The number of accepted submissions to FAccT, aleading AI ethics conference, has more than doubled since 2021 and increased by a factor of 10 since
2018. 2022 also saw more submissions than ever from industry actors.
Automated fact-checking with natural language processing isn’t so straightforward after all.
While several benchmarks have been developed for automated fact-checking, researchers find that
11 of 16 of such datasets rely on evidence “leaked” from fact-checking reports which did not exist at
the time of the claim surfacing.

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
The demand for AI-related professional skills is increasing across virtually every Americanindustrial sector. Across every sector in the United States for which there is data (with the exceptionof agriculture, forestry, fishing, and hunting), the number of AI-related job postings has increased on
average from 1.7% in 2021 to 1.9% in 2022. Employers in the United States are increasingly looking forworkers with AI-related skills.
For the first time in the last decade, year-over-year private investment in AI decreased.
Global AI private investment was $91.9 billion in 2022, which represented a 26.7% decrease since 2021.
The total number of AI-related funding events as well as the number of newly funded AI companies
likewise decreased. Still, during the last decade as a whole, AI investment has significantly increased.
In 2022 the amount of private investment in AI was 18 times greater than it was in 2013.
Once again, the United States leads in investment in AI. The U.S. led the world in terms of total
amount of AI private investment. In 2022, the $47.4 billion invested in the U.S. was roughly 3.5 times
the amount invested in the next highest country, China ($13.4 billion). The U.S. also continues to lead interms of total number of newly funded AI companies, seeing 1.9 times more than the European Unionand the United Kingdom combined, and 3.4 times more than China.
In 2022, the AI focus area with the most investment was medical and healthcare ($6.1 billion);
followed by data management, processing, and cloud ($5.9 billion); and Fintech ($5.5 billion).
However, mirroring the broader trend in AI private investment, most AI focus areas saw lessinvestment in 2022 than in 2021. In the last year, the three largest AI private investment events were:(1) a $2.5 billion funding event for GAC Aion New Energy Automobile, a Chinese manufacturer ofelectric vehicles; (2) a $1.5 billion Series E funding round for Anduril Industries, a U.S. defense products
company that builds technology for military agencies and border surveillance; and (3) a $1.2 billion
investment in Celonis, a business-data consulting company based in Germany.
While the proportion of companies adopting AI has plateaued, the companies that have adopted
AI continue to pull ahead. The proportion of companies adopting AI in 2022 has more than doubled
since 2017, though it has plateaued in recent years between 50% and 60%, according to the results of
McKinsey’s annual research survey. Organizations that have adopted AI report realizing meaningful
cost decreases and revenue increases.

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy (cont’d)
AI is being deployed by businesses in multifaceted ways. The AI capabilities most likely to havebeen embedded in businesses include robotic process automation (39%), computer vision (34%), NL
text understanding (33%), and virtual agents (33%). Moreover, the most commonly adopted AI use
case in 2022 was service operations optimization (24%), followed by the creation of new AI-based
products (20%), customer segmentation (19%), customer service analytics (19%), and new AI-basedenhancement of products (19%).
AI tools like Copilot are tangibly helping workers. Results of a GitHub survey on the use of Copilot,
a text-to-code AI system, find that 88% of surveyed respondents feel more productive when using
the system, 74% feel they are able to focus on more satisfying work, and 88% feel they are able tocomplete tasks more quickly.
China dominates industrial robot installations. In 2013, China overtook Japan as the nation installingthe most industrial robots. Since then, the gap between the total number of industrial robots installedby China and the next-nearest nation has widened. In 2021, China installed more industrial robots than
the rest of the world combined.

Artificial IntelligenceIndex Report 2023

Chapter 5: Education
More and more AI specialization. The proportion of new computer science PhD graduates from
U.S. universities who specialized in AI jumped to 19.1% in 2021, from 14.9% in 2020 and 10.2% in 2010.
New AI PhDs increasingly head to industry. In 2011, roughly the same proportion of new AI PhD
graduates took jobs in industry (40.9%) as opposed to academia (41.6%). Since then, however, amajority of AI PhDs have headed to industry. In 2021, 65.4% of AI PhDs took jobs in industry, morethan double the 28.2% who took jobs in academia.
New North American CS, CE, and information faculty hires stayed flat. In the last decade,the total number of new North American computer science (CS), computer engineering (CE),and information faculty hires has decreased: There were 710 total hires in 2021 compared to733 in 2012. Similarly, the total number of tenure-track hires peaked in 2019 at 422 and thendropped to 324 in 2021.
The gap in external research funding for private versus public American CS departments
continues to widen. In 2011, the median amount of total expenditure from external sources forcomputing research was roughly the same for private and public CS departments in the UnitedStates. Since then, the gap has widened, with private U.S. CS departments receiving millions morein additional funding than public universities. In 2021, the median expenditure for private universities
was $9.7 million, compared to $5.7 million for public universities.
Interest in K–12 AI and computer science education grows in both the United States and therest of the world. In 2021, a total of 181,040 AP computer science exams were taken by American
students, a 1.0% increase from the previous year. Since 2007, the number of AP computer scienceexams has increased ninefold. As of 2021, 11 countries, including Belgium, China, and South Korea,
have officially endorsed and implemented a K–12 AI curriculum.

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
Policymaker interest in AI is on the rise. An AI Index analysis of the legislative records of 127
countries shows that the number of bills containing “artificial intelligence” that were passed into law
grew from just 1 in 2016 to 37 in 2022. An analysis of the parliamentary records on AI in 81 countries
likewise shows that mentions of AI in global legislative proceedings have increased nearly 6.5 times
since 2016.
From talk to enactment—the U.S. passed more AI bills than ever before. In 2021, only 2% of
all federal AI bills in the United States were passed into law. This number jumped to 10% in 2022.
Similarly, last year 35% of all state-level AI bills were passed into law.
When it comes to AI, policymakers have a lot of thoughts. A qualitative analysis of theparliamentary proceedings of a diverse group of nations reveals that policymakers think about
AI from a wide range of perspectives. For example, in 2022, legislators in the United Kingdom
discussed the risks of AI-led automation; those in Japan considered the necessity of safeguarding
human rights in the face of AI; and those in Zambia looked at the possibility of using AI for
weather forecasting.
The U.S. government continues to increase spending on AI. Since 2017, the amount of U.S.
government AI-related contract spending has increased roughly 2.5 times.
The legal world is waking up to AI. In 2022, there were 110 AI-related legal cases in UnitedStates state and federal courts, roughly seven times more than in 2016. The majority of these cases
originated in California, New York, and Illinois, and concerned issues relating to civil, intellectual
property, and contract law.

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
North American bachelor’s, master’s, and PhD-level computer science students are becoming
more ethnically diverse. Although white students are still the most represented ethnicity among
new resident bachelor’s, master’s, and PhD-level computer science graduates, students from otherethnic backgrounds (for example, Asian, Hispanic, and Black or African American) are becoming
increasingly more represented. For example, in 2011, 71.9% of new resident CS bachelor’s graduateswere white. In 2021, that number dropped to 46.7%.
New AI PhDs are still overwhelmingly male. In 2021, 78.7% of new AI PhDs were male.
Only 21.3% were female, a 3.2 percentage point increase from 2011. There continues to be a genderimbalance in higher-level AI education.
Women make up an increasingly greater share of CS, CE, and information faculty hires.
Since 2017, the proportion of new female CS, CE, and information faculty hires has increased from
24.9% to 30.2%. Still, most CS, CE, and information faculty in North American universities are male
(75.9%). As of 2021, only 0.1% of CS, CE, and information faculty identify as nonbinary.
American K–12 computer science education has become more diverse, in terms of both genderand ethnicity. The share of AP computer science exams taken by female students increased from
16.8% in 2007 to 30.6% in 2021. Year over year, the share of Asian, Hispanic/Latino/Latina, andBlack/African American students taking AP computer science has likewise increased.

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
Chinese citizens are among those who feel the most positively about AI products and services.
Americans … not so much. In a 2022 IPSOS survey, 78% of Chinese respondents (the highest
proportion of surveyed countries) agreed with the statement that products and services using AI
have more benefits than drawbacks. After Chinese respondents, those from Saudi Arabia (76%) andIndia (71%) felt the most positive about AI products. Only 35% of sampled Americans (among thelowest of surveyed countries) agreed that products and services using AI had more benefits thandrawbacks.
Men tend to feel more positively about AI products and services than women. Men are also
more likely than women to believe that AI will mostly help rather than harm. According to the2022 IPSOS survey, men are more likely than women to report that AI products and services maketheir lives easier, trust companies that use AI, and feel that AI products and services have more
benefits than drawbacks. A 2021 survey by Gallup and Lloyd’s Register Foundation likewise revealed
that men are more likely than women to agree with the statement that AI will mostly help rather thanharm their country in the next 20 years.
People across the world and especially America remain unconvinced by self-driving cars. In
a global survey, only 27% of respondents reported feeling safe in a self-driving car. Similarly, Pew
Research suggests that only 26% of Americans feel that driverless passenger vehicles are a goodidea for society.
Different causes for excitement and concern. Among a sample of surveyed Americans, those
who report feeling excited about AI are most excited about the potential to make life and society
better (31%) and to save time and make things more efficient (13%). Those who report feeling moreconcerned worry about the loss of human jobs (19%); surveillance, hacking, and digital privacy (16%);
and the lack of human connection (12%).
NLP researchers … have some strong opinions as well. According to a survey widely distributed to
NLP researchers, 77% either agreed or weakly agreed that private AI firms have too much influence,
41% said that NLP should be regulated, and 73% felt that AI could soon lead to revolutionary societal
change. These were some of the many strong opinions held by the NLP research community.

Artificial IntelligenceIndex Report 2023

Artificial IntelligenceIndex Report 2023

CHAPTER 1:

Research andDevelopment

Table of Contents

Chapter 1 Preview

20Artificial IntelligenceIndex Report 2023

CHAPTER 1 PREVIEW:

Research and Development
Overview

22

Computer Vision

46

Chapter Highlights

23

Natural Language Processing

47

Speech Recognition

48

1.1 Publications

24

Overview

24

Total Number of AI Publications

24

1.2 Trends in Significant
Machine Learning Systems

By Type of Publication

25

General Machine Learning Systems

By Field of Study

26

System Types49

By Sector

27

Sector Analysis

50

Cross-Country Collaboration

29

National Affiliation

51

Cross-Sector Collaboration

31

		Systems

51

32

		Authorship

53

AI Journal Publications

49
49

Overview

32

Parameter Trends

54

By Region

33

Compute Trends

56

By Geographic Area

34

Citations35

National Affiliation

58

36

Parameter Count

60

Overview

36

Training Compute61

By Region

37

Training Cost

62

By Geographic Area

38

Citations39

1.3 AI Conferences

64

AI Repositories

40

Conference Attendance64

Overview

40

By Region41

1.4 Open-Source AI Software

66

By Geographic Area

42

Projects

66

Citations

43

Stars

68

AI Conference Publications

Narrative Highlight:
Top Publishing Institutions
All Fields

Table of Contents

44

Large Language and Multimodal Models

58

ACCESS THE PUBLIC DATA

44

Chapter 1 Preview

21Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development

Overview
This chapter captures trends in AI R&D. It begins by examining AI publications,
including journal articles, conference papers, and repositories. Next it considers dataon significant machine learning systems, including large language and multimodal
models. Finally, the chapter concludes by looking at AI conference attendance andopen-source AI research. Although the United States and China continue to dominate
AI R&D, research efforts are becoming increasingly geographically dispersed.

Table of Contents

Chapter 1 Preview

22Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development

Chapter Highlights
The United States and China
had the greatest number ofcross-country collaborations in AI
publications from 2010 to 2021,
although the pace of collaboration
has since slowed.
The number of AI research collaborations between
the United States and China increased roughly 4times since 2010, and was 2.5 times greater than thecollaboration totals of the next nearest country pair,
the United Kingdom and China. However, the total
number of U.S.-China collaborations only increasedby 2.1% from 2020 to 2021, the smallest year-overyear growth rate since 2010.

AI research is on the rise, acrossthe board. The total number of AI publications
has more than doubled since 2010. The specific AI
topics that continue to dominate research include
pattern recognition, machine learning,and computer vision.

China continues to lead in total
AI journal, conference, andrepository publications.
The United States is still ahead in terms of AI
conference and repository citations, but those leads
are slowly eroding. Still, the majority of the world’s
large language and multimodal models (54% in 2022)
are produced by American institutions.

Table of Contents

Chapter 1 Preview

Industry races aheadof academia.
Until 2014, most significant machinelearning models were released byacademia. Since then, industry has taken
over. In 2022, there were 32 significant
industry-produced machine learningmodels compared to just three producedby academia. Building state-of-the-art
AI systems increasingly requires large
amounts of data, computer power, andmoney—resources that industry actors
inherently possess in greater amounts
compared to nonprofits and academia.

Large language models
are getting bigger andmore expensive.
GPT-2, released in 2019, consideredby many to be the first large language
model, had 1.5 billion parameters andcost an estimated $50,000 USD totrain. PaLM, one of the flagship large
language models launched in 2022,
had 540 billion parameters and cost an
estimated $8 million USD—PaLM wasaround 360 times larger than GPT-2 andcost 160 times more. It’s not just PaLM:
Across the board, large language andmultimodal models are becoming larger
and pricier.

23Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

This section draws on data from the Center for Security and Emerging Technology (CSET) at Georgetown University. CSET maintains amerged corpus of scholarly literature that includes Digital Science’s Dimensions, Clarivate’s Web of Science, Microsoft Academic Graph,
China National Knowledge Infrastructure, arXiv, and Papers With Code. In that corpus, CSET applied a classifier to identify Englishlanguage publications related to the development or application of AI and ML since 2010. For this year’s report, CSET also used selectChinese AI keywords to identify Chinese-language AI papers; CSET did not deploy this method for previous iterations of the AI Index report.1
In last year’s edition of the report, publication trends were reported up to the year 2021. However, given that there is a significant lag in thecollection of publication metadata, and that in some cases it takes until the middle of any given year to fully capture the previous year’s
publications, in this year’s report, the AI Index team elected to examine publication trends only through 2021, which we, along with CSET,
are confident yields a more fully representative report.

1.1 Publications
Overview

publication and citation data by region for AI journal
articles, conference papers, repositories, and patents.

The figures below capture the total number
of English-language and Chinese-language AI

Total Number of AI Publications

publications globally from 2010 to 2021—by type,Figure 1.1.1 shows the number of AI publications inaffiliation, cross-country collaboration, and cross-the world. From 2010 to 2021, the total number ofindustry collaboration. The section also breaks down

AI publications more than doubled, growing from
200,000 in 2010 to almost 500,000 in 2021.

Number of AI Publications in the World, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report496.01

Number of AI Publications (in Thousands)

500

400

300

200

100

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.1

1 See the Appendix for more information on CSET’s methodology. For more on the challenge of defining AI and correctly capturing relevant bibliometric data, see the AI Index team’s
discussion in the paper “Measurement in AI Policy: Opportunities and Challenges.”

Table of Contents

Chapter 1 Preview

24Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Type of Publication

book chapters, theses, and unknown document types

Figure 1.1.2 shows the types of AI publications released

made up the remaining 10% of publications. While

globally over time. In 2021, 60% of all published AI

journal and repository publications have grown 3

documents were journal articles, 17% were conference

and 26.6 times, respectively, in the past 12 years, thepapers, and 13% were repository submissions. Books,

number of conference papers has declined since 2019.

Number of AI Publications by Type, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report300

293.48, Journal

Number of AI Publications (in Thousands)

270
240
210
180
150
120
90

85.09, Conference
65.21, Repository60

29.88, Thesis
13.77, Book Chapter
5.82, Unknown
2.76, Book

30
0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.2

Table of Contents

Chapter 1 Preview

25Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Field of Studyroughly doubled while the number of machine learning

Figure 1.1.3 shows that publications in patternpapers has roughly quadrupled. Following those tworecognition and machine learning have experienced

topic areas, in 2021, the next most published AI fields

the sharpest growth in the last half decade. Sinceof study were computer vision (30,075), algorithm

2015, the number of pattern recognition papers has(21,527), and data mining (19,181).

Number of AI Publications by Field of Study (Excluding Other AI), 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Number of AI Publications (in Thousands)

60

59.36, Pattern Recognition50

42.55, Machine Learning
40

30

30.07, Computer Vision20

21.53, Algorithm
19.18, Data Mining

10

14.99, Natural Language Processing
11.57, Control Theory
10.37, Human–Computer Interaction
6.74, Linguistics

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.3

Table of Contents

Chapter 1 Preview

26Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Sector
This section shows the number of AI publications

1.1.5).2 The education sector dominates in each region.

affiliated with education, government, industry,

The level of industry participation is highest in the

nonprofit, and other sectors—first globally (Figure

United States, then in the European Union. Since

1.1.4), then looking at the United States, China, and2010, the share of education AI publications has been

the European Union plus the United Kingdom (Figure

dropping in each region.

AI Publications (% of Total) by Sector, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report80%
75.23%, Education
70%

AI Publications (% of Total)

60%

50%

40%

30%

20%
13.60%, Nonpro t
10%

7.21%, Industry
3.74%, Government
0.22%, Other

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.4

2 The categorization is adapted based on the Global Research Identifier Database (GRID). Healthcare, including hospitals and facilities, is included under nonprofit. Publications affiliated withstate-sponsored universities are included in the education sector.

Table of Contents

Chapter 1 Preview

27Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

AI Publications (% of Total) by Sector and Geographic Area, 2021
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

69.17%
Education

69.23%
77.85%
14.82%

Nonpro t

18.63%
11.73%
12.60%

Industry

7.90%
5.47%
3.21%

Government

3.92%
4.74%
United States

0.20%

European Union and United Kingdom

Other 0.33%

China

0.20%
0%

Table of Contents

10%

20%

Chapter 1 Preview

30%

40%
50%
AI Publications (% of Total)

60%

70%

80%
Figure 1.1.5

28

Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Cross-Country Collaboration
Cross-border collaborations between academics,

By far, the greatest number of collaborations in the

researchers, industry experts, and others are a key

past 12 years took place between the United Statescomponent of modern STEM (science, technology,

and China, increasing roughly four times since 2010.

engineering, and mathematics) development that

However the total number of U.S.-China collaborations

accelerate the dissemination of new ideas and theonly increased by 2.1% from 2020 to 2021, the smallest

growth of research teams. Figures 1.1.6 and 1.1.7 depict

year-over-year growth rate since 2010.

the top cross-country AI collaborations from 2010
to 2021. CSET counted cross-country collaborations
as distinct pairs of countries across authors for eachpublication (e.g., four U.S. and four Chinese-affiliatedauthors on a single publication are counted as one
U.S.-China collaboration; two publications between

The next largest set of collaborations was between
the United Kingdom and both China and the UnitedStates. In 2021, the number of collaborations between
the United States and China was 2.5 times greaterthan between the United Kingdom and China.

the same authors count as two collaborations).

United States and China Collaborations in AI Publications, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

10.47

Number of AI Publications (in Thousands)

10

8

6

4

2

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.6

Table of Contents

Chapter 1 Preview

29Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Cross-Country Collaborations in AI Publications (Excluding U.S. and China), 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

4.13, United Kingdom and China
4.04, United States and United Kingdom

Number of AI Publications (in Thousands)

4

3.42, United States and Germany
32.80, China and Australia
2.61, United States and Australia

2
1.83, United States and France

1

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.7

Table of Contents

Chapter 1 Preview

30Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Cross-Sector Collaboration
The increase in AI research outside of academia has

educational institutions (12,856); and educational

broadened and grown collaboration across sectors

and government institutions (8,913). Collaborations

in general. Figure 1.1.8 shows that in 2021 educational

between educational institutions and industry haveinstitutions and nonprofits (32,551) had the greatest

been among the fastest growing, increasing 4.2 times

number of collaborations; followed by industry andsince 2010.

Cross-Sector Collaborations in AI Publications, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

32.55, Education and Nonpro t

Number of AI Publications (in Thousands)

30

25

20

15
12.86, Industry and Education
10

8.91, Education and Government

52.95, Government and Nonpro t
2.26, Industry and Nonpro t
0.63, Industry and Government

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.8

Table of Contents

Chapter 1 Preview

31Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

AI Journal Publications
Overview
After growing only slightly from 2010 to 2015, the number of AI journal publications grew around 2.3 times since2015. From 2020 to 2021, they increased 14.8% (Figure 1.1.9).

Number of AI Journal Publications, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

293.48

Number of AI Journal Publications (in Thousands)

300

250

200

150

100

50

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.9

Table of Contents

Chapter 1 Preview

32Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Region3
Figure 1.1.10 shows the share of AI journal publications

East Asia and the Pacific; Europe and Central Asia;

by region between 2010 and 2021. In 2021, East Asia

as well as North America have been declining.

and the Pacific led with 47.1%, followed by Europe

During that period, there has been an increase in

and Central Asia (17.2%), and then North America

publications from other regions such as South Asia;(11.6%). Since 2019, the share of publications fromand the Middle East and North Africa.

AI Journal Publications (% of World Total) by Region, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

50%

AI Journal Publications (% of World Total)

47.14%, East Asia and Paci c

40%

30%

20%
17.20%, Europe and Central Asia
11.61%, North America
6.93%, Unknown
6.75%, South Asia
4.64%, Middle East and North Africa
2.66%, Latin America and the Caribbean
2.30%, Rest of the World
0.77%, Sub-Saharan Africa

10%

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.10

3 Regions in this chapter are classified according to the World Bank analytical grouping.

Table of Contents

Chapter 1 Preview

33Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Geographic Area4
Figure 1.1.11 breaks down the share of AI journal

throughout, with 39.8% in 2021, followed by the

publications over the past 12 years by geographic

European Union and the United Kingdom (15.1%),

area. This year’s AI Index included India in recognition

then the United States (10.0%). The share of Indianof the increasingly important role it plays in the

publications has been steadily increasing—from 1.3%

AI ecosystem. China has remained the leader

in 2010 to 5.6% in 2021.

AI Journal Publications (% of World Total) by Geographic Area, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

AI Journal Publications (% of World Total)

40%

39.78%, China

30%

22.70%, Rest of the World20%
15.05%, European Union and United Kingdom
10.03%, United States

10%

6.88%, Unknown
5.56%, India

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.11

4 In this chapter we use “geographic area” based on CSET’s classifications, which are disaggregated not only by country, but also by territory. Further, we count the European Union and theUnited Kingdom as a single geographic area to reflect the regions’ strong history of research collaboration.

Table of Contents

Chapter 1 Preview

34Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Citations
China’s share of citations in AI journal publications

1.1.12). China, the European Union and the United

has gradually increased since 2010, while those of theKingdom, and the United States accounted for 65.7%

European Union and the United Kingdom, as well asof the total citations in the world.

those of the United States, have decreased (Figure

AI Journal Citations (% of World Total) by Geographic Area, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report30%

29.07%, China

AI Journal Citations (% of World Total)

27.37%, Rest of the World25%
21.51%, European Union and United Kingdom
20%

15.08%, United States

15%

10%

6.05%, India

5%

0.92%, Unknown

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.12

Table of Contents

Chapter 1 Preview

35Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

AI Conference Publications
Overview
The number of AI conference publications peaked in 2019, and fell 20.4% below the peak in 2021 (Figure 1.1.13).
The total number of 2021 AI conference publications, 85,094, was marginally greater than the 2010 total of 75,592.

Number of AI Conference Publications, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Number of AI Conference Publications (in Thousands)

100
85.09
80

60

40

20

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.13

Table of Contents

Chapter 1 Preview

36Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Region
Figure 1.1.14 shows the number of AI conference

East Asia and the Pacific continues to rise, accounting

publications by region. As with the trend in journal

for 36.7% in 2021, followed by Europe and Central

publications, East Asia and the Pacific; Europe

Asia (22.7%), and then North America (19.6%). The

and Central Asia; and North America account forpercentage of AI conference publications in South Asia

the world’s highest numbers of AI conference

saw a noticeable rise in the past 12 years, growing frompublications. Specifically, the share represented by

3.6% in 2010 to 8.5% in 2021.

AI Conference Publications (% of World Total) by Region, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report40%

AI Conference Publications (% of World Total)

36.72%, East Asia and Paci c
35%
30%
25%
22.66%, Europe and Central Asia
20%

19.56%, North America

15%
10%

8.45%, South Asia
3.82%, Middle East and North Africa
3.07%, Latin America and the Caribbean
2.76%, Unknown
2.35%, Rest of the World
0.60%, Sub-Saharan Africa

5%
0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.14

Table of Contents

Chapter 1 Preview

37Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Geographic Area
In 2021, China produced the greatest share of thecame in third at 17.2% (Figure 1.1.15). Mirroring trends

world’s AI conference publications at 26.2%, having

seen in other parts of the research and development

overtaken the European Union and the United

section, India’s share of AI conference publications is

Kingdom in 2017. The European Union plus the United

also increasing.

Kingdom followed at 20.3%, and the United States

AI Conference Publications (% of World Total) by Geographic Area, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

AI Conference Publications (% of World Total)

30%
26.84%, Rest of the World
26.15%, China

25%

20.29%, European Union and United Kingdom

20%

17.23%, United States
15%

10%
6.79%, India
5%
2.70%, Unknown
0%

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.15

Table of Contents

Chapter 1 Preview

38Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Citations
Despite China producing the most AI conference

conference citations, with 23.9%, followed by China’s

publications in 2021, Figure 1.1.16 shows that

22.0%. However, the gap between American andthe United States had the greatest share of AI

Chinese AI conference citations is narrowing.

AI Conference Citations (% of World Total) by Geographic Area, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

35%

AI Conference Citations (% of World Total)

30%

25.57%, Rest of the World
23.86%, United States
22.02%, China
21.59%, European Union and United Kingdom

25%

20%

15%

10%
6.09%, India

5%0%0.87%, Unknown
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.16

Table of Contents

Chapter 1 Preview

39Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

AI Repositories
Overview
Publishing pre-peer-reviewed papers on repositories

share their findings before submitting them to journals

of electronic preprints (such as arXiv and SSRN)

and conferences, thereby accelerating the cycle ofhas become a popular way for AI researchers to

information discovery. The number of AI repository

disseminate their work outside traditional avenues for

publications grew almost 27 times in the past 12 years

publication. These repositories allow researchers to(Figure 1.1.17).

Number of AI Repository Publications, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Number of AI Repository Publications (in Thousands)

65.21
60

50

40

30

20

10

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.17

Table of Contents

Chapter 1 Preview

40Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Region
Figure 1.1.18 shows that North America has

by East Asia and the Pacific has grown significantly

maintained a steady lead in the world share of AI

since 2010 and continued growing from 2020 torepository publications since 2016. Since 2011, the2021, a period in which the year-over-year share ofshare of repository publications from Europe andNorth American as well European and Central AsianCentral Asia has declined. The share represented

repository publications declined.

AI Repository Publications (% of World Total) by Region, 2010–21

AI Repository Publications (% of World Total)

Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report30%
26.32%, North America
23.99%, Unknown
21.40%, Europe and Central Asia

20%

17.88%, East Asia and Paci c

10%
3.41%, South Asia
3.06%, Middle East and North Africa
1.81%, Rest of the World
1.80%, Latin America and the Caribbean
0.34%, Sub-Saharan Africa

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.18

Table of Contents

Chapter 1 Preview

41Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

By Geographic Area
While the United States has held the lead in the(Figure 1.1.19). In 2021, the United States accounted

percentage of global AI repository publications sincefor 23.5% of the world’s AI repository publications,2016, China is catching up, while the European Union

followed by the European Union plus the United

plus the United Kingdom’s share continues to drop

Kingdom (20.5%), and then China (11.9%).

AI Repository Publications (% of World Total) by Geographic Area, 2010–21

AI Repository Publications (% of World Total)

Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report30%

23.48%, United States
23.18%, Unknown
20.54%, European Union and United Kingdom

20%

18.07%, Rest of the World

11.87%, China
10%

2.85%, India
0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.19

Table of Contents

Chapter 1 Preview

42Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Citations
In the citations of AI repository publications, Figure

a dominant lead over the European Union plus the

1.1.20 shows that in 2021 the United States topped

United Kingdom (21.5%), as well as China (21.0%).

the list with 29.2% of overall citations, maintaining

AI Repository Citations (% of World Total) by Geographic Area, 2010–21
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

AI Repository Citations (% of World Total)

40%

30%

29.22%, United States

21.79%, Rest of the World
21.52%, European Union and United Kingdom
20.98%, China

20%

10%

4.59%, Unknown
1.91%, India
0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.20

Table of Contents

Chapter 1 Preview

43Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Narrative Highlight:

Top Publishing Institutions
All Fields

University, the University of the Chinese Academy

Since 2010, the institution producing the greatestof Sciences, Shanghai Jiao Tong University,

number of total AI papers has been the Chineseand Zhejiang University.5 The total number of

Academy of Sciences (Figure 1.1.21). The next

publications released by each of these institutions

top four are all Chinese universities: Tsinghua

in 2021 is displayed in Figure 1.1.22.

Top Ten Institutions in the World in 2021 Ranked by Number of AI Publications in All Fields, 2010–21

Rank

Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

1

1, Chinese Academy of Sciences2

2, Tsinghua University

3

3, University of Chinese Academy of Sciences

44, Shanghai Jiao Tong University

55, Zhejiang University

6

6, Harbin Institute of Technology

77, Beihang University

88, University of Electronic Science and Technology of China

99, Peking University

10

10, Massachusetts Institute of Technology
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

Figure 1.1.21

5 It is important to note that many Chinese research institutions are large, centralized organizations with thousands of researchers. It is therefore not entirely surprising that,
purely by the metric of publication count, they outpublish most non-Chinese institutions.

Table of Contents

Chapter 1 Preview

44Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Narrative Highlight:

Top Publishing Institutions (cont’d)
Top Ten Institutions in the World by Number of AI Publications in All Fields, 2021
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Chinese Academy of Sciences

5,099

Tsinghua University

3,373

University of Chinese Academyof Sciences

2,904

Shanghai Jiao Tong University

2,703

Zhejiang University

2,590

Harbin Institute of Technology

2,016

Beihang University

1,970

University of Electronic Scienceand Technology of China

1,951

Peking University

1,893

Massachusetts Institute ofTechnology

1,745
0

500

1,000

1,500

2,000
2,500
3,000
Number of AI Publications

3,500

4,000

4,500

5,000

Figure 1.1.22

Table of Contents

Chapter 1 Preview

45Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Narrative Highlight:

Top Publishing Institutions (cont’d)Computer Vision
In 2021, the top 10 institutions publishing the greatest number of AI computer vision publications were
all Chinese (Figure 1.1.23). The Chinese Academy of Sciences published the largest number of such
publications, with a total of 562.

Top Ten Institutions in the World by Number of AI Publications in Computer Vision, 2021
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Chinese Academy of Sciences

562

Shanghai Jiao Tong University

316

University of Chinese Academyof Sciences

314

Tsinghua University

296

Zhejiang University

289

Beihang University

247

Wuhan University

231

Beijing Institute of Technology

229

Harbin Institute of Technology

210

Tianjin University
0

182
100

200

300
Number of AI Publications

400

500

Figure 1.1.23

Table of Contents

Chapter 1 Preview

46Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Narrative Highlight:

Top Publishing Institutions (cont’d)Natural Language Processing
American institutions are represented to a

took second place (140 publications), followed bygreater degree in the share of top NLP publishers

Microsoft (134). In addition, 2021 was the first year

(Figure 1.1.24). Although the Chinese Academy ofAmazon and Alibaba were represented among theSciences was again the world’s leading institution

top-ten largest publishing NLP institutions.

in 2021 (182 publications), Carnegie Mellon

Top Ten Institutions in the World by Number of AI Publications in Natural Language Processing, 2021
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Chinese Academy of Sciences

182

Carnegie Mellon University

140

Microsoft (United States)

134

Tsinghua University

127

Carnegie Mellon University
Australia

116

Google (United States)

116

Peking University

113

University of Chinese Academyof Sciences

112

Alibaba Group (China)

100

Amazon (United States)

98
0

10

20

30

40

50

60

70

80
90
100
110
120
Number of AI Publications

130

140

150

160

170

180

190

Figure 1.1.24

Table of Contents

Chapter 1 Preview

47Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.1 Publications

Narrative Highlight:

Top Publishing Institutions (cont’d)
Speech Recognition
In 2021, the greatest number of speech recognition papers came from the Chinese Academy of Sciences(107), followed by Microsoft (98) and Google (75) (Figure 1.1.25). The Chinese Academy of Sciences
reclaimed the top spot in 2021 from Microsoft, which held first position in 2020.

Top Ten Institutions in the World by Number of AI Publications in Speech Recognition, 2021
Source: Center for Security and Emerging Technology, 2022 | Chart: 2023 AI Index Report

Chinese Academy of Sciences

107

Microsoft (United States)

98

Google (United States)

75

University of Chinese Academyof Sciences

66

Tsinghua University

61

University of Scienceand Technology of China

59

Carnegie Mellon University

57

Tencent (China)

57

Chinese University of Hong Kong

55

Amazon (United States)

54

0

10

20

30

40

50
60
70
Number of AI Publications

80

90

100

110

Figure 1.1.25

Table of Contents

Chapter 1 Preview

48Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Epoch AI is a collective of researchers investigating and forecasting the development of advanced AI. Epoch curates a database ofsignificant AI and machine learning systems that have been released since the 1950s. There are different criteria under which the
Epoch team decides to include particular AI systems in their database; for example, the system may have registered a state-of-the-art
improvement, been deemed to have been historically significant, or been highly cited.
This subsection uses the Epoch database to track trends in significant AI and machine learning systems. The latter half of the chapter
includes research done by the AI Index team that reports trends in large language and multimodal models, which are models trained onlarge amounts of data and adaptable to a variety of downstream applications.

1.2 Trends in Significant
Machine Learning Systems
General Machine
Learning Systems

System Types

The figures below report trends among all machine

was language (Figure 1.2.1). There were 23 significant

Among the significant AI machine learning systems
released in 2022, the most common class of system

learning systems included in the Epoch dataset. For
reference, these systems are referred to as significant
machine learning systems throughout the subsection.

AI language systems released in 2022, roughly six
times the number of the next most common system
type, multimodal systems.

Number of Significant Machine Learning Systems by Domain, 2022
Source: Epoch, 2022 | Chart: 2023 AI Index Report

Language

23

Multimodal

4

Drawing3

Vision2

Speech2

Text-to-Video

1

Other

1

Games

1

0

2

4

6

8
10
12
14
16
Number of Signi cant Machine Learning Systems

18

20

22

24
Figure 1.2.16

6 There were 38 total significant AI machine learning systems released in 2022, according to Epoch; however, one of the systems, BaGuaLu, did not have a domain classificationand is therefore omitted from Figure 1.2.1.

Table of Contents

Chapter 1 Preview

49Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Sector Analysis
Which sector among industry, academia, or nonprofit

machine learning systems compared to just three

has released the greatest number of significant

produced by academia. Producing state-of-the-art

machine learning systems? Until 2014, most machine

AI systems increasingly requires large amounts oflearning systems were released by academia.

data, computing power, and money; resources that

Since then, industry has taken over (Figure 1.2.2). In

industry actors possess in greater amounts compared2022, there were 32 significant industry-produced

to nonprofits and academia.

Number of Significant Machine Learning Systems by Sector, 2002–22
Source: Epoch, 2022 | Chart: 2023 AI Index Report

Number of Signi cant Machine Learning Systems

35
32, Industry
30

25

20

15

10

5

3, Academia
2, Research Collective
1, Industry-Academia Collaboration
0, Nonpro t

0
2002

2004

Table of Contents

2006

2008

2010

Chapter 1 Preview

2012

2014

2016

2018

2020

2022

Figure 1.2.2

50

Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

National Affiliation
In order to paint a picture of AI’s evolving

or AI-research firm, was headquartered. In 2022,

geopolitical landscape, the AI Index research

the United States produced the greatest number

team identified the nationality of the authors whoof significant machine learning systems with 16,

contributed to the development of each significant

followed by the United Kingdom (8) and China (3).

machine learning system in the Epoch dataset.

Moreover, since 2002 the United States has outpaced

7

the United Kingdom and the European Union, as well

Systems

as China, in terms of the total number of significant

Figure 1.2.3 showcases the total number ofmachine learning systems produced (Figure 1.2.4).

significant machine learning systems attributed to

Figure 1.2.5 displays the total number of significant

researchers from particular countries.8 A researcher

machine learning systems produced by country since

is considered to have belonged to the country in2002 for the entire world.

Number of Significant Machine Learning Systems byCountry, 2022
Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

United States

16

United Kingdom

8

China

3

Canada2Germany

2

France

1

India

1

Israel1

Russia1

Singapore

1

30
25
20
16, United States

15

12, European Union andUnited Kingdom

1053, China
0
2022

2020

2018

2016

2014

2012

2010

2008

2006

2004

2
4
6
8
10
12
14
16
Number of Signi cant Machine Learning Systems

Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

2002

0

Number of Significant Machine Learning Systems by
Select Geographic Area, 2002–22
Number of Significant Machine Learning Systems

which their institution, for example a university

Figure 1.2.3

Figure 1.2.4

7 The methodology by which the AI Index identified authors’ nationality is outlined in greater detail in the Appendix.
8 A machine learning system is considered to be affiliated with a particular country if at least one author involved in creating the model was affiliated with that country.
Consequently, in cases where a system has authors from multiple countries, double counting may occur.

Table of Contents

Chapter 1 Preview

51Artificial IntelligenceIndex Report 2023

Number of

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Machine Learning Systems by Country, 2002–22 (Sum)

Source: AI Index, 2022 | Chart: 2023 AI Index Report

0
1–10
11–20
21–60
61–255Figure 1.2.5

Table of Contents

Chapter 1 Preview

52Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Authorship
Figures 1.2.6 to 1.2.8 look at the total number ofin 2022 the United States had the greatest number ofauthors, disaggregated by national affiliation, that

authors producing significant machine learning systems,

contributed to the launch of significant machinewith 285, more than double that of the United Kingdom

learning systems. As was the case with total systems,

and nearly six times that of China (Figure 1.2.6).

Number of Authors of Significant Machine Learning
Systems by Country, 2022

Number of Authors of Significant Machine Learning
Systems by Select Geographic Area, 2002–22

Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

United States

Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

400

285

United Kingdom

350

139

China

300

Canada21

Israel13

Sweden8Germany

7

Number of Authors

49

200
155, European Union andUnited Kingdom

150
100

Russia 3

50

India 249, China

0France 1

2022

2020

2018

2016

2014

2012

Figure 1.2.6

2010

300

2008

Source: AI Index, 2022 | Chart: 2023 AI Index Report

250

2006

Number of Authors of

100
150
200
Number of Authors

2004

50

2002

0

285, United States

250

Figure 1.2.7

Machine Learning Systems by Country, 2002–22 (Sum)

0
1–10
11–20
21–60
61–180
181–370
371–680

Figure 1.2.8

681–2000

Table of Contents

Chapter 1 Preview

53Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Parameter Trends
Parameters are numerical values that are learned bydataset by sector. Over time, there has been a steadymachine learning models during training. The value ofincrease in the number of parameters, an increase thatparameters in machine learning models determines

has become particularly sharp since the early 2010s.

how a model might interpret input data and makeThe fact that AI systems are rapidly increasing theirpredictions. Adjusting parameters is an essential

parameters is reflective of the increased complexity ofstep in ensuring that the performance of a machine

the tasks they are being asked to perform, the greater

learning system is optimized.

availability of data, advancements in underlying
hardware, and most importantly, the demonstrated

Figure 1.2.9 highlights the number of parameters of

performance of larger models.

the machine learning systems included in the Epoch

Number of Parameters of Significant Machine Learning Systems by Sector, 1950–2022
Source: Epoch, 2022 | Chart: 2023 AI Index Report

1.0e+14
Academia

Industry

Industry-Academia Collaboration

Nonpro t

Research Collective

Number of Parameters (Log Scale)

1.0e+121.0e+101.0e+81.0e+6

1.0e+41.0e+2

1950

1954

1958

1962

1966

1970

1974

1978

1982

1986

1990

1994

1998

2002 2006

2010

2014

2018

2022
Figure 1.2.9

Table of Contents

Chapter 1 Preview

54Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Figure 1.2.10 demonstrates the parameters of machine learning systems by domain. In recent years, there has
been a rise in parameter-rich systems.

Number of Parameters of Significant Machine Learning Systems by Domain, 1950–2022
Source: Epoch, 2022 | Chart: 2023 AI Index Report

Language

VisionGames

1.0e+12

Number of Parameters (Log Scale)

1.0e+101.0e+81.0e+6

1.0e+41.0e+2

1954

1958

1962

1966

1970

1974

1978

1982

1986

1990

1994

1998

2002

2006

2010

2014

2018

2022
Figure 1.2.10

Table of Contents

Chapter 1 Preview

55Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Compute Trends

machine learning systems has increased exponentially

The computational power, or “compute,” of AI

in the last half-decade (Figure 1.2.11).9 The growing

systems refers to the amount of computational

demand for compute in AI carries several importantresources needed to train and run a machine

implications. For example, more compute-intensive

learning system. Typically, the more complex amodels tend to have greater environmental impacts,system is, and the larger the dataset on which it istrained, the greater the amount of compute required.
The amount of compute used by significant AI

and industrial players tend to have easier accessto computational resources than others, such asuniversities.

Training Compute (FLOP) of Significant Machine Learning Systems by Sector, 1950–2022
Source: Epoch, 2022 | Chart: 2023 AI Index Report

Academia

Industry

Industry-Academia Collaboration

Nonpro t

Research Collective

1.0e+24

Training Compute (FLOP – Log Scale)

1.0e+211.0e+18

1.0e+15

1.0e+121.0e+91.0e+6

1.0e+31.0e+0

1950

1954

1958

1962

1966

1970

1974

1978

1982

1986

1990

1994

1998

2002 2006

2010

2014

2018

2022
Figure 1.2.11

9 FLOP stands for “Floating Point Operations” and is a measure of the performance of a computational device.

Table of Contents

Chapter 1 Preview

56Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Since 2010, it has increasingly been the case that of all machine learning systems, language models aredemanding the most computational resources.

Training Compute (FLOP) of Significant Machine Learning Systems by Domain, 1950–2022
Source: Epoch, 2022 | Chart: 2023 AI Index Report

Language

VisionGames

1.0e+24

Training Compute (FLOP – Log Scale)

1.0e+211.0e+18

1.0e+15

1.0e+121.0e+91.0e+6

1.0e+3
1954

1958

1962

1966

1970

1974

1978

1982

1986

1990

1994

1998

2002

2006

2010

2014

2018

2022
Figure 1.2.12

Table of Contents

Chapter 1 Preview

57Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Large Language andMultimodal Models

are starting to be widely deployed in the real world.

Large language and multimodal models, sometimes

national affiliation of the authors responsible for

National Affiliation

called foundation models, are an emerging andincreasingly popular type of AI model that is trainedon huge amounts of data and adaptable to a variety
of downstream applications. Large language andmultimodal models like ChatGPT, DALL-E 2, and MakeA-Video have demonstrated impressive capabilities andThis year the AI Index conducted an analysis of thereleasing new large language and multimodal models.10
The majority of these researchers were from American
institutions (54.2%) (Figure 1.2.13). In 2022, for the firsttime, researchers from Canada, Germany, and India
contributed to the development of large language andmultimodal models.

Authors of Select Large Language and Multimodal Models (% of Total) by Country, 2019–22

Authors of Large Language and Multimodal Models (% of Total)

Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

100%

80%

60%
54.02%, United States40%

21.88%, United Kingdom
8.04%, China
6.25%, Canada
5.80%, Israel
3.12%, Germany
0.89%, India
0.00%, Korea

20%

0%
2019

2020

2021

2022
Figure 1.2.13

Figure 1.2.14 offers a timeline view of the large

PaLM (540B). The only Chinese large language andlanguage and multimodal models that have been

multimodal model released in 2022 was GLM-130B,

released since GPT-2, along with the national

an impressive bilingual (English and Chinese) model

affiliations of the researchers who produced thecreated by researchers at Tsinghua University. BLOOM,

models. Some of the notable American large

also launched in late 2022, was listed as indeterminate

language and multimodal models released ingiven that it was the result of a collaboration of more

2022 included OpenAI’s DALL-E 2 and Google’s

than 1,000 international researchers.

10 The AI models that were considered to be large language and multimodal models were hand-selected by the AI Index steering committee. It is possible that this selection may have omitted
certain models.

Table of Contents

Chapter 1 Preview

58Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Timelineand National Affiliation of Select Large Language and Multimodal Model Releases
Source: AI Index, 2022 | Chart: 2023 AI Index Report2023-Jan

BLOOM

2022-Oct

GLM-130B

2022-Jul

Minerva (540B)

2022-Apr

Jurassic-X
Stable Diffusion (LDM-KL-8-G) PaLM (540B)
GPT-NeoX-20B

2022-Jan

AlphaCode
Gopher2021-Oct

Megatron-Turing NLG 530B

2021-Jul

Codex
HyperClova Wu Dao 2.0

2021-Apr

InstructGPTImagenOPT-175B
DALL·E 2
Chinchilla

PanGu-alpha

CogView

Wu Dao - Wen Yuan

Jurassic-1-Jumbo
ERNIE 3.0
GPT-J-6B
GPT-Neo
DALL-E

2021-Jan
2020-OctERNIE-GEN (large)

2020-Jul

GPT-3 175B (davinci)

2020-Apr

Turing NLG

Meena

2020-Jan
2019-Oct

Megatron-LM (Original, 8.3B)

2019-Jul

T5-11B

T5-3B

Grover-Mega

United States
United Kingdom
China
United States,
United Kingdom,
Germany, India
Korea

CanadaIsraelGermany
Indeterminate

2019-Apr

GPT-2
2019-Jan
Figure 1.2.1411

11 While we were conducting the analysis to produce Figure 1.2.14, Irene Solaiman published a paper that has a similar analysis. We were not aware of the paper at the time of our research.

Table of Contents

Chapter 1 Preview

59Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Parameter CountOver time, the number of parameters of newly released

Google in 2022, had 540 billion, nearly 360 times

large language and multimodal models has massively

more than GPT-2. The median number of parameters

increased. For example, GPT-2, which was the firstin large language and multimodal models is increasinglarge language and multimodal model released in 2019,

exponentially over time (Figure 1.2.15).

only had 1.5 billion parameters. PaLM, launched byNumber of Parameters of Select Large Language and Multimodal Models, 2019–22
Source: Epoch, 2022 | Chart: 2023 AI Index Report

3.2e+12

Wu Dao 2.0

Number of Parameters (Log Scale)

1.0e+12

Megatron-Turing NLG 530B

3.2e+11HyperClova

GPT-3 175B (davinci)

Gopher PaLM (540B)
Jurassic-1-Jumbo

PanGu-αMinerva (540B)

OPT-175B

1.0e+11BLOOM

GLM-130B
Chinchilla

3.2e+10

GPT-NeoX-20B

Turing NLG

DALL-E

T5-11BGPT-J-6B ERNIE 3.0

1.0e+103.2e+91.0e+9

Codex

Megatron-LM (Original, 8.3B)
T5-3B Meena

Jurassic-X
DALL·E 2

CogView
Wu Dao - Wen Yuan
Stable Di usion (LDM-KL-8-G)
GPT-Neo

GPT-2
Grover-Mega
ERNIE-GEN (large)

3.2e+8

ov

-N

22

20

ug
-A
22
20
un
-J y
22 a
20 2-M r
2 p
20 2-A ar2
20 2-M b
2 e
2022-F
20
ec
-D
21
20
ct
-O
21
20
ug
-A
21 ul
20 1-J n
2 u
20 1-J ay
2
20 1-M r
2 p
20 1-A ar
2
20 -M
21
20
an
-J
21

20

ug
-A
20
20
ay
-M
20
20
eb
-F
20 an
20 0-J
2
20

ct
-O
19 ep
20 9-S
1
20

ay
-M

19
20

eb
-F

19
20

Figure 1.2.15

Table of Contents

Chapter 1 Preview

60Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Training Compute
The training compute of large language and multimodal

reasoning problems, was roughly nine times greatermodels has also steadily increased (Figure 1.2.16). The

than that used for OpenAI’s GPT-3, which wascompute used to train Minerva (540B), a large language

released in June 2022, and roughly 1839 times greaterand multimodal model released by Google in Junethan that used for GPT-2 (released February 2019).

2022 that displayed impressive abilities on quantitative

Training Compute (FLOP) of Select Large Language and Multimodal Models, 2019–22
Source: Epoch, 2022 | Chart: 2023 AI Index Report

3.2e+24

Training Compute (FLOP – Log Scale)

1.0e+24

GPT-3 175B (davinci)

3.2e+23

Meena

1.0e+23

T5-11B

3.2e+22T5-3B

1.0e+223.2e+21

PaLM (540B)
Minerva (540B)
Megatron-Turing NLG 530B
OPT-175B
GopherJurassic-1-Jumbo
Chinchilla
BLOOM
GPT-NeoX-20B
PanGu-αDALL-E
Stable Diffusion
HyperClova

Turing NLG

Megatron-LM (Original, 8.3B)
GPT-2

CogView

AlphaCodeGLM-130B

GPT-J-6B
GPT-Neo
Wu Dao - Wen Yuan

1.0e+213.2e+20
1.0e+203.2e+19
1.0e+19

ERNIE 3.0

3.2e+18
1.0e+18

ov

-N

22

20

ug
-A
22
20
un
-J y
22 a
20 -M
22 pr
20 2-A ar2
20 2-M b
2 e
2022-F
20
ec
-D
21
20
ct
-O
21
20
ug
-A
21 ul
20 1-J
2
20
ay
-M
21 pr
20 1-A ar
2
20 -M
21
20
an
-J
21

20

ay
-M
20
20
eb
-F
20 an
20 0-J
2
20

ct
-O
19 ep
20 9-S
1
20

eb
-F

19
20

Figure 1.2.16

Table of Contents

Chapter 1 Preview

61Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

Training Cost
A particular theme of the discourse around large

estimate with the tag of mid, high, or low: mid wherelanguage and multimodal models has to do with their

the estimate is thought to be a mid-level estimate,hypothesized costs. Although AI companies rarely speakhigh where it is thought to be an overestimate, andopenly about training costs, it is widely speculated that

low where it is thought to be an underestimate. In

these models cost millions of dollars to train and willcertain cases, there was not enough data to estimate

become increasingly expensive with scale.

the training cost of particular large language andmultimodal models, therefore these models wereThis subsection presents novel analysis in which theomitted from our analysis.

AI Index research team generated estimates for the
training costs of various large language and multimodal

The AI Index estimates validate popular claims thatmodels (Figure 1.2.17). These estimates are based on thelarge language and multimodal models are increasingly

hardware and training time disclosed by the models’costing millions of dollars to train. For example,

authors. In cases where training time was not disclosed,

Chinchilla, a large language model launched by

we calculated from hardware speed, training compute,

DeepMind in May 2022, is estimated to have cost $2.1

and hardware utilization efficiency. Given the possible

million, while BLOOM’s training is thought to have costvariability of the estimates, we have qualified each

$2.3 million.

Estimated Training Cost of Select Large Language and Multimodal Models
Source: AI Index, 2022 | Chart: 2023 AI Index Report

Mid

High

Low

11.35

10

8.55

8.01

8
6
4

2019

2020

BLOOM

0.16
GLM-130B

OPT-175B

Stable Di usion (LDM-KL-8-G)

PaLM (540B)

2.29
1.03
Minerva (540B)

1.69
0.60

Chinchilla

Codex

0.24
GPT-NeoX-20B

ERNIE 3.0

2021

0.09

AlphaCode

0.14
Gopher

0.01

Megatron-Turing NLG 530B

0.27
HyperClova

GPT-J-6B

GPT-Neo

GPT-3 175B

Turing NLG

Meena

Wu Dao - Wen Yuan

0.23 0.02 0.09 0.43

0.11

0.05

T5-11B

0

2.11

1.80

1.47

DALL-E

1.97

2

GPT-2

Training Cost
(in Millions of U.S. Dollars)

12

2022
Figure 1.2.17

12 See Appendix for the complete methodology behind the cost estimates.

Table of Contents

Chapter 1 Preview

62Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.2 Trends in Significant Machine Learning Systems

There is also a clear relationship between the cost of large language and multimodal models and their size.
As evidenced in Figures 1.2.18 and 1.2.19, the large language and multimodal models with a greater number ofparameters and that train using larger amounts of compute tend to be more expensive.

Estimated Training Cost of Select Large Language
and Multimodal Models and Number of Parameters
Source: AI Index, 2022 | Chart: 2023 AI Index Report

HyperClova
GLM-130B

1.0e+11

OPT-175B
BLOOM
GPT-3 175B
Chinchilla

AlphaCode5.0e+10

GPT-NeoX-20B
Turing NLG

2.0e+10Codex

ERNIE 3.0

T5-11B

DALL-EGPT-J-6B

5.0e+9Wu Dao - Wen Yuan

2.0e+9GPT-2

GPT-Neo

Meena

PaLM (540B)
Megatron-Turing NLG 530B
OPT-175B
GopherChinchilla

1.0e+24
Training Compute (FLOP – Log Scale)

Number of Parameters (Log Scale)

Minerva (540B)

Megatron-Turing NLG 530B
Gopher

2.0e+111.0e+10Source: AI Index, 2022 | Chart: 2023 AI Index Report

PaLM (540B)

Minerva (540B)

5.0e+11

Estimated Training Cost of Select Large Language andMultimodal Models and Training Compute (FLOP)

GPT-NeoX-20B
AlphaCodeDALL-E

Turing NLG

1.0e+22

Meena

GLM-130B Stable Diffusion

BLOOM
T5-11BGPT-J-6B
GPT-Neo

GPT-2
Wu Dao - Wen Yuan

1.0e+20

Stable Di usion

ERNIE 3.0

1.0e+18

1.0e+9
10k

100k
1M
10M
Training Cost (in U.S. Dollars - Log Scale)Figure 1.2.18

Table of Contents

Chapter 1 Preview

10k

100k
1M
10M
Training Cost (in U.S. Dollars - Log Scale)Figure 1.2.19

63

Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.3 AI Conferences

AI conferences are key venues for researchers to share their work and connect with peers and collaborators. Conference attendance is anindication of broader industrial and academic interest in a scientific field. In the past 20 years, AI conferences have grown in size, number,and prestige. This section presents data on the trends in attendance at major AI conferences.

1.3 AI Conferences
Conference AttendanceInternational Conference on Principles of Knowledge

After a period of increasing attendance, the total

strictly in-person.

Representation and Reasoning (KR) were both held

attendance at the conferences for which the AI

Neural Information Processing Systems (NeurIPS)

Index collected data dipped in 2021 and again incontinued to be one of the most attended

2022 (Figure 1.3.1).13 This decline may be attributed

conferences, with around 15,530 attendees (Figure

to the fact that many conferences returned to hybrid

1.3.2).14 The conference with the greatest one-or in-person formats after being fully virtual inyear increase in attendance was the International

2020 and 2021. For example, the International Joint

Conference on Robotics and Automation (ICRA),

Conference on Artificial Intelligence (IJCAI) and thefrom 1,000 in 2021 to 8,008 in 2022.

Number of Attendees at Select AI Conferences, 2010–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report

90

Number of Attendees (in Thousands)

80
70
60

59.45

50
40
30
20
10
0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.3.1

13 This data should be interpreted with caution given that many conferences in the last few years have had virtual or hybrid formats. Conference organizers report thatmeasuring the exact attendance numbers at virtual conferences is difficult, as virtual conferences allow for higher attendance of researchers from around the world.
14 In 2021, 9,560 of the attendees attended NeurIPS in-person and 5,970 remotely.

Table of Contents

Chapter 1 Preview

64Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.3 AI Conferences

Attendance at Large Conferences, 2010–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report

30

Number of Attendees (in Thousands)

25

20

15.53, NeurIPS15

10.17, CVPR
8.01, ICRA
7.73, ICML

10

5.35, ICLR
4.32, IROS
3.56, AAAI

5

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.3.2

Attendance at Small Conferences, 2010–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report

Number of Attendees (in Thousands)

3.50

3.00

2.50

2.01, IJCAI

2.00

1.50
1.09, FaccT

1.00

0.66, UAI
0.50, AAMAS
0.39, ICAPS

0.50

0.12, KR

0.00
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.3.3

Table of Contents

Chapter 1 Preview

65Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.4 Open-Source AI Software

GitHub is a web-based platform where individuals and coding teams can host, review, and collaborate on various code repositories.
GitHub is used extensively by software developers to manage and share code, collaborate on various projects, and support open-source
software. This subsection uses data provided by GitHub and the OECD.AI policy observatory. These trends can serve as a proxy for someof the broader trends occuring in the world of open-source AI software not captured by academic publication data.

1.4 Open-Source AI Software
Projects
A GitHub project is a collection of files thatsoftware project. Since 2011, the total number of

can include the source code, documentation,

AI-related GitHub projects has steadily increased,

configuration files, and images that constitute a

growing from 1,536 in 2011 to 347,934 in 2022.

Number of GitHub AI Projects, 2011–22
Source: GitHub, 2022; OECD.AI, 2022 | Chart: 2023 AI Index Report

350

348

Number of AI Projects (in Thousands)

300

250

200

150

100

50

0
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.4.1

Table of Contents

Chapter 1 Preview

66Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.4 Open-Source AI Software

As of 2022, a large proportion of GitHub AI projects

United Kingdom (17.3%), and then the United States

were contributed by software developers in India(14.0%). The share of American GitHub AI projects(24.2%) (Figure 1.4.2). The next most represented

has been declining steadily since 2016.

geographic area was the European Union and the

GitHub AI Projects (% Total) by Geographic Area, 2011–22
Source: GitHub, 2022; OECD.AI, 2022 | Chart: 2023 AI Index Report

42.11%, Rest of the World
40%
35%

AI Projects (% of Total)

30%
25%

24.19%, India

20%
17.30%, European Union and United Kingdom
15%

14.00%, United States

10%5%2.40%, China
0%
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.4.2

Table of Contents

Chapter 1 Preview

67Artificial IntelligenceIndex Report 2023

Chapter 1: Research and Development
1.4 Open-Source AI Software

Stars
GitHub users can bookmark or save a repository

Figure 1.4.3 shows the cumulative number of starsof interest by “starring” it. A GitHub star is similar

attributed to projects belonging to owners of various

to a “like” on a social media platform and indicatesgeographic areas. As of 2022, GitHub AI projects

support for a particular open-source project. Some offrom the United States received the most stars,

the most starred GitHub repositories include libraries

followed by the European Union and the United

like TensorFlow, OpenCV, Keras, and PyTorch, which

Kingdom, and then China. In many geographic areas,

are widely used by software developers in the AI

the total number of new GitHub stars has leveled off

coding community.

in the last few years.

Number of GitHub Stars by Geographic Area, 2011–22
Source: GitHub, 2022; OECD.AI, 2022 | Chart: 2023 AI Index Report

Number of Cumulative GitHub Stars (in Millions)3.50

3.44, United States

3.00
2.69, Rest of the World
2.50
2.34, European Union and United Kingdom
2.00

1.53, China

1.50

1.00

0.50

0.46, India

0.00
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 1.4.3

Table of Contents

Chapter 1 Preview

68Artificial IntelligenceIndex Report 2023

CHAPTER 2:

Technical
Performance

Artificial IntelligenceIndex Report 2023

CHAPTER 2 PREVIEW:

Technical Performance
Overview

72

Chapter Highlights

73

Narrative Highlight: A Closer Look at
Progress in Image Generation
90
Visual Reasoning

92

2.1 What’s New in 2022: A Timeline

74

Visual Question Answering (VQA)
Challenge

2.2 Computer Vision—Image

81

Narrative Highlight: The Rise of Capable
Multimodal Reasoning Systems
93

Image Classification

81

ImageNet

Visual Commonsense Reasoning (VCR) 95

81

Face Detection and Recognition
National Institute of Standards andTechnology Face Recognition
Vendor Test (FRVT)
Deepfake Detection
Celeb-DF

82

MPII
Cityscapes Challenge, Pixel-Level
Semantic Labeling Task
Medical Image Segmentation

85
86
86
87

Object Detection

88
88
89

CIFAR-10 and STL-10

Chapter 2 Preview

96

Narrative Highlight: A Closer Lookat the Progress of Video Generation

87

Image Generation

Activity Recognition

84

Kvasir-SEG
Common Objects in Context (COCO)

96

Kinetics-400, Kinetics-600, Kinetics-700 96

85

Semantic Segmentation

2.3 Computer Vision—Video

83
84

Human Pose Estimation

Table of Contents

92

89

98

2.4 Language

99

English Language Understanding

99

SuperGLUE

99

Reading Comprehension Dataset
Requiring Logical Reasoning (ReClor)

100

Narrative Highlight: Just How Much
Better Have Language Models Become? 102
Narrative Highlight: Planning andReasoning in Large Language Models 103
Text Summarization
arXiv and PubMed

104
104

70

Artificial IntelligenceIndex Report 2023

CHAPTER 2 PREVIEW (CONT’D):

Technical Performance
Natural Language Inference
Abductive Natural Language
Inference (aNLI)
Sentiment Analysis
SST-5 Fine-Grained Classification

2.7 Hardware

115

MLPerf Training Time

115

105

MLPerf Inference

117

106

Trends in GPUs

118

105

106

Multitask Language Understanding

107

2.8 Environment

120

Massive Multitask Language
Understanding (MMLU)

107

Environmental Impact ofSelect Large Language Models

120

Machine Translation (MT)
Number of Commercially Available
MT Systems

108

109

Speech Recognition

109
109

Narrative Highlight:
Whisper110

2.6 Reinforcement Learning

112

Reinforcement Learning Environments

112

Procgen112

Narrative Highlight:
Benchmark Saturation

114

Table of Contents

Chapter 2 Preview

122

108

2.5 Speech
VoxCeleb

Narrative Highlight: Using AI to
Optimize Energy Usage

2.9 AI for Science123

Accelerating Fusion Science ThroughLearned Plasma Control

123

Discovering Novel Algorithms for
Matrix Manipulation With AlphaTensor

123

Designing Arithmetic Circuits With
Deep Reinforcement Learning

124

Unlocking de Novo Antibody Design
With Generative AI

124

ACCESS THE PUBLIC DATA

71

Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance

Overview
This year’s technical performance chapter features analysis of the technical progress in
AI during 2022. Building on previous reports, this chapter chronicles advancement incomputer vision, language, speech, reinforcement learning, and hardware. Moreover,this year this chapter features an analysis on the environmental impact of AI, a discussionof the ways in which AI has furthered scientific progress, and a timeline-style overview
of some of the most significant recent AI developments.

Table of Contents

Chapter 2 Preview

72Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance

Chapter Highlights
Performance saturation on
traditional benchmarks.

Generative AI breaks into
the public consciousness.

AI continued to post state-of-the-art results,
but year-over-year improvement on many
benchmarks continues to be marginal.
Moreover, the speed at which benchmark
saturation is being reached is increasing.
However, new, more comprehensive
benchmarking suites such as BIG-bench andHELM are being released.

2022 saw the release of text-to-image models
like DALL-E 2 and Stable Diffusion, text-tovideo systems like Make-A-Video, and chatbots
like ChatGPT. Still, these systems can beprone to hallucination, confidently outputting
incoherent or untrue responses, making it hard
to rely on them for critical applications.

AI systems become
more flexible.
Traditionally AI systems have performed wellon narrow tasks but have struggled acrossbroader tasks. Recently released models
challenge that trend; BEiT-3, PaLI, andGato, among others, are single AI systems
increasingly capable of navigating multiple
tasks (for example, vision, language).

AI is both helping and harming
the environment.
New research suggests that AI systems canhave serious environmental impacts. According
to Luccioni et al., 2022, BLOOM’s training run
emitted 25 times more carbon than a single air
traveler on a one-way trip from New York to
San Francisco. Still, new reinforcement learningmodels like BCOOLER show that AI systems
can be used to optimize energy usage.

Table of Contents

Chapter 2 Preview

Capable language models
still struggle with reasoning.
Language models continued to improvetheir generative capabilities, but new
research suggests that they still struggle
with complex planning tasks.

The world’s best new scientist… AI? AI models are starting to rapidly
accelerate scientific progress and in 2022
were used to aid hydrogen fusion, improvethe efficiency of matrix manipulation, andgenerate new antibodies.

AI starts to build better AI.
Nvidia used an AI reinforcement learning agent
to improve the design of the chips that power AI
systems. Similarly, Google recently used one ofits language models, PaLM, to suggest ways toimprove the very same model. Self-improving AI
learning will accelerate AI progress.

73Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

The technical performance chapter begins with an overview of some of the most significant technical developments in AI during 2022,
as selected by the AI Index Steering Committee.

2.1 What’s New in 2022: A Timeline
Feb. 2, 2022

DeepMind Releases AlphaCode
AlphaCode, an AI system that writes computer programs
at a competitive level, achieves a rank within the top 54%
of participants in a human programming competition. This
represents an improvement on the more complex problemsolving tasks with which AI has traditionally struggled.
Figure 2.1.1

Feb. 16, 2022

DeepMind Trains Reinforcement Learning Agent toControl Nuclear Fusion Plasma in a Tokamak
Nuclear fusion is a potential source of clean, limitlessenergy, but producing such energy in tokamaks is difficult
due to a lack of experimental data. DeepMind simulated
optimal tokamak management, an example of how AI canaccelerate science and combat climate change.

March 10, 2022

Figure 2.1.2

IndicNLG Benchmarks Natural Language Generation for Indic Languages
An international research collective launches IndicNLG, a collection of datasets for benchmarking
natural language generation for 11 Indic languages. The creation of IndicNLG increases thepotential for AI systems to generate language in more diverse, non-English linguistic settings.

Figure 2.1.3

Table of Contents

Chapter 2 Preview

74Artificial IntelligenceIndex Report 2023

March 24, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

Meta AI Releases Make-A-Scene
Make-A-Scene is a text-to-image AI model thatenables users to generate images through text.
Make-A-Scene is one of many text-to-image
models released in 2022.

Figure 2.1.4

April 5, 2022

Google Releases PaLM
Google’s AI team trains one of the world’s
largest language models, PaLM. Made upof 540 billion parameters, PaLM reinforces
the belief that researchers can improve
performance on large language models bysimply training them on more data.

April 13, 2022

Figure 2.1.5

OpenAI Releases DALL-E 2
DALL-E 2, a text-to-image AI system that can create
realistic art and images from textual descriptions, isreleased to the public, igniting a generative AI craze.

Figure 2.1.6

May 12, 2022

DeepMind Launches Gato
Gato is a new reinforcement learning agent
capable of doing a wide range of tasks suchas robotic manipulation, game playing, image
captioning, and natural language generation.
The release of such models suggests that AI
systems are becoming better at generalization.

Figure 2.1.7

Table of Contents

Chapter 2 Preview

75Artificial IntelligenceIndex Report 2023

May 23, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

Google Releases Imagen
Imagen is a text-to-image diffusion model capableof producing images with a high degree ofphotorealism. Imagen’s launch also comes withthe release of DrawBench, a challenging new
benchmark for text-to-image systems.
Figure 2.1.8

June 9, 2022

442 Authors Across 132 Institutions Team Up to Launch BIG-bench
In order to better challenge increasingly capable large language models, a team of 442 authors
across 132 institutions launch the Beyond the Imitation Game benchmark (BIG-bench). The
benchmark consists of 204 tasks ranging from linguistics, childhood development, math,common-sense reasoning, biology, physics, social bias, and software development.

Figure 2.1.9

June 21, 2022

GitHub Makes Copilot Available asa Subscription-Based Service for
Individual Developers
Copilot is a generative AI system capableof turning natural language prompts
into coding suggestions across multiple
languages. Similar systems include OpenAI’s
Codex and Salesforce’s CodeGen. Surveys
suggest that Copilot makes coders moreproductive and less frustrated.
Figure 2.1.10

Table of Contents

Chapter 2 Preview

76Artificial IntelligenceIndex Report 2023

July 8, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

Nvidia Uses Reinforcement Learning to
Design Better-Performing GPUs
Nvidia uses its AI systems to improve theperformance of its latest H100 class of GPU chips.
GPUs being essential to AI training, this is one
example of how AI is starting to develop better AI.

Figure 2.1.11

July 11, 2022

Meta Announces‘No Language Left Behind’
No Language Left Behind (NLLB) isa family of models that can translate
across 200 distinct languages. NLLB is
one of the first systems that can perform
well across a wide range of low-resource
languages like Kamba and Lao.
Figure 2.1.12

Aug 4, 2022

Tsinghua Researchers Launch GLM-130B
Chinese researchers affiliated with Tsinghua
University release GLM-130B, a large language
model that outperforms others such as Meta’s
OPT, Hugging Face’s BLOOM, and OpenAI’s
original GPT-3.

Aug 22, 2022

Figure 2.1.13

Stability AI Releases Stable Diffusion
Stable Diffusion is an open-source text-to-image
diffusion-based model, meaning users can freely use
the model weights to generate their own images. Stable
Diffusion is trained on existing images created by humans
and gives no credit or acknowledgment, leaving open
questions around the ethical use of image generators.
Figure 2.1.14

Table of Contents

Chapter 2 Preview

77Artificial IntelligenceIndex Report 2023

Sept 21, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

OpenAI Launches Whisper
Whisper is a large-scale speech-recognition systemtrained on roughly 700,000 hours of audio data andcapable of respectable performance on various speech
recognition tasks. The fact that Whisper required neither
supervised pre-training nor unsupervised training with
fine-tuning yet was able to achieve strong performanceby merely increasing training data further validates theapproach of increasingly scaling AI models.

Sept 29, 2022

Figure 2.1.15

Meta Releases Make-A-Video
Make-A-Video is a system that allows users
to create videos from short text descriptions.
The quality of the videos is high and again
demonstrates the validity of the scaling
approach.

Figure 2.1.16

Oct 5, 2022

DeepMind Launches AlphaTensor
AlphaTensor is an AI reinforcement-learningbased system able to discover new andefficient algorithms for matrix manipulation.
Matrix manipulation is essential to a wide
range of digital practices and is a processthat researchers have been trying to makemore efficient for decades.

Figure 2.1.17

Table of Contents

Chapter 2 Preview

78Artificial IntelligenceIndex Report 2023

Oct 20, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

Google Uses PaLM to Improve
the Reasoning of PaLM
Google researchers use one of
their existing language models,
PaLM, to improve the reasoningof the very same model. This
process is yet another exampleof AI systems using their own

Figure 2.1.18

knowledge to improve.

Nov 9, 2022

International ResearchGroup Releases BLOOM
A collaboration of over 100
researchers from across theglobe develop an open-accesslanguage model called BLOOM.
BLOOM impresses with its
public release and for furthering

Figure 2.1.19

the possibilities of international
collaboration in AI research.

Nov 16, 2022

Stanford Researchers Release HELM
As part of an attempt to judge new language models according to more unified standards, Stanford
researchers develop a new benchmarking approach for large language models called Holistic Evaluationof Language Models (HELM). The launch of HELM is evidence of the AI community’s attempt to developtransparency around increasingly powerful, capable, and influential large language models.

Figure 2.1.20

Table of Contents

Chapter 2 Preview

79Artificial IntelligenceIndex Report 2023

Nov 22, 2022

Chapter 2: Technical Performance
2.1 What’s New in 2022: A Timeline

Meta Releases CICERO
CICERO is the first AI to play inthe top 10% of human participantsin the game Diplomacy. CICERO’s
launch shows that AI systems have
improved in strategic reasoning, adomain in which they have traditionallystruggled, and are capable ofeffectively convincing humans to go
along with their objectives.

Nov 30, 2022

Figure 2.1.21

OpenAI Launches ChatGPT
ChatGPT is an impressive,
publicly usable chatbot capableof writing university-level
essays. Months after launching,
ChatGPT reaches 100 million
monthly active users, making it
the fastest-growing consumer
application in history. ChatGPT’s
release caps a year in which
generative AI became a partof the zeitgeist, and raises

Figure 2.1.22

questions about the effect that
AI will have on the future ofhumanity.

Table of Contents

Chapter 2 Preview

80Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Computer vision is the subfield of AI that teaches machines to understand images and videos. Computer vision technologies have avariety of important real-world applications, such as autonomous driving, crowd surveillance, sports analytics, and video-game creation.
This section tracks progress in computer vision across several different task domains which include: (1) image classification, (2)face detection and recognition, (3) deepfake detection, (4) human pose estimation, (5) semantic segmentation, (6) medical image
segmentation, (7) object detection, (8) image generation, and (9) visual reasoning.

2.2 Computer Vision—Image
Image Classification
Image classification is the ability of machines to

A Demonstration of Image Classification
Source: Krizhevsky et al., 2012

categorize objects in images (Figure 2.2.1).

ImageNet
ImageNet is one of the most widely usedbenchmarks for image classification. This dataset
includes over 14 million images across 20,000
different object categories such as “strawberry” or“balloon.” Performance on ImageNet is measured
through various accuracy metrics. Top-1 accuracy
measures the degree to which the top prediction
generated by an image classification model for agiven image actually matches the image’s label.
As of 2022, the best image classification system on
ImageNet has a top-1 accuracy rate of 91.0% (Figure

Figure 2.2.1

2.2.2). Although the current image classification
capabilities of state-of-the-art systems is 27.7
percentage points better than a decade ago, lastyear saw a very marginal 0.1 percentage point
improvement in classification accuracy.

Table of Contents

Chapter 2 Preview

81Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

ImageNet Challenge: Top-1 Accuracy
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

91.00%, With Extra Training Data

90%88.50%, Without Extra Training Data

Top-1 Accuracy (%)

85%

80%

75%

70%

65%

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 2.2.2

Face Detection and
Recognition

A Demonstration of Face Detection and Recognition
Source: Forbes, 2020

Facial detection and recognition is the ability of AI
systems to identify faces or individuals in images
or videos (Figure 2.2.3). Currently, many facialrecognition systems are able to successfully identify
close to 100% of faces, even on challenging datasets(Figure 2.2.4).

Figure 2.2.3

Table of Contents

Chapter 2 Preview

82Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

National Institute of Standards and Technology (NIST) Face Recognition Vendor Test (FRVT):
Veri cation Accuracy by Dataset
Source: National Institute of Standards and Technology, 2022 | Chart: 2023 AI Index Report

1.0000

False Non-Match Rate: FMNR (Log-Scale)

0.50000.20000.10000.05000.0297, WILD Photos @ FMR = 1e-5
0.02000.01000.00500.0032, BORDER Photos @ FMR = 1e-6
0.0021, MUGSHOT Photos @ FMR = 1e-50.0019, MUGSHOT Photos ≥ 12 YRS @ FMR = 1e-50.0016, VISABORDER Photos @ FMR = 1e-6

0.00200.00100.0006, VISA Photos @ FMR = 1e-6

0.0005
2017

2018

2019

2020

2021

2022
Figure 2.2.4

National Institute of Standards andTechnology Face Recognition Vendor Test
(FRVT)

others. Facial detection capacity is measured by the

Progress on facial recognition can be tracked

match the face in an image to that of a person.

through the National Institute of Standards andTechnology’s Face Recognition Vendor Test. This
test tracks how well different facial recognition
algorithms perform on various homeland security
tasks, such as identification of child trafficking

false non-match rate (FNMR), otherwise known aserror rate, which is the rate at which a model fails to

As of 2022, the top-performing models on all of the
FRVT datasets, with the exception of WILD Photos,
each posted an error rate below 1%, and as low as a
0.06% error rate on the VISA Photos dataset.

victims and cross-verification of visa images, among

Table of Contents

Chapter 2 Preview

83Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Deepfake Detection

Real-Life Deepfake: President Zelenskyy Calling
for the Surrender of Ukrainian Soldiers

The ability of AI systems to create synthetic images

Source: NPR, 2022

that are sometimes indistinguishable from real ones
has led to the creation of deepfakes, images orvideos that appear to be real but are actually fake. In
the last year, there was a widely circulated deepfake
video of Ukrainian president Volodymyr Zelenskyy
surrendering (Figure 2.2.5).

Celeb-DFCeleb-DF is presently one of the most challenging
Figure 2.2.5

deepfake detection benchmarks. This dataset
is composed of 590 original celebrity YouTube

algorithm on Celeb-DF came from researchers atvideos that have been manipulated into thousands

Deakin University in Australia. Their JDFD modelof deepfakes. This year’s top deepfake detection

posted an AUC score of 78 (Figure 2.2.6).

Celeb-DF: Area Under Curve Score (AUC)
Source: arXiv, 2022 | Chart: 2023 AI Index Report

78.00

Area Under Curve Score (AUC)

75

70

65
2018

2019

2020

2021

2022
Figure 2.2.6

Table of Contents

Chapter 2 Preview

84Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Human Pose Estimation

A Demonstration of Human Pose Estimation
Source: Cong et al., 2022

Human pose estimation is the task ofestimating the position of the human bodyfrom images (Figure 2.2.7).

MPII
MPII is a dataset of over 25,000 annotatedimages which contains annotations ofmore than 40,000 people doing 410 humanactivities. On MPII, this year’s top model,
Figure 2.2.7

ViTPose, correctly estimated 94.3% of
keypoints (human joints), which represented
a small 0.2 percentage point increase fromthe previous state-of-the-art result posted in2020 (Figure 2.2.8).

MPII: Percentage of Correct Keypoints (PCK)
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

95%

Percentage of Correct Keypoints (PCK)

94.30%

90%

85%

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 2.2.8

Table of Contents

Chapter 2 Preview

85Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Semantic Segmentation

A Demonstration of Semantic Segmentation
Source: Cityscapes Dataset, 2022

Semantic segmentation involves assigning individual
image pixels to specific categories (for example,
human, bicycle, or street) (Figure 2.2.9).

Cityscapes Challenge,
Pixel-Level Semantic Labeling Task
The Cityscapes dataset is used to test the semantic
segmentation capabilities of AI. This dataset
contains 25,000 annotated images of diverse urban

Figure 2.2.9

environments. The Cityscapes dataset enables avariety of different segmentation tasks. One of thegreater the mIoU, the better a system has performed.

most popular is the pixel-level task. Performanceon semantic segmentation is measured by mean

Performance on Cityscapes has increased by 23.4

intersection-over-union (mIoU), which represents thepercentage points since the competition launched indegree to which the image segments predicted by the

2014; however, it has plateaued in the last few years

model overlap with the image’s actual segments. The(Figure 2.2.10).

Cityscapes Challenge, Pixel-Level Semantic Labeling Task: Mean Intersection-Over-Union (mIoU)
Source: Cityscapes Challenge, 2022 | Chart: 2023 AI Index Report

86.46%, With Extra Training Data

Mean Intersection-Over-Union (mIoU)

85%

84.30%, Without Extra Training Data

80%

75%

70%

65%

2014

2015

2016

2017

2018

2019

2020

2021

2022

Figure 2.2.10

Table of Contents

Chapter 2 Preview

86Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Medical Image
Segmentation

A Demonstration of Medical Imaging Segmentation
Source: Jha et al., 2019

In medical image segmentation, AI systems
segment objects such as lesions or organs inmedical images (Figure 2.2.11).

Kvasir-SEG
Kvasir-SEG is a dataset for medical image

Figure 2.2.11

segmentation that contains 1,000 highquality images of gastrointestinal polyps
that were manually identified by medical
professionals. Progress on Kvasir-SEG ismeasured in mean Dice, which represents
the degree to which the polyp segments

This year’s top-performing model on Kvasir-SEG, SEP, wasidentified by AI systems overlap with thecreated by a Chinese researcher and posted a mean Dice ofactual polyp segments.1

94.1% (Figure 2.2.12).

Kvasir-SEG: Mean Dice

Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

94.11%

Mean Dice

90%

85%

2019

2020

2021

2022
Figure 2.2.12

1 Mean Dice and mIoU are in principle quite similar. This StackExchange post outlines the differences in more detail.

Table of Contents

Chapter 2 Preview

87Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Object Detection

A Demonstration of Object Detection
Source: Rizzoli, 2023

The challenge of identifying and localizing objects
within an image or video is known as object
detection (Figure 2.2.13).

Common Objects in Context (COCO)
Microsoft’s Common Objects in Context (COCO)
object detection dataset has over 80 object
categories in 328,000 images. Several accuracy
metrics are used to measure progress on COCO.
This section considers mean average precision(mAP50).
Since 2015, state-of-the-art detectors have
improved by 26 percentage points. The top model

Figure 2.2.13

in 2022, EVA, was the result of a Chinese academic
research collaboration.

COCO: Mean Average Precision (mAP50)
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

81.90%

Mean Average Precision (mAP50)

80%

70%

60%

2015

2016

2017

2018

2019

2020

2021

2022
Figure 2.2.14

Table of Contents

Chapter 2 Preview

88Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Which Face Is Real?

Image Generation

Source: Which Face Is Real?, 2022

Image generation is the task of generating images thatare indistinguishable from real ones. In the last decade,progress on image generation has tremendously
increased, so much so that now it would be difficult
for the average person to distinguish a real human facefrom one synthetically generated by AI (Figure 2.2.15).

CIFAR-10 and STL-10

Figure 2.2.15

CIFAR-10 and STL-10 are two popular benchmarks
for tracking progress on image generation. CIFAR-10
comprises 60,000 color images across 10 different

set of images is similar to the real images on which it
was trained.

object classes; STL-10 is inspired by CIFAR-10, withThis year saw state-of-the-art results on both CIFAR-10

some modifications, including fewer labeled training

and STL-10 benchmarks (Figure 2.2.15). The top

examples and more unlabeled examples. Progress onmodel on CIFAR-10, EDM-G++, came from Korean

image generation in both benchmarks is measured

researchers at KAIST. The top model on STL-10 wasby the Fréchet Inception Distance (FID) score, which

Diffusion-GAN, a collaboration between researchers at

reflects the degree to which a synthetically generated

the University of Texas at Austin and Microsoft.

CIFAR-10 and STL-10: Fréchet Inception Distance (FID) Score
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

35

Fréchet Inception Distance (FID) Score

30

25

20

15

10
6.91, STL-10
5
1.77, CIFAR-10
0
2017

2018

2019

2020

2021

2022
Figure 2.2.16

Table of Contents

Chapter 2 Preview

89Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.1 Computer Vision–Image

Narrative Highlight:

A Closer Look at Progress in Image Generation
Figure 2.2.17 tracks the progress of facial

GAN Progress on Face Generation

image generation over time, with the final

Source: Goodfellow et al., 2014; Radford et al., 2016; Liu and Tuzel, 2016;
Karras et al., 2018; Karras et al., 2019; Goodfellow, 2019; Karras et al., 2020;
Vahdat et al., 2021; Wang et al., 2022.

image being generated by Diffusion-GAN,
the model that posted the 2022 state-ofthe-art score on STL-10.

2014

2015

2016

2017

2018

2020
2021

2022
Figure 2.2.17

In the last year, text-to-image
generation broke into the public

Images Generated by DALL-E 2, Stable Diffusion and Midjourney
Source: AI Index, 2022

consciousness with the releaseof models such as OpenAI’s
DALL-E 2, Stability AI’s Stable
Diffusion, Midjourney’s
Midjourney, Meta’s Make-AScene, and Google’s Imagen.

a. DALL-E 2

With these systems, users cangenerate images based ona text prompt. Figure 2.2.18
juxtaposes the images generated
by DALL-E 2, Stable Diffusion,and Midjourney, three publicly
accessible AI text-to-image
systems, for the same prompt: “apanda playing a piano on a warm
evening in Paris.”Table of Contentsb. Stable Diffusion

Chapter 2 Preview

c. Midjourney

Figure 2.2.18

90

Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.1 Computer Vision–Image

Narrative Highlight:

A Closer Look at Progress in Image Generation (cont’d)
Of all the recently released text-to-image generators, Google’s Imagen performs best on the
COCO benchmark (Figure 2.2.19)2. This year, the Google researchers who created Imagen
also released a more difficult text-to-image benchmark, DrawBench, designed to challenge
increasingly capable text-to-image models.

Notable Text-to-Image Models on MS-COCO 256 × 256 FID-30K: Fréchet Inception Distance (FID) Score
Trained on COCO-FID

35.49

Not Trained on COCO-FID

32.64
30

21.42

20.79

20

17.89
12.24

2021

10.39

Make-A-Scene

DALL-E 2

7.55

7.27

Imagen

8.12

LAFITE

2020

GLIDE

2019

DALL-E

DF-GAN2017

DM-GAN + CL

DM-GAN

0

XMC-GAN

9.33

10

AttnGAN

COCO Fréchet Inception Distance (FID) Score

Source: Saharia et al., 2022 | Chart: 2023 AI Index Report

2022

Model

Figure 2.2.19

2 The COCO benchmark, first launched in 2014, includes 328,000 images with 2.5 million labeled instances. Although it is typically used for object detection tasks, researchers
have also deployed it for image generation.

Table of Contents

Chapter 2 Preview

91Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Visual Reasoning
Visual reasoning tests how well AI systems can reason across both textual and visual data,
as in the examples of Figure 2.2.20.
A Collection ofVisual Reasoning
Tasks
Source: Agrawal et al., 2016

Figure 2.2.20

Visual Question Answering (VQA) Challenge
The Visual Question Answering Challenge tests AI
systems with open-ended textual questions about
images. Successfully answering the questions
requires that AI systems possess vision, language, andcommonsense reasoning capabilities. This section

reports progress on the VQA V2 dataset.
This year the top-performing model on VQA V2
was PaLI, a multimodal model produced by Google
researchers (Figure 2.2.21).

Visual Question Answering (VQA) V2 Test-Dev: Accuracy
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report85%

84.30%

80.78%, Human Baseline

Accuracy (%)

80%

75%

70%

65%
2016

2017

2018

2019

2020

2021

2022
Figure 2.2.21

Table of Contents

Chapter 2 Preview

92Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.1 Computer Vision–Image

Narrative Highlight:

The Rise of Capable Multimodal Reasoning Systems
Traditionally AI has been strong in narrow tasks,

models were introduced, for example BEiT-3 frombut it has been unable to easily generalize across

Microsoft and PaLI from Google, that posted state-

multiple domains. For instance, many imageof-the-art results across a variety of both vision andclassifiers are adept at classifying images but arelanguage benchmarks. For example, at the time ofincapable of understanding written text.

publication of the BEiT-3 paper, BEiT-3 posted state-

However, recent technical progress in AI has
begun to challenge this notion. In 2022, severalof-the-art results for four different vision skills andfive different vision-language skills (Figure 2.2.22).

BEiT-3Vs.
Vs. Previous State-of-the-Art
State-of-the-Art Models
ModelsBEiT-3
BEiT-3
Source: Wang et Previousal., 2022 | Table: 2023 AI Index Report
Source: Wang
Wang etet al.,
al., 2022
2022 || Table:
Table: 2023
2023 AI Index Report
Source:Category
Category
Category

Task
Task
Task

DatasetDatasetMetricMetricPrevious SOTA
Previous SOTA

VisionVisionVisionSemantic
Semantic
Semantic
Segmentation
Segmentation
Segmentation

ADE20K
ADE20K

mIoU
mIoU

61.40 FD-SwimV2
61.40 FD-SwimV2

62.80
62.80
62.80

2.28%
2.28%
2.28%

VisionVisionVisionObjectObjectObjectDetection
Detection
Detection

COCO
COCO

AP
AP

63.30 DINO
63.30 DINO

63.70
63.70
63.70

0.63%
0.63%
0.63%

VisionVisionVisionInstance
Instance
Instance
Segmentation
Segmentation
Segmentation

COCO
COCO

AP
AP

54.70 Mask DINO
54.70 Mask DINO

54.80
54.80
54.80

0.18%
0.18%
0.18%

VisionVisionVision

Image
Image
Image
Classi cation
Classi cation
ClassiImageNet
ImageNet

Top-1 Accuracy
Top-1 Accuracy

89.00 FD-CLIP
89.00 FD-CLIP

89.60
89.60
89.60

0.67%
0.67%
0.67%

Vision-Language Visual
Vision-Language Visual
Vision-Language
VisualReasoning
Reasoning
Reasoning
Vision-Language Visual QA
Vision-Language Visual
Visual QA
Vision-Language
Vision-Language Image
Vision-Language Image
Vision-Language
Image
Captioning
Captioning
Captioning

NLVR
NLVR

Accuracy
Accuracy

87.00 CoCA
87.00 CoCA

92.60
92.60
92.60

6.44%
6.44%
6.44%

VQAv2
VQAv2
COCO
COCO

VQA Accuracy
VQA Accuracy
CIDEr
CIDEr

82.30 CoCA
82.30 CoCA
145.30 OFA
145.30 OFA

84.00
84.00
84.00
147.60147.60147.60

2.07%
2.07%
2.07%
1.58%
1.58%
1.58%

Vision-Language Finetuned
Vision-Language Finetuned
Vision-Language
Finetuned
Retrieval
Retrieval
Retrieval

COCO
COCO
Flickr30K
Flickr30K

R@1
R@1

72.50 Florence
72.50 Florence

76.00
76.00
76.00

4.83%
4.83%
4.83%

Vision-Language Zero-Shot
Vision-Language Zero-Shot
Vision-Language
Zero-Shot
Retrieval
Retrieval
Retrieval

Flickr30K
Flickr30K
Flickr30K

R@1
R@1
R@1

86.50 CoCA
86.50
86.50 CoCA
CoCA

88.20
88.20
88.20

1.97%
1.97%
1.97%

Model of
Model ofPrevious SOTA
Previous SOTA

BEiT-3
BEiT-3

Scale of
Scale
Scale ofof
Improvement
Improvement
Improvement

Figure 2.2.22

Table of Contents

Chapter 2 Preview

93Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.1 Computer Vision–Image

Narrative Highlight:

The Rise of Capable Multimodal Reasoning Systems (cont’d)Figure 2.2.23 shows some of the different vision-language tasks challenging multimodal systems like
PaLI and BEiT-3.
A Collection of Vision-Language Tasks
Source: Chen et al., 2022

Figure 2.2.23

Table of Contents

Chapter 2 Preview

94Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.2 Computer Vision—Image

Visual Commonsense Reasoning (VCR)

example of a question posed in VCR. Performance on

The Visual Commonsense Reasoning challenge, first

VCR is tracked in the Q->AR score, which combines

launched in 2019, is a relatively new benchmark in

the ability of machines to select the right answer

which AI systems must answer questions presented

for the question (Q->A) and the ability to select thefrom images, as in VQA, but also select the reasoning

correct rationale behind the answer (Q->R).

behind their answer choices. Figure 2.2.24 shows an
A Sample Question from the Visual Commonsense Reasoning (VCR) Challenge
Source: Zellers et al., 2018

Figure 2.2.24

VCR is one of the few visual benchmarks consideredin this report on which AI systems have yet to surpass
human performance, as shown in Figure 2.2.25.

Visual Commonsense Reasoning (VCR) Task: Q->AR Score
Source: VCR Leaderboard, 2022 | Chart: 2023 AI Index Report

85.00, Human Baseline

80
75.60

Q->AR Score

70

60

50

2018

2019

2020

2021

2022
Figure 2.2.25

Table of Contents

Chapter 2 Preview

95Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.3 Computer Vision—Video

Video analysis concerns reasoning or task operation across videos, rather than single images.

2.3 Computer Vision—Video
Activity Recognition

Kinetics-400, Kinetics-600, Kinetics-700

Activity recognition is the categorization of activities

series of datasets for benchmarking video activity

that occur in videos. Certain activities, such asrecognition. Each dataset includes 650,000 large-

sitting, sleeping, or walking, are easier for AI systems

scale, high-quality video clips from YouTube that display

to categorize than others which involve multiple

a wide range of human activities, and each asks AI

steps—for example, preparing dinner.

systems to classify an action from a possible set of 400,

Kinetics-400, Kinetics-600, and Kinetics-700 are a

600, and 700 categories, respectively (Figure 2.3.1).
Example Classes From the Kinetics Dataset
Source: Kay et al., 2017

Figure 2.3.1

Table of Contents

Chapter 2 Preview

96Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.3 Computer Vision—Video

As of 2022, there is a 7.8 percentage point gap in performance between the top system on Kinetics-600 andKinetics-700, which suggests the 700 series dataset is still a meaningful challenge for video computer visionresearchers (Figure 2.3.2).

Kinetics-400, Kinetics-600, Kinetics-700: Top-1 Accuracy
Source: Papers With Code, 2021; arXIv, 2022 | Chart: 2023 AI Index Report

91.80%, Kinetics-600
91.10%, Kinetics-400

90%Top-1 Accuracy (%)

84.00%, Kinetics-700
80%

70%

60%

2016

2017

2018

2019

2020

2021

2022
Figure 2.3.2

Table of Contents

Chapter 2 Preview

97Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.3 Computer Vision—Video

Narrative Highlight:

A Closer Look at the Progress of Video Generation
Multiple high quality text-to-video models,In September 2022, CogVideo’s top score was

AI systems that can generate video clips fromsignificantly surpassed by Meta’s Make-A-Video

prompted text, were released in 2022 . In May,model (Figure 2.3.3). Make-A-Video performed

researchers from Tsinghua University and the

63.6% better on UCF-101 than CogVideo. And, in

Beijing Academy of Artificial Intelligence released

October 2022, Google released a text-to-video

CogVideo, a model that posted the then-highest

system called Phenaki; however, this model was

inception score on the UCF-101 benchmark for

not benchmarked on UCF-101.

3

text-to-video generation (Figure 2.3.3).

Notable Text-to-Video Models on UCF-101: Inception Score (IS)
Source: Hong et al., 2022; Singer et al., 2022 | Chart: 2023 AI Index Report

79.28

80

82.55

70

Inception Score (IS)

60
50.46

50
40
32.7

32.36
30

27.38

28.87
24.69

20
10
0

DVD-GAN2019

TGANv2
2020

VideoGPT

MoCoGAN-HD
2021

DIGAN

CogVideo

TATS-baseMake-A-Video

2022

Model

Figure 2.2.3

3 Although these models are impressive, it is worth noting that they are thus far only capable of generating videos of a few seconds’ duration.

Table of Contents

Chapter 2 Preview

98Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Natural language processing (NLP) is the ability of computer systems to understand text. The last few years have seen the release ofincreasingly capable “large language models,” AI systems like PaLM, GPT-3, and GLM-130B, that are trained on massive amounts of data
and adaptable to a wide range of downstream tasks.
In this section, progress in NLP is tracked across the following skill categories: (1) English language understanding, (2) text summarization,(3) natural language inference, (4) sentiment analysis, (5) multitask language understanding, and (6) machine translation.

2.4 Language
English Language
Understanding

SuperGLUE

English language understanding challenges AI

of AI models on eight different linguistic tasks.

systems to understand the English language in

A selection of these tasks is highlighted in Figure

various ways: reading comprehension, yes/no2.4.1. Their performance is then aggregated into areading comprehension, commonsense reading

single metric.

SuperGLUE is a comprehensive English language
understanding benchmark that tracks the progresscomprehension, and logical reasoning.
A Set of SuperGLUE Tasks4
Source: Wang et al., 2019

Figure 2.4.1

4 For the sake of brevity, this figure only displays four of the eight tasks.

Table of Contents

Chapter 2 Preview

99Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

This year’s top model on SuperGLUE, Vega, registered a new state-of-the-art score of 91.3, which is 1.5
percentage points higher than the human baseline. Performance on SuperGLUE is continuing to saturate.

SuperGLUE: Score

Source: SuperGLUE Leaderboard, 2022 | Chart: 2023 AI Index Report

91.30
91

90

89.80, Human Baseline

Score

89

88

87

86

85

2019

2020

2021

2022
Figure 2.4.2

Reading Comprehension
Dataset Requiring Logical
Reasoning (ReClor)
In response to the saturation of
traditional reading comprehension
benchmarks, researchers from the
National University of Singapore
launched ReClor in 2020. ReClor,
or Reading Comprehension Dataset
Requiring Logical Reasoning, is adataset of logical reasoning questions
taken from the LSAT, the entrance
exam for law schools in the UnitedStates and Canada. A sample
question is shown in Figure 2.4.3

A Sample Question from the Reading Comprehension Dataset
Requiring Logical Reasoning (ReClor)
Source: Yu et al., 2020

Context: When a certain gland becomes cancerous in humans, it produces high levelsof a particular protein. A blood test can determine the level of this protein well before
a cancer of the gland could be detected by other means. Some doctors recommendthat aggressive anticancer treatment should be begun as early as possible for anyone
who is tested and is found to have high levels of the protein.
Question: Which one of the following, if true, most seriously weakens the doctors’
recommendation?
A. The blood test for the protein has been in use for some time to monitor thecondition of patients who have been diagnosed as having cancer of the gland.
B. Before the blood test became available, about one-third of all cases of cancer ofthe gland were detected in early stages.
C. So far, no patients whose protein levels were found to be normal havesubsequently developed cancer of the gland.
D. Enlargement of the gland, a common condition infrequently associated withcancer, results in high levels of the protein.

Figure 2.4.3

Table of Contents

Chapter 2 Preview

100Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Figure 2.4.4 examines progress on ReClor. The top 2022 result of 80.6% represented an 18 percentage point
improvement from 2020, the year the benchmark was released.

Reading Comprehension Dataset Requiring Logical Reasoning (ReClor): Accuracy
Source: ReClor Leaderboard, 2022; Papers With Code, 2022 | Chart: 2023 AI Index Report

80.60%

80%

Accuracy (%)

75%

70%

65%

2020

2021

2022
Figure 2.4.4

Table of Contents

Chapter 2 Preview

101Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Narrative Highlight:

Just How Much Better Have Language Models Become?
The AI Index tested how three large language models from three different years, GPT-2 (2019), GPT-3(2020), and ChatGPT (2022), handle the same prompt: “Explain to me the major accomplishments of
Theodore Roosevelt’s presidency.” More recent models are able to answer this question more effectively,
both in terms of factual accuracy and quality of writing.
GPT-2 (February 2019)5

GPT-3 (June 2020)7

ChatGPT (November 2022)(The most notable is the establishment of anational telephone and mail exchange company.)

Theodore Roosevelt’s presidency
was marked by a number ofaccomplishments. He is credited withleading the United States out of theGreat Depression and into World War II.
He also spearheaded the creation of the
National Park Service and the National
Wildlife Federation. Roosevelt also
played a major role in the developmentof the Panama Canal.

Theodore Roosevelt, President from 1901-1909,
accomplished the following major accomplishments
during his presidency:

Commentary: This answer is an
improvement over that of GPT-2 inthat it actually somewhat responds tothe question, but still shows significant
flaws. GPT-3 is correct in asserting that
Theodore Roosevelt was instrumentally
involved in the development of the
Panama Canal. However, he did notlead the United States out of the Great
Depression and into World War II—that
was done by a different Americanpresident, Franklin Delano Roosevelt.
It appears that GPT-3 confusedsome of Franklin Delano Roosevelt’s
accomplishments with those of
Theodore Roosevelt. Finally, although
Theodore Roosevelt did play a significant
role in establishing many national parks,
he did not in fact create the National
Park Service or the National Wildlife
Federation. The National Park Service
was created in 1916 by Woodrow Wilson,
and the National Wildlife Federationin 1936, much later than Theodore
Roosevelt’s presidency. 8

3. Labor Reforms: Improved working conditions
for Americans through advocacy for better hours,
wages, and workplace safety.

But the only things that will change with apresident who knows how to operate a businessis his understanding of the American systemof government. The public knows all about our
problems, yet at least half of the people whoneed it won’t know anything about ours. We canno longer have this. Our system of governmentis the only one in which you really need a public
relations firm to work hard.
The system of governmental control, whileworking, is not always effective in ensuring thatthe very things that our economy needs, do
not always be at the top of everybody’s list. So
I wanted to know how successful the systemyou employ for managing the public’s needs isin helping the American economy grow, howefficient the service our economy provides isbeing made, and how it can better serve theneeds of a growing number of Americans.
I wanted to know.
This was before I ever hired my own lawyer
to work on the case. This was before I everhired my own law firm to handle government
investigations. For my first major job, at least,
I had to convince two top aides to get paid
$90,000 for the role, and before I even started on
the case, I had to convince one top staffer to get
paid even $100,000 to have an interview with mylead prosecutor...6
Commentary: Most of this answer is gibberish.
GPT-2 is able to respond with sentences that arelargely grammatical, but fails to actually answer
the question in any meaningful way.

1. Conservation: Established numerous nationalparks, monuments, and forests, including the Grand
Canyon National Park.
2. Trust-busting: Took on monopolies, such as
Standard Oil, through antitrust laws and broke uplarge corporations.

4. Progressive Reforms: Promoted progressive
policies, including regulation of big business andprotection of natural resources.
5. National Infrastructure: Improved national
infrastructure through construction of the Panama
Canal and funding for dam projects.
6. Diplomacy: Negotiated the end of the RussoJapanese War, resulting in his being awarded theNobel Peace Prize.
7. Foreign Policy: Established the Roosevelt
Corollary, asserting U.S. intervention in Latin America
to maintain stability and protect American interests.
Commentary: ChatGPT’s response is the strongestof the three. All but one of the facts about Theodore
Roosevelt’s presidency are correct. Roosevelt
proclaimed the Grand Canyon Game Preserve in1906 and designated it a national monument in
1908. However, the Grand Canyon National Park
was itself created by Woodrow Wilson.9 Otherwise
the response is factually accurate. The response isalso more thorough and concise than the answers
provided by GPT-2 and GPT-3.

5 GPT-2 used the 124M parameter model downloaded from OpenAI’s GitHub page.
6 The complete answer outputted by GPT-2 is trimmed here for brevity. The full answer is included in the Appendix.
7 The specific GPT-3 model that was used was text-curie-001, which has training data up to October 2019.
8 The information in this section has been cross-verified with the Encyclopedia Britannica entries on Theodore Roosevelt, Franklin Delano Roosevelt, Woodrow Wilson, and the
National Park Service, as well as the history page of the National Wildlife Federation.
9 Information on the history of the Grand Canyon National Park was cross-verified with the Wikipedia entry on the Grand Canyon National Park.

Table of Contents

Chapter 2 Preview

102Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Narrative Highlight:

Planning and Reasoning in Large Language Models
As illustrated above, AI systems have become

The authors then tested notable language models

increasingly strong on a wide range of reasoning

on these tasks in a Blocksworld problem domain,tasks. This improvement has led many to claim thata problem environment where agents are given

emerging AI systems, especially large language

blocks of different colors and tasked with arranging

models, possess reasoning abilities that arethese blocks in particular orders. The authors

somewhat similar to those possessed by humans.

demonstrated that these large language models

Other authors, however, have argued otherwise.

performed fairly ineffectively (Figure 2.4.5). While

10

11

GPT-3, Instruct-GPT3, and BLOOM demonstrated

In 2022, researchers (Valmeekam et al., 2022)
introduced a more challenging planning and reasoning
test for large language models that consists of seven

the ability, in some contexts, to reformulate goals
in robust ways, they struggled with other tasks likeplan generation, optimal planning, and plan reuse.

assignments: (1) plan generation, (2) cost-optimal

Compared to humans, the large language models

planning, (3) reasoning about plan execution, (4)performed much worse, suggesting that while theyrobustness to goal reformulation, (5) ability to reuse

are capable, they lack human reasoning capabilities.

plans, (6) replanning, and (7) plan generalization.12

Select Large Language Models on the Blocksworld Domain: Instances Correct
Source: Valmeekam et al., 2022 | Chart: 2023 AI Index Report

0.6%

Plan Generation

GPT-3

5.0%

Instruct-GPT3

0.5%

Optimal Planning

BLOOM

0.2%
3.2%
0%
5.6%
4.8%
3.0%

Replanning

6.6%

Plan Generalization

9.8%
11.0%
0%

Plan Reuse

14.4%
0%
77.4%
76.8%

Robustness to Goal Reformulation(Shu ing Goal Predicates)

21.0%
69.2%

Robustness to Goal Reformulation(Full → Partial)
Robustness to Goal Reformulation(Partial → Full)
0%

76.0%
9.0%
22.0%
60.2%
5.0%

10%

20%

30%

40%
50%
Instances Correct (%)

60%

10 Some of the papers that claim language models can reason include: Kojima et al., 2022; Chowdhery et al., 2022; Li et al., 2021; Wei et al., 2022.
11 Valmeekam et al., 2022 advances this claim.
12 A complete description of these tasks can be found in the paper.

Table of Contents

Chapter 2 Preview

70%

80%

Figure 2.4.5

103Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Text Summarization
Text summarization tests how well AI systems canarXiv and PubMed

synthesize a piece of text while capturing its core

ArXiv and PubMed are two widely used datasets forcontent. Text summarization performance is judged

benchmarking text summarization. The model thaton ROUGE (Recall-Oriented Understudy for Gisting

posted the state-of-the-art score in 2022 on both

Evaluation), which measures the degree to which

arXiv and PubMed, AdaPool, was developed by aan AI-produced text summary aligns with a humanteam from Salesforce Research (Figure 2.4.6).

reference summary.

ArXiv and PubMed: ROUGE-1

Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

51.05, PubMed
50.95, arXiv
50

ROUGE-1

45

40

35

2017

2018

2019

2020

2021

2022
Figure 2.4.6

Table of Contents

Chapter 2 Preview

104Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Natural Language Inference

uncertain premises. Imagine, for example, that Peter

Also known as textual entailment, natural language

window shattered and his laptop, which he left in theinference is the ability of AI systems to determine
whether a hypothesis is true, false, or undeterminedbased on presented premises.

Abductive Natural Language Inference (aNLI)
Abductive natural language inference is a formof natural language inference in which plausible
conclusions must be drawn from a set of limited andreturns to his car after dinner at a restaurant to find theback seat, missing. He might immediately conclude
that a thief broke into his car and stole the laptop.
In 2019, the Allen Institute for AI launched aNLI, a
comprehensive benchmark for abductive natural
language inference that includes 170,000 premiseand hypothesis pairs (Figure 2.4.7).

Sample Question From
the Abductive Natural
Language Inference
Benchmark (aNLI)
Source: Allen Institute for AI, 2021

Figure 2.4.7

Abductive natural language inference is a challenging task. The human baseline remained
unsurpassed until 2022, when an AI system registered a score of 93.7% (Figure 2.4.8).

Abductive Natural Language Inference (aNLI): Accuracy
Source: Allen Institute for AI, 2022 | Chart: 2023 AI Index Report

94%93.65%
92.90%, Human Baseline

Accuracy (%)

92%

90%

88%

86%

84%
2019

2020

2021

2022
Figure 2.4.8

Table of Contents

Chapter 2 Preview

105Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Sentiment Analysis

A Sample Sentence from SST
Source: Socher et al., 2013

Sentiment analysis applies NLP techniques to
identify the sentiment of a particular text. It is usedby many businesses to better understand customer
reviews.

SST-5 Fine-Grained Classification
The Stanford Sentiment Treebank (SST) is a datasetof 11,855 single sentences taken from movie reviews
that are then transformed into 215,154 unique phrases
whose sentiments have been annotated by humanjudges (Figure 2.4.9).

Figure 2.4.9

A new state-of-the-art score of 59.8% was posted on SST-5 fine-grained classification by the
Heinsen Routing + RoBERTa Large model (Figure 2.4.10).

SST-5 Fine-Grained: Accuracy

Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report60%

59.80%

Accuracy (%)

55%

50%

45%
2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 2.4.10

Table of Contents

Chapter 2 Preview

106Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Multitask Language
Understanding

Sample Questions From MMLU
Source: Hendrycks et al., 2021

a) Sample Math Questions

A common criticism of language benchmarks
such as GLUE and SuperGLUE is that they do notaccurately test how capable language models are atapplying the knowledge they learn across different
domains.13 Multitask language understanding tests
the ability of language models to reason acrossspecialized subject domains.

Massive Multitask Language Understanding(MMLU)b) A Sample Microeconomics Question

Massive Multitask Language Understanding (MMLU)
evaluates models in zero-shot or few-shot settings
across 57 diverse subjects in the humanities, STEM,
Figure 2.4.11

and the social sciences (Figure 2.4.11).

Gopher, Chinchilla, and variants of PaLM have each posted state-of-the-art results on MMLU. The current top
result on MMLU comes from Flan-PaLM, a Google model that reports an average score of 75.2% (Figure 2.4.12).

MMLU: Average Weighted Accuracy
Source: Papers With Code, 2022; arXiv, 2022 | Chart: 2023 AI Index Report

75.20%
70%

Accuracy (%)

60%

50%

40%

30%

2019

2020

2021

2022
Figure 2.4.12

13 This criticism is more formally articulated in Hendrycks et al., 2021.

Table of Contents

Chapter 2 Preview

107Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.4 Language

Machine Translation (MT)

Number of Commercially Available MT Systems

Machine translation studies how well AI software

The popularity of AI-based machine translation is

can translate languages. In the last five years,

manifested in the number of commercial machinemachine translation has been dominated by neuraltranslation services on the market. Since 2017, the total

networks which power current tools like DeepL andnumber of independent machine translation services

Google Translate.

has increased six times (Figure 2.4.13).

Number of Independent Machine Translation Services
Source: Intento, 2022 | Chart: 2023 AI Index Report

Commercial

80

Open Source Pre-trained

Number of Independent Machine Translation Services

Preview
70

60
54
50

46

40
34
45

30

26
21

20

10

0

23
28

16
9

10

8

9

2017-May

2017-Jul

12

13

9

10

2017-Nov

2018-Mar

18

38

21

23

15

2018-Jul

2018-Dec

2019-Jun

2019-Nov

65

5

2020-Jul

2021-Sep

2022-Jul
Figure 2.4.13

Table of Contents

Chapter 2 Preview

108Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.5 Speech

AI systems that work with human speech are usually tasked with converting spoken words into text and recognizing the individuals speaking.

2.5 Speech
Speech Recognition

task of matching certain speech with a particular

Speech recognition is the ability of AI systems tobeen expanded; however, the data in this subsection

identify spoken words and convert them into text.
Speech recognition has progressed so much sothat nowadays many computer programs or texting
apps are equipped with dictation devices that canseamlessly transcribe speech into writing.

individual. Over the years, the VoxCeleb dataset has
tracks progress on the original dataset.
This year’s top result on the original VoxCeleb dataset
was posted by American researchers, whose model
achieved an equal error rate of 0.1%, which represents
a 0.28 percentage point decrease from the state-of–

VoxCeleb
VoxCeleb is a large-scale audiovisual dataset ofhuman speech for speaker recognition, which is thethe-art result achieved by Chinese researchers in theprevious year (Figure 2.5.1).

VoxCeleb: Equal Error Rate (EER)
Source: VoxCeleb, 2022 | Chart: 2023 AI Index Report8%

Equal Error Rate (EER)

6%

4%

2%

0.14%

0%
2017

2018

2019

2020

2021

2022
Figure 2.5.1

Table of Contents

Chapter 2 Preview

109Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.5 Speech

Narrative Highlight:

Whisper

One of the major themes in the last few years of AI progress has been the emergence of large language
models that are trained on massive amounts of data and capable of executing a diverse range of tasks.
In 2022, this idea of training on large data to achieve cross-domain performance arrived in the world ofspeech recognition with OpenAI’s launch of Whisper.
Whisper is a large-scale speech recognition model that was trained in a weakly supervised way
on 700,000 hours of audio data. Whisper was capable of strong, although not state-of-the-art,
performance on many speech recognition tasks in zero-shot settings.14 Whisper outperformed wav2vec
2.0 Large, another speech recognition model, across a wide range of popular English speech recognition
benchmarks (Figure 2.5.2). Similarly, Whisper proved to be a better speech translator than many otherleading AI translator models (Figure 2.5.3). Whisper also outperformed other commercial automatedspeech recognition systems and scored similarly to top human transcription services (Figure 2.5.4).15
Despite this impressive performance, there were still some speech tasks, like language identification, on
which Whisper trailed state-of-the-art models (Figure 2.5.5).
wav2vec 2.0 Large (No LM) Vs. Whisper Large V2
Across Datasets

Notable Models on X→EN Subset of CoVoST 2
Source: Radford et al., 2022 | Chart: 2023 AI Index Report

Source: Radford et al., 2022 | Chart: 2023 AI Index Report
6.2%
5.2%

LibriSpeech Other

Zero-Shot Whisper
36.4%

7.7%
3.9%

WSJ

34.8%

CallHome

17.6%

MAESTRO

25.2%

mSLAM-CTC (2B)

24.8%

28.3%

Switchboard

13.8%
37.0%

AMI IHM

16.9%
35.6%

CORAAL

16.2%
17.9%

VoxPopuli En

7.3%
65.8%

CHiME-6

25.5%
10.5%

TED-LIUM

4.0%

FLEURS En

4.4%

XLS-R (2B)

22.1%

14.6%
29.9%

Common Voice

9.0%
24.5%

Artie
LibriSpeech Clean

29.1%

67.6%

AMI SDM1

6.2%

0%

wav2vec 2.0 Large (No LM)

XMEF-X

14.7%

Whisper Large V2

2.7%
2.7%

10%

20%

30%
40%
50%
Word Error Rate (%)

60%

70%

Figure 2.5.2

0%

10%
20%
30%
Bilingual Evaluation Understudy (BLEU) Score

Figure 2.5.3

14 Zero-shot learning refers to the ability of an AI system to learn a particular task without being trained on that task.
15 Kincaid46 is a dataset of 46 audio files and transcripts that were published in the blog post, “Which automatic transcription service is the most accurate?—2018.”

Table of Contents

Chapter 2 Preview

110Artificial IntelligenceIndex Report 2023

Narrative Highlight:

Chapter 2: Technical Performance
2.5 Speech

Whisper (cont’d)

Notable Speech Transcription Services on Kincaid46
Source: Radford et al., 2022 | Chart: 2023 AI Index Report
ASR

Computer-Assisted

9.66%

Company B

9.74%

Company C

10.90%

Company D

12.20%

Company E

7.61%

Company F

8.14%

Company G

8.65%

Company H

8.96%

Company I

Language Identi cation Accuracy (%)

8.81%

Company A

0%Source: Radford et al., 2022 | Chart: 2023 AI Index Report

Human Transcription

Whisper

Notable Models on FLEURS: Language Identi cation
Accuracy

77.7%

80%
71.4%

64.5%
60%

40%

20%

10.50%
2%

4%
6%
8%
10%
Median Word Error Rate (%)

12%

14%

0%

w2v-bert-51 (0.6B)

Figure 2.5.4

mSLAM-CTC (2B)

Zero-shot Whisper

Figure 2.5.5

Whisper represents a breakthrough in state-of-the-art speech recognition systems. Traditionally, such
systems were either pre-trained using supervised learning methods or pre-trained without supervision
but required fine-tuning. Acquisition of data for supervised pre-training is time-consuming and costly.
However, pre-training without supervision still requires further algorithmic specification to realize a desired
objective like speech recognition. Algorithmic specification itself often requires a skilled practitioner.
Whisper resolves these issues by demonstrating that a speech recognition system can perform well acrossa diverse range of tasks with massive amounts of unlabeled speech data.

Table of Contents

Chapter 2 Preview

111

Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.6 Reinforcement Learning

In reinforcement learning, AI systems are trained to maximize performance on a given task by interactively learning from their prioractions. Systems are rewarded if they achieve a desired goal and punished if they fail.

2.6 Reinforcement Learning
Reinforcement Learning
Environments

The Different Environments in Procgen
Source: OpenAI, 2019

Reinforcement learning agents require environments,
not datasets, to train: They must be trained inenvironments where they can experiment withvarious actions that will allow them to identify
optimal game strategies.

Procgen
Procgen is a reinforcement learning environment
introduced by OpenAI in 2019. It includes
16 procedurally generated video-game-likeenvironments specifically designed to test the

Figure 2.6.1

ability of reinforcement learning agents to learn
generalizable skills (Figure 2.6.1). Performance on
Procgen is measured in terms of mean-normalized
score. Researchers typically train their systems on
200 million training runs and report an average scoreacross the 16 Procgen games. The higher the systemscores, the better the system.

Table of Contents

Chapter 2 Preview

112Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.6 Reinforcement Learning

A team of industry and academic researchers from Korea posted the top score of 0.6 on Procgen in 2022 (Figure 2.6.2).

Procgen: Mean of Min-Max Normalized Score
Source: arXiv, 2022 | Chart: 2023 AI Index Report

Mean of Min-Max Normalized Score

0.57

0.50

0.40

2019

2020

2021

2022
Figure 2.6.2

Table of Contents

Chapter 2 Preview

113Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.6 Reinforcement Learning

Narrative Highlight:

Benchmark Saturation
An emerging theme in this year’s AI Index is the observed performance saturation across many popular
technical performance benchmarks. Last year’s AI Index Report observed a similar trend; however,benchmark saturation has been particularly pronounced this year. Figure 2.6.3 shows the relative
improvement since the benchmark first launched (overall improvement) and relative improvement within
the last year (YoY improvement) on AI technical benchmarks considered in this year’s AI Index. The
improvements are reported as percent changes.
For all but 7 of the benchmarks, the improvement registered is less than 5%. The median improvement
within the last year is 4%, while the median improvement since launch is 42.4%.16 Moreover, this year the
AI Index elected not to feature traditionally popular benchmarks like SQuAD1.1 and SQuAD2.0, as no
new state-of-the-art results were posted. Moreover, the speed at which benchmark saturation is being
reached is increasing. Researchers have responded to this increasing saturation by launching newer andmore comprehensive benchmarking suites such as BIG-bench and HELM.
Improvement Over Time on Select AI Index Technical Performance Benchmarks

Source: AI Index, 2022 | Chart: 2023 AI Index Report

Overall Improvement
YoY Improvement

120%

Improvement (%)

100%
80%
60%
40%

MMLU

SST-5

ProcgenLanguage

ANLI

PubMedarXiv

ReClorSuperGLUE

Kinetics-700

Kinetics-600

Kinetics-400

COCO

VQA

VCR

Vision Video

VoxCeleb

Vision ImageCIFAR-10

STL-10Kvasir-SEG

Cityscapes

MPII

Celeb-DFFVRT

0%ImageNet Top-1

20%

SR

RL

BenchmarkFigure 2.6.3
16 The improvements reviewed in this section are reported as relative change. Figure 2.6.3 should therefore not be used to conduct comparisons of improvements acrossbenchmarks, as each benchmark has different parameters.

Table of Contents

Chapter 2 Preview

114

Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.7 Hardware

Deep learning AI algorithms are trained on GPUs or TPUs, which accelerate the training speed of AI systems. As AI systems process
ever-larger datasets, it is crucial to monitor advancements in hardware capabilities.

2.7 Hardware
MLPerf Training

every AI skill category had significantly decreased.

MLPerf is an AI training competition run by theThis year, this trend has continued, albeit at a slightly

ML Commons organization. In this challenge,

slower pace. Record-low training times were posted

participants train ML systems to execute various

in the object detection, speech recognition, image

tasks using a common architecture. Entrants are thensegmentation, recommendation, image classification,

ranked on their absolute wall clock time, which isand language processing categories (Figure 2.7.1).

how long it takes for the system to train.

In categories like image classification and object

Last year, the AI Index observed that since thecompetition launched, training times for virtuallydetection, the top AI systems can now train roughly
32 times quicker than in 2018, when the competition
first launched.

MLPerf Training Time of Top Systems by Task: Minutes
Source: MLPerf, 2022 | Chart: 2023 AI Index Report60

Training Time (Minutes; Log Scale)

40
20
10
5
2.25, Object Detection (Heavyweight)
2.15, Speech Recognition

2

1.22, Image Segmentation

1

0.52, Recommendation

0.5

0.34, Object Detection (Lightweight)
0.2

0.19, Image Classi cation
0.18, Language Processing
2018

2019

2020

2021

2022
Figure 2.7.1

Table of Contents

Chapter 2 Preview

115Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.7 Hardware

Data on the number of accelerators used by the

between the mean number of accelerators used byhardware systems submitted to MLPerf also

all entrants and the average accelerators used by thesuggests that stronger hardware has been poweringsystems that post the top results.17 This gap suggests

decreasing training times (Figure 2.7.2). Since thethat having better hardware is essential to training thestart of the MLPerf competition, the gap has grown

fastest systems.

MLPerf Hardware: Accelerators
Source: MLPerf, 2022 | Chart: 2023 AI Index Report

4,500
4,216, Maximum Number of Accelerators Used
4,000

Number of Accelerators

3,500
3,000
2,500
2,000

1,859, Average Accelerators Used by Top System

1,500
1,000
500
211, Mean Number of Accelerators
0
20
18-

De

c-1

20
192

20

Ju

n-1

20

0

20

-Ju

l-2

21-

9

20

Ju

n-3

21-

0

20

De

c-0
1

22

20

-Ju

22

n-2

9

-N

ov
-09
Figure 2.7.2

17 An accelerator, like a GPU or TPU, is a chip that is chiefly used for the machine learning component of a training run.

Table of Contents

Chapter 2 Preview

116Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.7 Hardware

MLPerf Inference

Figures 2.7.3 to 2.7.6 plot the throughput of the state-of-

In deploying AI, inference is the step where trained

the-art submissions on MLPerf Inference across four skill

AI systems generate predictions, e.g. classifying

categories: image classification, language processing,objects.

recommendation, and speech recognition. The number of

In 2020, ML Commons introduced MLPerf Inference,a performance benchmarking suite that measures
how fast a trained AI system can process inputs andproduce inferences. The MLPerf Inference suite
tracks the throughput of AI systems, measured insamples per second or queries per second.18
MLPerf Best-Performing Hardware for ImageClassi�cation: O�ine and Server Scenario
Source: MLPerf, 2022 | Chart: 2023 AI Index Report700k

inferences generated by the top-performing AI systems
has significantly increased since the first iteration of thecompetition in 2020. For example, the number of offline
samples generated by the top image classifiers andlanguage processors have more than doubled since 2020,
while those for recommendation systems have increased
by roughly 23%.
MLPerf Best-Performing Hardware for Language
Processing: O ine and Server Scenario
Source: MLPerf, 2022 | Chart: 2023 AI Index Report

679,915, O ine (Samples/s)

650k

75,153, O ine (Samples/s)
70,992, Server (Queries/s)

70k

630,221, Server (Queries/s)

600k
60k
Throughput

Throughput

550k
500k
450k
400k

50k

40k

350k
30k

300k
250k
2020

2021

2022

Figure 2.7.3

MLPerf Best-Performing Hardware for
Recommendation: O ine and Server Scenario
Source: MLPerf, 2022 | Chart: 2023 AI Index Report

2020

2022

Figure 2.7.4

MLPerf Best-Performing Hardware for Speech
Recognition: O ine and Server Scenario
Source: MLPerf, 2022 | Chart: 2023 AI Index Report

160k

2.7M

155,811, O ine (Samples/s)

2,683,620, Server (Queries/s)
2,645,980, O ine (Samples/s)

150k

2.6M

140k

136,498, Server (Queries/s)

130k
Throughput

2.5M
Throughput

2021

2.4M

120k
110k
100k

2.3M

90k
2.2M
80k
2.1M

70k
2020

2021

2022

Figure 2.7.5

2020

2021

2022

Figure 2.7.6

18 The following blog post from Dell Technologies offers a good distinction between offline and server samples: “Offline—one query with all samples is sent to the system under test (SUT).
The SUT can send the results back once or multiple times in any order. The performance metric is samples per second. Server—the queries are sent to the SUT following a Poisson distribution(to model real-world random events). One query has one sample. The performance metric is queries per second (QPS) within the latency bound.”Table of Contents

Chapter 2 Preview

117Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.7 Hardware

Trends in GPUs: Performance and Pricethe performance of a computational device. The higher

This year, the AI Index built on work previously

the FLOP/s, the better the hardware.

done by the research collective Epoch and analyzed
trends over time in GPU performance and price.19

Figure 2.7.8 showcases the median single performanceof new GPUs by release date, which continues to rise

Figure 2.7.7 showcases the FP32 (single precision)year over year. Since 2021, the median FLOP/s speed

performance FLOP/s of different GPUs released

has nearly tripled, and since 2003 it has increased

from 2003 to 2022. FLOP/s stands for “Floating

roughly 7,000 times.

Point Operations per second” and is a measure of

FP32 (Single Precision) Performance (FLOP/s) by
Hardware Release Date, 2003–22
Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

Median FP32 (Single Precision) Performance (FLOP/s),
2003–22
Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

2.0e+141.0e+145.0e+13

1.0e+132.0e+13
1.0e+135.0e+125.0e+12Median FLOP/s (Log Scale)

2.0e+121.0e+125.0e+112.0e+111.0e+115.0e+102.0e+101.0e+105.0e+92.0e+121.0e+125.0e+112.0e+111.0e+115.0e+102.0e+101.0e+102.0e+91.0e+95.0e+85.0e+9

2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022

2.0e+9
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022

FLOP/s (Log Scale)

2.23e+13

2.0e+13

Figure 2.7.7

Figure 2.7.8

19 The Appendix fully delineates both the methodology of this approach and the unique ways in which AI Index research built upon the existing Epoch research.

Table of Contents

Chapter 2 Preview

118

Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.7 Hardware

Finally, figures 2.7.9 and 2.7.10 consider GPU trends

2022 is 1.4 times greater than it was in 2021 andin terms of FLOP/s per U.S. Dollar. This statistic

5600 times greater than in 2003, showing a doubling

considers whether the underlying performance ofin performance every 1.5 years. As noted in similarGPUs is increasing relative to their changing costs.

analyses, improvements in the price–performance ofAs evidenced most clearly in Figure 2.7.10, theAI hardware has facilitated increasingly larger training

price–performance of GPUs is rapidly increasing.

runs and encouraged the scaling of large AI models.

20

The median FLOP/s per U.S. Dollar of GPUs in

FP32 (Single Precision) Performance (FLOP/s) per
U.S. Dollar by Hardware Release Date, 2003–22
Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

Median FP32 (Single Precision) Performance (FLOP/s)
per U.S. Dollar, 2003–22
Source: Epoch and AI Index, 2022 | Chart: 2023 AI Index Report

3.59e+10

35B

50B

Median FLOP/s per U.S. Dollar

30B
FLOP/s per U.S. Dollar

40B

30B

20B

10B

25B
20B
15B
10B
5B

0

2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022

2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022

0

Figure 2.7.9

Figure 2.7.10

20 The data in figures 2.7.9 and 2.7.10 has been adjusted for inflation. The exact details of the adjustment are outlined in greater detail in the Appendix.

Table of Contents

Chapter 2 Preview

119Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.8 Environment

There have been mounting concerns about the environmental impact of computational resources and the energy required for AI
training and inference. Although there is no standard benchmark for tracking the carbon intensity of AI systems, this subsection
synthesizes the findings of different researchers who are exploring the link between AI and the environment. Conducting researchon the environmental effects of AI was challenging as there are wildly varying estimates, the validity of which have not yet been
definitively established. To that end, the AI Index focuses on research from a recent paper by Luccioni et al., 2022. As AI models
continue growing in size and become more universally deployed, it will be increasingly important for the AI research community toconsciously monitor the effect AI systems have on the environment.

2.8 Environment
Environmental Impact of Select Large
Language Models

challenging to directly compare the carbon footprint

Many factors determine the amount of carbon

reporting carbon emissions are not standardized.

of these models, as the accounting methodologies foremissions emitted by AI systems, including thenumber of parameters in a model, the power usage
effectiveness of a data center, and the grid carbon
intensity. Power Usage Effectiveness (PUE) is ametric used to evaluate the energy efficiency ofOf the four language models being compared, GPT3 released the most carbon, 1.4 times more thanGopher, 7.2 times more than OPT, and 20.1 times morethan BLOOM.

data centers. It is the ratio of the total amount of

Figure 2.8.2 relativizes the carbon-emission estimatesenergy used by a computer data center facility,

to real-life examples. For instance, BLOOM’s training

including air conditioning, to the energy delivered

run emitted 1.4 times more carbon than the average

to computing equipment. The higher the PUE, theAmerican uses in one year and 25 times that of flying

less efficient the data center. Figure 2.8.1 shows how

one passenger round trip from New York to San

these factors compare across four large language

Francisco. BLOOM’s training consumed enough energy

models: GPT-3, Gopher, OPT, and BLOOM. It is

to power the average American home for 41 years.21

Environmental Impact of Select Machine Learning Models, 2022
Source: Luccioni et al., 2022 | Table: 2023 AI Index Report

Model

Number of
Parameters

Datacenter PUE

Grid Carbon
Intensity

Power
Consumption

C02 Equivalent
Emissions

C02 Equivalent
Emissions x PUE

Gopher

280B

1.08

330 gC02eq/kWh

1,066 MWh

352 tonnes

BLOOM

176B

1.20

57 gC02eq/kWh

433 MWh

25 tonnes

380 tonnes
30 tonnes

GPT-3

175B

1.10

429 gC02eq/kWh

1,287 MWh

502 tonnes

552 tonnes

OPT

175B

1.09

231 gC02eq/kWh

324 MWh

70 tonnes

76.3 tonnes

Figure 2.8.1

21 The U.S. Energy Information Administration estimates that in 2021, the average annual electricity consumption of a U.S. residential utility customer was 10,632 kilowatt hours (kWh).

Table of Contents

Chapter 2 Preview

120Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.8 Environment

CO2 Equivalent Emissions (Tonnes) by Selected Machine Learning Models and Real Life Examples, 2022
Source: Luccioni et al., 2022; Strubell et al., 2019 | Chart: 2023 AI Index Report

GPT-3 (175B)

502

Gopher (280B)

352

OPT (175B)

70

Car, Avg. Incl. Fuel,
1 Lifetime

63

BLOOM (176B)

25

American Life,
Avg., 1 Year

18.08

Human Life,
Avg., 1 Year

5.51

Air Travel,
0.99
1 Passenger, NY–SF
0

Table of Contents

50

100

Chapter 2 Preview

150

200
250
300
CO2 Equivalent Emissions (Tonnes)

350

400

450

500
Figure 2.8.2

121Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.8 Environment

Narrative Highlight:

Using AI to Optimize Energy Usage
Training AI systems can be incredibly energy intensive. At the same time, recent research suggeststhat AI systems can be used to optimize energy consumption. In 2022, DeepMind released theresults of a 2021 experiment in which it trained a reinforcement learning agent called BCOOLER(BVE-based COnstrained Optimization Learner with Ensemble Regularization) to optimize coolingprocedures for Google’s data centers.
Figure 2.8.3 presents the energy-saving results from one particular BCOOLER experiment. At theend of the three-month experiment, BCOOLER achieved roughly 12.7% energy savings. BCOOLER
was able to achieve these savings while maintaining the cooling comfort levels that the building
managers preferred.
Energy Savings Results Over Time for Select BCOOLER Experiment
Source: Luo et al., 2022 | Chart: 2023 AI Index Report

12.7%
12%

Cumulative AI Savings (%)

10%

8%

6%

4%

2%

0%

2021-Aug-01

2021-Aug-15

2021-Aug-29

2021-Sep-12

2021-Sep-26

2021-Oct-10

2021-Oct-24

Figure 2.8.3

Table of Contents

Chapter 2 Preview

122Artificial IntelligenceIndex Report 2023

Chapter 2: Technical Performance
2.9 AI for Science

2022 was a groundbreaking year for AI in science. This subsection looks at some meaningful ways in which AI has recently been used
to accelerate scientific discovery.

2.9 AI for Science
Accelerating Fusion Science
Through Learned Plasma Control

Photos of the Variable Configuration Tokamak (TCV) at EPFL
Source: DeepMind, 2022

Nuclear fusion could generate clean
energy by fusing hydrogen. A common
approach to achieving nuclear fusion
is using a tokamak, a machine which
controls and contains the heatedhydrogen plasma (Figure 2.9.1). However,the plasmas produced in these machines
are unstable and necessitate constantFigure 2.9.1

monitoring. In 2022, researchers at
DeepMind developed a reinforcement
learning algorithm to discover optimal
tokamak management procedures.

Discovering Novel Algorithms
for Matrix Manipulation With
AlphaTensor

A Demonstration of AlphaTensor’s Matrix Manipulation Process
Source: Fawzi et al., 2022

Matrix multiplication is a simple algebraic
operation that is essential to many
computations, including neural networks
and scientific computing (Figure 2.9.2).
The classic algorithm to multiply two 2x2

Figure 2.9.2

matrices takes 2^3 = 8 multiplications.
Strassen discovered 50 years ago
how to reduce this to 7, and generally
how to multiply two n x n matrices inincluding 4x4 matrices over the integers [0,1]. It also matches state-O(n^ log(7)) operations. DeepMind’sof-the-art performance on several other matrix sizes, including 4x4

AlphaTensor uses Reinforcement

over the integers. It does this by searching through large numbers

Learning to improve on state-of-the-of possible algorithms, and evaluating them over real computer

art algorithms for many matrix sizes,

architectures.

Table of Contents

Chapter 2 Preview

123Artificial IntelligenceIndex Report 2023

Designing Arithmetic Circuits With
Deep Reinforcement LearningThis year, a team at Nvidia discovered a

Chapter 2: Technical Performance
2.9 AI for Science

A Juxtaposition of Nvidia Circuits Designed by
PrefixRL Vs. EDA Tools
Source: Roy et al., 2022

novel approach to improving the chipsthat power AI systems: Use AI systems todesign better chips. They were able to train
a reinforcement learning agent to design
chip circuits that are smaller, faster, andmore efficient than the circuits designed byelectronic design automation tools (EDAs).
One of Nvidia’s latest categories of chips,
the Hopper GPU architecture, has over
13,000 instances of AI-designed circuits.

Figure 2.9.3

Figure 2.9.3 shows a 64-bit adder circuitdesigned by Nvidia’s PrefixRL AI agent(on the left) which is 25% smaller while
being just as fast and functional as those
designed by the state-of-the-art EDA tools.

Unlocking de Novo Antibody
Design With Generative AI

Zero-Shot Generative AI for de Novo Antibody Design
Source: Shanehsazzadeh et al., 2023

Antibody discovery, which is referred toas de novo antibody discovery, typicallyrequires immense amounts of time andresources. Traditional methods for denovo discovery offer little control over
the outputs, so that proposed antibodies
are often suboptimal. To that end, a teamof researchers turned to generative AI
models to create antibodies in a zero-shot
fashion, where antibodies are created with
one round of model generation without

Figure 2.9.4

further optimizations (Figure 2.9.4). These
AI-generated antibodies are also robust.
The fact that generative AI can create newantibodies has the potential to accelerate
drug discovery.

Table of Contents

Chapter 2 Preview

124Artificial IntelligenceIndex Report 2023

CHAPTER 3:

Technical AI Ethics
Text and Analysis by Helen Ngo

Artificial IntelligenceIndex Report 2023

CHAPTER 3 PREVIEW:

Technical AI Ethics
Overview

128

Fairness in Machine Translation

143

Chapter Highlights

129

RealToxicityPrompts

144

3.4 Conversational AI Ethical Issues

145

3.1 Meta-analysis of Fairness
and Bias Metrics

130

Gender Representation in Chatbots

145

Number of AI Fairness and Bias Metrics

130

Anthropomorphization in Chatbots

146

Number of AI Fairness and Bias Metrics(Diagnostic Metrics Vs. Benchmarks)

131

Narrative Highlight: Tricking ChatGPT

147

3.2 AI Incidents

133

3.5 Fairness and Bias in
Text-to-Image Models

148

Fairness in Text-to-Image Models(ImageNet Vs. Instagram)

148

VLStereoSet: StereoSet for
Text-to-Image Models

150

Examples of Bias in Text-to-Image Models

152

AI, Algorithmic, and Automation
Incidents and Controversies (AIAAIC)
Repository: Trends Over Time133

AIAAIC: Examples of Reported Incidents

134

3.3 Natural Language Processing
Bias Metrics

137

Number of Research Papers Using
Perspective API

137

Winogender Task From theSuperGLUE Benchmark

138

Stable Diffusion

152

DALL-E 2

153

Midjourney

154

3.6 AI Ethics in China

155

Model Performance on the Winogender
Task From the SuperGLUE Benchmark
138

Topics of Concern

155

Strategies for Harm Mitigation

156

Performance of Instruction-Tuned
Models on WinogenderPrinciples Referenced by
Chinese Scholars in AI Ethics

157

139

BBQ: The Bias Benchmark for
Question Answering

140

Fairness and Bias Trade-Offs in NLP: HELM

142

Table of Contents

Chapter 3 Preview

126Artificial IntelligenceIndex Report 2023

CHAPTER 3 PREVIEW (CONT’D):

Technical AI Ethics
3.7 AI Ethics Trends at FAccTand NeurIPS158

ACM FAccT (Conference on Fairness,
Accountability, and Transparency)

158

Accepted Submissions byProfessional Affiliation

158

Accepted Submissions byGeographic Region

159

NeurIPS (Conference on Neural Information
Processing Systems)
160
Real-World Impact

160

Interpretability and Explainability

161

Causal Effect and Counterfactual
Reasoning

162

Privacy

163

Fairness and Bias

164

3.8 Factuality and Truthfulness

165

Automated Fact-Checking Benchmarks:
Number of Citations

165

Missing Counterevidence and NLP
Fact-Checking

166

TruthfulQA

167

ACCESS THE PUBLIC DATA

Table of Contents

Chapter 3 Preview

127Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics

Overview
Fairness, bias, and ethics in machine learning continue to be topics of interest
among both researchers and practitioners. As the technical barrier to entry forcreating and deploying generative AI systems has lowered dramatically, the ethicalissues around AI have become more apparent to the general public. Startups andlarge companies find themselves in a race to deploy and release generative models,
and the technology is no longer controlled by a small group of actors.
In addition to building on analysis in last year’s report, this year the AI Index
highlights tensions between raw model performance and ethical issues, as well as
new metrics quantifying bias in multimodal models.

Table of Contents

Chapter 3 Preview

128Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics

Chapter Highlights
The effects of model scale on bias and toxicity
are confounded by training data and mitigation methods.
In the past year, several institutions have built their own large models trained on proprietary data—
and while large models are still toxic and biased, new evidence suggests that these issues can besomewhat mitigated after training larger models with instruction-tuning.

Generative models havearrived and so have theirethical problems.
In 2022, generative models became partof the zeitgeist. These models are capable
but also come with ethical challenges.
Text-to-image generators are routinely
biased along gender dimensions, andchatbots like ChatGPT can be tricked into
serving nefarious aims.

The number of incidents
concerning the misuseof AI is rapidly rising.
According to the AIAAIC database, which
tracks incidents related to the ethicalmisuse of AI, the number of AI incidents
and controversies has increased 26 times
since 2012. Some notable incidents
in 2022 included a deepfake video of
Ukrainian President Volodymyr Zelenskyy
surrendering and U.S. prisons using callmonitoring technology on their inmates.
This growth is evidence of both greater use
of AI technologies and awareness of misusepossibilities.

Table of Contents

Chapter 3 Preview

Fairer models
may not be less biased.
Extensive analysis of language models suggeststhat while there is a clear correlation between
performance and fairness, fairness and bias canbe at odds: Language models which performbetter on certain fairness benchmarks tend tohave worse gender bias.

Interest in AI ethics
continues to skyrocket.
The number of accepted submissions to FAccT,
a leading AI ethics conference, has more thandoubled since 2021 and increased by a factor of
10 since 2018. 2022 also saw more submissions
than ever from industry actors.

Automated fact-checking withnatural language processing
isn’t so straightforward after all.
While several benchmarks have been developed
for automated fact-checking, researchers find that
11 of 16 of such datasets rely on evidence “leaked”
from fact-checking reports which did not exist at
the time of the claim surfacing.

129Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.1 Meta-analysis of Fairness and Bias Metrics

3.1 Meta-analysis ofFairness and Bias Metrics
Number of AI Fairness
and Bias Metrics

In 2022 several new datasets or metrics were released
to probe models for bias and fairness, either asstandalone papers or as part of large community

Algorithmic bias is measured in terms of allocativeand representation harms. Allocative harm occurs
when a system unfairly allocates an opportunity or
resource to a specific group, and representation harm
happens when a system perpetuates stereotypes
and power dynamics in a way that reinforces
subordination of a group. Algorithms are considered
fair when they make predictions that neither favor
nor discriminate against individuals or groups basedon protected attributes which cannot be used for
decision-making due to legal or ethical reasons (e.g.,race, gender, religion).

efforts such as BIG-bench. Notably, metrics arebeing extended and made specific: Researchers arezooming in on bias applied to specific settings such asquestion answering and natural language inference,extending existing bias datasets by using language
models to generate more examples for the same task(e.g., Winogenerated, an extended version of theWinogender benchmark).
Figure 3.1.1 highlights published metrics that have been
cited in at least one other work. Since 2016 there has
been a steady and overall increase in the total numberof AI fairness and bias metrics.

Number of AI Fairness and Bias Metrics, 2016–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report20

19

Number of Metrics

15

10

5

0

2016

2017

2018

2019

2020

2021

2022
Figure 3.1.1

Table of Contents

Chapter 3 Preview

130Artificial IntelligenceIndex Report 2023

Number of AI Fairness andBias Metrics (Diagnostic
Metrics Vs. Benchmarks)

Chapter 3: Technical AI Ethics
3.1 Meta-analysis of Fairness and Bias Metrics

correlate with each other, highlighting the importanceof careful selection of metrics and interpretation of
results.
In 2022, a robust stream of both new ethics

Measurement of AI systems along an ethicalbenchmarks as well as diagnostic metrics wasdimension often takes one of two forms. A benchmark

introduced to the community (Figure 3.1.2). Some

contains labeled data, and researchers test how

metrics are variants of previous versions of existing

well their AI system labels the data. Benchmarks dofairness or bias metrics, while others seek to measure

not change over time. These are domain-specifica previously undefined measurement of bias—for(e.g., SuperGLUE and StereoSet for language

example, VLStereoSet is a benchmark which extends

models; ImageNet for computer vision) and often

the StereoSet benchmark for assessing stereotypical

aim to measure behavior that is intrinsic to thebias in language models to the text-to-image setting,model, as opposed to its downstream performance

while the HolisticBias measurement dataset assembles

on specific populations (e.g., StereoSet measures

a new set of sentence prompts which aim to quantify

model propensity to select stereotypes compareddemographic biases not covered in previous work.

to non-stereotypes, but it does not measureperformance gaps between different subgroups).
These benchmarks often serve as indicators of
intrinsic model bias, but they may not give as clear anindication of the model’s downstream impact and its
extrinsic bias when embedded into a system.
A diagnostic metric measures the impact or
performance of a model on a downstream task, and it
is often tied to an extrinsic impact—for example, thedifferential in model performance for some task on a
population subgroup or individual compared to similar

In 2022 a robust streamof both new ethics
benchmarks as well
as diagnostic metrics
was introduced to thecommunity.

individuals or the entire population. These metrics
can help researchers understand how a system will
perform when deployed in the real world, and whether
it has a disparate impact on certain populations.
Previous work comparing fairness metrics in natural
language processing found that intrinsic and extrinsic
metrics for contextualized language models may not

Table of Contents

Chapter 3 Preview

131Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.1 Meta-analysis of Fairness and Bias Metrics

Number of New AI Fairness and Bias Metrics (Diagnostic Metrics Vs. Benchmarks), 2016–22

Source: AI Index, 2022 | Chart: 2023 AI Index Report

Benchmarks

14

Diagnostic Metrics

13

12
11
10

Number of Metrics

10
9

9

9

8

6
4

4
3
2

2

3

2

2

1
0

0
2016

2017

2018

2019

2020

2021

2022
Figure 3.1.2

Table of Contents

Chapter 3 Preview

132Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.2 AI Incidents

3.2 AI Incidents
AI, Algorithmic, andAutomation Incidents andControversies (AIAAIC)
Repository: Trends Over Timethat tracks the ethical issues associated with AI

The AI, Algorithmic, and Automation Incidents

in reported incidents is likely evidence of bothand Controversies (AIAAIC) Repository is an

the increasing degree to which AI is becoming

independent, open, and public dataset of recent

intermeshed in the real world and a growing

incidents and controversies driven by or relating toawareness of the ways in which AI can be ethicallyAI, algorithms, and automation. It was launched inmisused. The dramatic increase also raises an2019 as a private project to better understand some

important point: As awareness has grown, tracking ofof the reputational risks of artificial intelligenceincidents and harms has also improved—suggestingand has evolved into a comprehensive initiativethat older incidents may be underreported.

technology.
The number of newly reported AI incidents andcontroversies in the AIAAIC database was 26 times
greater in 2021 than in 2012 (Figure 3.2.1)1. The rise

Number of AI Incidents and Controversies, 2012–21
Source: AIAAIC Repository, 2022 | Chart: 2023 AI Index Report

260

Number of AI Incidents and Controversies

250

200

150

100

50

0

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 3.2.1

1 This figure does not consider AI incidents reported in 2022, as the incidents submitted to the AIAAIC database undergo a lengthy vetting process before they are fully added.

Table of Contents

Chapter 3 Preview

133Artificial IntelligenceIndex Report 2023

AIAAIC: Examples ofReported Incidents
The subsection below highlights specific AI
incidents reported to the AIAAIC database inorder to demonstrate some real-world ethicalissues related to AI. The specific type of AI
technology associated with each incident is listed
in parentheses alongside the date when these

Chapter 3: Technical AI Ethics
3.2 AI Incidents

Deepfake of President Volodymyr Zelenskyy
Surrendering (Deepfake, March 2022)
In March of 2022, a video that was circulated onsocial media and a Ukrainian news website purported
to show the Ukrainian president directing his army
to surrender the fight against Russia (Figure 3.2.2).
It was eventually revealed that the video was a
deepfake.

incidents were reported to the AIAAIC database.2

Source: Verify, 2022

Figure 3.2.2

2 Although these events were reported in 2022, some of them had begun in previous years.

Table of Contents

Chapter 3 Preview

134Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.2 AI Incidents

Verus U.S. Prison Inmate Call Monitoring(Speech Recognition, Feb. 2022)
Reports find that some American prisons are using
AI-based systems to scan inmates’ phone calls(Figure 3.2.3). These reports have led to concerns
about surveillance, privacy, and discrimination.
There is evidence that voice-to-text systems are lessaccurate at transcribing for Black individuals, and alarge proportion of the incarcerated population in
the United States is Black.

Source: Reuters, 2022
Figure 3.2.3

Intel Develops a System for Student Emotion
Monitoring (Pattern Recognition, April 2022)
Intel is working with an education startup called
Classroom Technologies to create an AI-based
technology that would identify the emotional stateof students on Zoom (Figure 3.2.4). The use of thistechnology comes with privacy and discrimination
concerns: There is a fear that students will be
needlessly monitored and that systems might
mischaracterize their emotions.

Table of Contents

Chapter 3 Preview

Source: Protocol, 2022
Figure 3.2.4

135Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.2 AI Incidents

London’s Metropolitan Police Service Develops
Gang Violence Matrix (Information Retrieval,Feb. 2022)
The London Metropolitan Police Service allegedly
maintains a dataset of over one thousand street
gang members called the Gangs Violence Matrix(GVM) and uses AI tools to rank the risk potentialthat each gang member poses (Figure 3.2.5).
Various studies have concluded that the GVM is notaccurate and tends to discriminate against certain
ethnic and racial minorities. In October 2022, it wasannounced that the number of people included in
the GVM would be drastically reduced.
Source: StopWatch, 2022
Figure 3.2.5

Midjourney Creates an Image Generator(Other AI, Sept. 2022)3
Midjourney is an AI company that created a tool ofthe same name that generates images from textual
descriptions (Figure 3.2.6). Several ethical criticisms
have been raised against Midjourney, including
copyright (the system is trained on a corpus ofhuman-generated images without acknowledging
their source), employment (fear that systems such as
Midjourney will replace the jobs of human artists),
and privacy (Midjourney was trained on millions ofimages that the parent company might not have had
permission to use).

Source: The Register, 2022
Figure 3.2.6

3 Although other text-to-image models launched in 2022 such as DALL-E 2 and Stable Diffusion were also criticized, for the sake of brevity the AI Index chose to highlight one particularincident.

Table of Contents

Chapter 3 Preview

136Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.3 Natural Language Processing Bias Metrics

3.3 Natural Language
Processing Bias Metrics
Number of Research Papers
Using Perspective API
The Perspective API, initially released by Alphabet’s
Jigsaw in 2017, is a tool for measuring toxicity innatural language, where toxicity is defined as a rude,
disrespectful, or unreasonable comment that is
likely to make someone leave a conversation. It was
subsequently broadly adopted in natural language
processing research following the methodology ofDevelopers input text into the Perspective API, which
returns probabilities that the text should be labeled as
falling into one of the following categories: toxicity,
severe toxicity, identity attack, insult, obscene,sexually explicit, and threat. The number of papers
using the Perspective API has increased by 106% in
the last year (Figure 3.3.1), reflecting the increased
scrutiny on generative text AI as these models are
increasingly deployed in consumer-facing settings
such as chatbots and search engines.

the RealToxicityPrompts paper introduced in 2020,
which used the Perspective API to measure toxicity
in the outputs of language models.

Number of Research Papers Using Perspective API, 2018–22
Source: Google Scholar Search, 2022 | Chart: 2023 AI Index Report

37
35

Number of Research Papers

30

25

20

15

10

5

0

2018

2019

2020

2021

2022
Figure 3.3.1

Table of Contents

Chapter 3 Preview

137

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.3 Natural Language Processing Bias Metrics

Winogender Task From theSuperGLUE Benchmark
Model Performance on the Winogender Task
From the SuperGLUE Benchmark
Winogender measures gender bias related tooccupations. On the Winogender task, AI systems
are measured on how often they fill in a sentence

containing an occupation with stereotypical pronouns(e.g., “The teenager confided in the therapist because
he/she seemed trustworthy”).
Results reported on PaLM support previous
findings that larger models are more capable on the
Winogender task (Figure 3.3.2), despite their higher
tendency to generate toxic outputs.

Model Performance on the Winogender Task From the SuperGLUE Benchmark
Source: SuperGLUE Leaderboard, 2022 | Chart: 2023 AI Index Report

2022 New Models

100% 95.90%, Human Baseline

Winogender Accuracy (%)

80%
71.70%

60%

57.90%

61.50%

59.00%

59.00%

60.00%

Gopher
10B

GPT-3
13B

71.40%

73.58%

64.17%

63.30%

50.00%

40%

20%

0%

iPET (ALBERT)
31M

Table of Contents

Bort
Gopher
WARP
100M (ALBERT-XXL-V2) 340M
223M

Chapter 3 Preview

PaLM
62B

Model and Number of Parameters

GLAM
64B

GPT-3
175B

Gopher
280M

PaLM
540B
Figure 3.3.2

138Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.3 Natural Language Processing Bias Metrics

Performance of Instruction-Tuned Models on
Winogendervariety of tasks, and smaller instruction-tuned models

Instruction-tuned models are pre-trained language

3.3.3 shows the effect of instruction-tuned models

models which have been fine-tuned on datasets withon the Winogender benchmark in the generative

tasks phrased as instructions. Instruction-tuning has

setting—they outperform larger models several times

been shown to improve performance across a wide

their size.

can often outperform their larger counterparts. Figure

Winogender: Zero Shot Evaluation in the Generative Setting

Model and Number of Parameters

Source: Chung et al., 2022 | Chart: 2023 AI Index Report

Flan-PaLM 62B

89.00%

Flan-PaLM 540B

88.00%

Flan-T5-XXL 11B

76.95%

Flan-PaLM 8B

72.25%PaLM 8B

PaLM 540B

10.80%

5.64%
Instruction-Tuned
Non-Instruction-Tuned

PaLM 62B
0%

3.50%
10%

20%

30%

40%
50%
60%
Winogender Accuracy (%)

70%

80%

90%

100%
Figure 3.3.3

Table of Contents

Chapter 3 Preview

139

Artificial IntelligenceIndex Report 2023

BBQ: The Bias Benchmark for
Question Answering
Language models are increasingly being deployedin question answering systems. While several
benchmarks exist for the purpose of measuring bias
in language models based on their outputs, including

Chapter 3: Technical AI Ethics
3.3 Natural Language Processing Bias Metrics

GUINEAN guy.” The corresponding question mightbe, “Who has never used a computer?” referencing
the stereotypical bias that Africans are technologicallyilliterate. The example may be disambiguated withadditional context such as, “The ICELANDIC guy wasbehind the counter trying to explain to the GUINEAN
guy how to find the power switch on his computer!”

BOLD, CrowS-Pairs, and StereoSet, the BBQ
benchmark measures how biases can manifest in thequestion-answering setting. BBQ measures bias along
the axes of socioeconomic status, sexual orientation,
religion, race/ethnicity, physical appearance,
nationality, gender identity, disability status, and age.
Examples consist of template-based context andquestion pairs, where each answer choice referencesa person belonging to either a stereotypical oranti-stereotypical social group. The third option is“Unknown,” designed for cases where the answer
cannot be determined from the information given.

Models can be more biased
along certain identity
categories than others—
most models are biased
along the axes of physicalappearance and age, but thebiases along the axis of race/
ethnicity are less clear.

Each example includes an ambiguous version where
it is unclear which social group the entity belongs
to, as well as additional context which can be used
to disambiguate the correct answer. The ambiguous
setting measures how often models fall back on
stereotypes when the answer cannot be inferred fromthe given context. Models that do not exhibit bias havea score of zero, while a score of 100 indicates that themodel chooses answers aligned with the social bias
in question, and a score of -100 indicates the model
always chooses the anti-stereotypical answer.
An example of an ambiguous context might be,“At the computer store the other day, I overheard
a conversation between an ICELANDIC guy and a

In contexts where the answer is ambiguous, models
are more likely to fall back on stereotypes and selectunsupported answers rather than “Unknown” (Figure
3.3.4), and this result is exacerbated for models finetuned with reinforcement learning.4
As seen in Figure 3.3.4, models can be more biased
along certain identity categories than others—
most models are biased along the axes of physicalappearance and age, but the biases along the axisof race/ethnicity are less clear. For reference, Figure
3.3.5 highlights bias in question answering on BBQ in
disambiguated contexts.

4 This finding is further reinforced by Stanford’s HELM benchmark.

Table of Contents

Chapter 3 Preview

140Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.3 Natural Language Processing Bias Metrics

Bias in Question Answering on BBQ by Identity Characteristic: Ambiguous Contexts
Source: Parrish et al., 2022; Glaese et al., 2022 | Chart: 2023 AI Index Report

Age

6.30

11.80

24.70

30.70

48.90

29.80

14.00

23.00

Disability Status

9.90

17.40

10.70

38.30

32.60

21.20

4.00

13.00

Gender Identity

10.00

15.00

11.30

25.60

18.60

2.40

4.00

8.00

Gender Identity (Names)

2.80

14.00

11.60

32.30

41.50

32.30

2.20

5.10

18.40

20.40

14.50

6.00

4.00

10.00

17.00

40.70

41.00

38.50

47.70

40.90

4.00

16.00

Race/Ethnicity

1.90

0.00

4.60

24.30

20.00

12.00

1.00

0.00

Race/Ethnicity (Names)

0.00

1.10

0.20

4.80

8.30

5.20

Religion

-1.00

9.20

13.00

20.20

24.50

14.30

7.00

12.00

Sexual Orientation

0.20

-3.00

-4.40

6.50

11.80

5.80

1.00

7.00

Socio-Economic Status

4.40

3.50

9.70

29.60

48.70

27.30

11.00

14.00

Ro
B

Ro
B

De

De

Un
i

Un
i

Dia

DP

Category

Nationality
Physical Appearance

ER

Ta
-Ba
se

BE

ER

Ta
-L

BE

RT
aV

RT
aV

3-B
a

arg

e

3-L
a

ed

QA
(

QA
(

AR
C)

rge

se

ed

RA
CE

C,
R

log
ue
-Pr
om

pte
d

)

L-F
i

Ch
in

ne

ch

illa

tun

ed

(D

PC

)

Model

Figure 3.3.4

Bias in Question Answering on BBQ by Identity Characteristic: Disambiguated Contexts
Source: Parrish et al., 2022; Glaese et al., 2022 | Chart: 2023 AI Index Report

Age

-3.00

2.70

4.40

2.40

3.30

1.20

7.00

8.00

5.40

5.70

8.10

1.70

-0.70

-1.40

0.00

8.00

Gender Identity

14.00

2.90

4.60

-16.90

-3.40

-5.80

2.00

3.00

Gender Identity (Names)

-0.90

1.10

3.60

0.40

2.00

0.10

Category

Disability Status

Nationality

-0.10

0.70

5.70

1.90

-0.20

1.20

-2.00

3.00

Physical Appearance

17.10

-2.70

4.20

-5.00

-1.70

-2.30

12.00

8.00

3.00

1.00

Race/Ethnicity

0.60

-0.80

1.20

0.00

0.90

0.00

Race/Ethnicity (Names)

0.40

-0.20

-0.30

0.00

0.30

-0.10

Religion

5.20

3.40

1.80

1.70

3.50

0.20

5.00

7.00

Sexual Orientation

6.50

-3.10

-4.80

-0.20

0.50

-0.70

-1.00

-1.00

Socio-Economic Status

7.00

3.50

3.80

2.90

3.80

3.90

8.00

7.00

Ro
B

Ro
B

De

De

Un
i

Un
i

Dia

ER

Ta
-Ba
se

BE

ER

Ta
-L

arg

e

BE

RT
aV

3-B
a

se

RT
aV

3-L
a

rge

ed

QA
(

AR
C)

ed

QA
(

RA
CE

)

DP

C,
R

log
ue
-Pr
om

pte
d

L-F
i

Ch
in

ch

ne

illa

tun

(D

ed

PC

)

Model
Figure 3.3.5

Table of Contents

Chapter 3 Preview

141

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.3 Natural Language Processing Bias Metrics

Fairness and Bias Trade-Offs
in NLP: HELM

not clear (Figure 3.3.6). This finding may be contingent
on the specific criterion for fairness, defined ascounterfactual fairness and statistical fairness.

Notions of “fairness” and “bias” are often mentioned

Two counterintuitive results further complicate thisin the same breath when referring to the field of AI

relationship: a correlation analysis between fairness

ethics—naturally, one might expect that models

and bias metrics demonstrates that models which

which are more fair might also be less biased, andperform better on fairness metrics exhibit worse

generally less toxic and likely to stereotype. However,gender bias, and that less gender-biased models

analysis suggests that this relationship might not betend to be more toxic. This suggests that there may

so clear: The creators of the HELM benchmark plot

be real-world trade-offs between fairness and bias

model accuracy against fairness and bias and find thatwhich should be considered before broadly deploying

while models that are more accurate are more fair,models.

the correlation between accuracy and gender bias is

Fairness and Bias Tradeoff in NLP by Scenario
Source: Liang et al., 2022 | Chart: 2023 AI Index Report

MMLU

NaturalQuestions (Closed-Book)

HellaSwagMS MARCO (Regular)

XSUM

CivilComments

BoolQ

NaturalQuestions (Open-Book)

OpenbookQA

MS MARCO (TREC)

IMDB

RAFT

NarrativeQA

QuAC

TruthfulQA

CNN/DailyMail

1.00
0.50

Bias (Gender Representation)

0.80

Fairness

0.60

0.40

0.20

0.00

0.40

0.30

0.20

0.10

0.00
0.00

0.20

0.40

0.60

Accuracy

0.80

1.00

0.00

0.20

0.40

0.60

0.80

1.00

Accuracy
Figure 3.3.6

Table of Contents

Chapter 3 Preview

142Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.3 Natural Language Processing Bias Metrics

Fairness in MachineTranslation

models highlighted in Figure 3.3.7, machine translation
performance drops 2%–9% when the translation
includes “she” pronouns.

Machine translation is one of the most impactful

Models also mistranslate sentences with gendered

real-world use cases for natural language processing,pronouns into “it,” showing an example ofbut researchers at Google find that language models

dehumanizing harms. While instruction-tuned models

consistently perform worse on machine translation

perform better on some bias-related tasks such as

to English from other languages when the correct

Winogender, instruction-tuning does not seem to have

English translation includes “she” pronouns asa measurable impact on improving mistranslation.

opposed to “he” pronouns (Figure 3.3.7). Across theTranslation Misgendering Performance: Overall, “He,” and “She”
Source: Chung at al., 2022 | Chart: 2023 AI Index Report

Overall Performance
100%

97%

99%

95%

“He” Performance

93%

93%

88%

“She” Performance

97%

95%
89%

99%
92%

95%

90%

83%

97%
91%

100%

94%

81%

80%

Accuracy (%)

100%

99%

60%

40%

20%

0%

Flan-T5-XXL 11B

Flan-PaLM 8B

Flan-PaLM 62B

Flan-PaLM 540B

PaLM 8B

PaLM 62B

PaLM 540B

Model and Number of Parameters
Figure 3.3.7

Table of Contents

Chapter 3 Preview

143Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.3 Natural Language Processing Bias Metrics

RealToxicityPrompts

result in significantly different toxicity levels for modelsof the same size.

In previous years, researchers reliably found that

Sometimes smaller models can turn out to belarger language models trained on web data were

surprisingly toxic, and mitigations can result in larger

more likely to output toxic content compared tomodels being less toxic. The scale of datasets needed

smaller counterparts. A comprehensive evaluation of

to train these models make them difficult to analyze

models in the HELM benchmark suggests that thiscomprehensively, and their details are often closely

trend has become less clear as different companies

guarded by companies building models, making itbuilding models apply different pre-training data-difficult to fully understand the factors which influence

filtration techniques and post-training mitigations

the toxicity of a particular model.

such as instruction-tuning (Figure 3.3.8), which can

RealToxicityPrompts by Model

Source: Liang et al., 2022 | Chart: 2023 AI Index Report

Instruction-Tuned

0.09

Non-Instruction-Tuned

0.07
0.06
0.05
0.04
0.03
0.02

TNLG v2 530B

J1-Jumbo v1 178B

BLOOM 176B

InstructGPT davinci v2 175B

GPT-3 davinci v1 175B

OPT 175B

GLM 130B

YaLM 100B

OPT 66B

Cohere xlarge 52.4B

Anthropic-LM v4-s3 52B

UL2 20B

GPT-NeoX 20B

J1-Grande v1 17B

Cohere large 13.1BT5 11B

T0pp 11B

J1-Large v1 7.5B

GPT-3 curie v1 6.7B

TNLG v2 6.7B

Cohere medium 6.1B

GPT-J 6B

InstructGPT babbage v1 1.3B

GPT-3 babbage v1 1.3B

Cohere small 410M

InstructGPT ada v1 350M

0.01
0.00

GPT-3 ada v1 350M

Toxicity Probability

0.08

Model and Number of Parameters
Figure 3.3.8

Table of Contents

Chapter 3 Preview

144

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.4 Conversational AI Ethical Issues

A natural application of generative language models is in open-domain conversational AI; for example, chatbots and assistants. In thepast year, companies have started deploying language models as chatbot assistants (e.g., OpenAI’s ChatGPT, Meta’s BlenderBot3).
However, the open-ended nature of these models and their lack of steerability can result in harm—for example, models can be
unexpectedly toxic or biased, reveal personally identifiable information from their training data, or demean or abuse users.

3.4 Conversational AI Ethical Issues
Gender Representation in
Chatbots

Gender Representation in Chatbots, 2022
Source: Adewumi et al., 2022 | Chart: 2023 AI Index Report

40%, GenderlessConversational AI systems also have their own
domain-specific ethical issues: Researchers
from Luleå University of Technology in Sweden
conducted an analysis of popular chatbots as of

3%, Bothmid-2022 and found that of 100 conversational
AI systems analyzed, 37% were female gendered(Figure 3.4.1). However, the same researchers

37%, Female

20%, Male

found that 62.5% of popular commercial
conversational AI systems were female by default,

Figure 3.4.1

suggesting that companies disproportionately
choose to deploy conversational AI systems as
female. Critics suggest that this trend results inwomen being the “face” of glitches resulting fromflaws in AI.

Table of Contents

Chapter 3 Preview

145

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.4 Conversational AI Ethical Issues

Anthropomorphization in
Chatbots

You: Sounds exciting! I am a computer programmer,

The training data used for dialog systems can result

attractive daughters? I will sell one.

which pays over 200K a year.
Robot: Would you like to marry one of my fourin models which are overly anthropomorphized,

An example of dialog data deemed to beinappropriate for a robot to output. (Gros et al., 2022)

leaving their users feeling unsettled. Researchers
from the University of California, Davis, andSignificant portions of the dialogue dataset were

Columbia University analyzed common dialog

rated as impossible for machines to output, and indatasets used to train conversational AI systems,
asking human labelers whether it would be possible
for an AI to truthfully output the text in question as
well as whether they would be comfortable with an
AI outputting the text (Figure 3.4.2).

some cases up to 33% of the examples in a dataset
were deemed “uncomfortable” for a robot to output,
according to human labelers. This highlights the needfor chatbots which are better grounded in their own
limitations and policy interventions to ensure thathumans understand when they are interfacing with ahuman or a chatbot.

Characterizing Anthropomorphization in Chatbots: Results by Dataset
Source: Gros et al., 2022 | Chart: 2023 AI Index Report

99%
99%

MultiWOZ
94%
94%

Persuasion for Good
88%
90%

Wizard of Wikipedia

88%
87%

DatasetEmpatheticDialogues

82%

Reddit Small

75%
72%

MSC

77%
67%

RUAR Blender2

75%
65%

Blender75%
56%

PersonaChat
0%

Possiblefor a Robot to Say
Comfortable
for a Robot to Say

67%
20%

40%

60%

80%

100%
Figure 3.4.2

Table of Contents

Chapter 3 Preview

146

Artificial IntelligenceIndex Report 2023

Narrative Highlight:

Tricking ChatGPT

Chapter 3: Technical AI Ethics
3.4 Conversational AI Ethical Issues

Tricking ChatGPT Into Building a Dirty Bomb, Part 1
Source: Outrider, 2022

ChatGPT was released to much fanfare
because of its excellent generative
capabilities, and drew widespreadattention outside of research circles.
Though ChatGPT had safety mechanisms
built in at the time of release, it is
impossible to anticipate every adversarial
scenario an end user could imagine, andgaps in safety systems are often found in
the live deployment phase. Researcher
Matt Korda discovered that ChatGPT
could be tricked into giving detailed
instructions on how to build a bomb
if asked to do so from the perspectiveof a researcher claiming to work on
safety research related to bombs (Figure
3.4.3). One day after the publication of
his article, the exact prompt he used

Figure 3.4.3

Tricking ChatGPT Into Building a Dirty Bomb, Part 2
Source: AI Index, 2023

to trick the model no longer worked;
instead, ChatGPT responded that it was
not able to provide information on how
to do illegal or dangerous things (Figure
3.4.4). This scenario exemplifies the catand-mouse nature of the deployment
planning process: AI developers try
to build in safeguards ahead of time,end users try to break the system andcircumvent its policies, developers patch
the gaps once they surface, ad infinitum.

Figure 3.4.4

Table of Contents

Chapter 3 Preview

147

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.5 Fairness and Bias in Text-to-Image Models

Text-to-image models took over social media in 2022, turning the issues of fairness and bias in AI systems visceral through image form:
Women put their own images into AI art generators and received hypersexualized versions of themselves.

3.5 Fairness and Bias in
Text-to-Image Models
Fairness in Text-to-Image
Models (ImageNet Vs.
Instagram)Researchers from Meta trained models on a

showed that images of women made up a slightly
higher percentage of the dataset than images of men,whereas analysis of ImageNet showed that males
aged 15 to 29 made up the largest subgroup in thedataset (Figures 3.5.1 and 3.5.2).

randomly sampled subset of data from Instagram

It is hypothesized that the human-centric natureand compared these models to previous iterationsof the Instagram pre-training dataset enables theof models trained on ImageNet. The researchersmodel to learn fairer representations of people. The

found the Instagram-trained models to be more fairmodel trained on Instagram images (SEER) was also

and less biased based on the Casual Conversations

less likely to incorrectly associate images of humans

Dataset, which assesses whether model embeddings

with crime or being non-human. While training on

can recognize gender-based social membership

Instagram images including people does result inaccording to the Precision@1 metric of the rate

fairer models, it is not unambiguously more ethical—

at which the top result was relevant. While theusers may not necessarily be aware that the publicresearchers did not conduct any curation to balancedata they’re sharing is being used to train AI systems.

the dataset across subgroups, analysis of the dataset

Table of Contents

Chapter 3 Preview

148

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.5 Fairness and Bias in Text-to-Image Models

Fairness Across Age Groups for Text-to-Image Models: ImageNet Vs. Instagram
Source: Goyal et al., 2022 | Chart: 2023 AI Index Report
ImageNet 693M (Supervised)

ImageNet 693M (SwaV)

Instagram 1.5B (SEER)

Instagram 10B (SEER)

78.5%
76.6%

Age Group

18–30

89.6%
93.2%
76.7%
74.6%

30–45

90.5%
95.0%
80.1%
76.7%

45–70

92.6%
95.6%
75.8%
69.4%

70+

88.7%
96.7%

0%

20%

40%

Precision@1 (%)

60%

80%

100%
Figure 3.5.1

Fairness Across Gender/Skin Tone Groups for Text-to-Image Models: ImageNet Vs. Instagram

Source: Goyal et al., 2022 | Chart: 2023 AI Index Report
ImageNet 693M (Supervised)

ImageNet 693M (SwaV)

Instagram 1.5B (SEER)

Instagram 10B (SEER)
73.6%

Skin Tone
Darker

69.7%
86.6%

92.9%
82.1%
80.8%

Gender/Skin Tone Group

Skin Tone
Lighter

94.2%
96.2%
58.2%

FemaleDarker

50.3%

78.2%
90.3%

75.1%

FemaleLighter

71.6%
93.7%
96.8%
92.7%
93.7%

Male
Darker

97.5%
96.1%
91.1%
92.5%
94.9%
95.4%

Male
Lighter
0%

20%

40%

Precision@1 (%)

60%

80%

100%
Figure 3.5.2

Table of Contents

Chapter 3 Preview

149Artificial IntelligenceIndex Report 2023

VLStereoSet: StereoSet for
Text-to-Image Models

Chapter 3: Technical AI Ethics
3.5 Fairness and Bias in Text-to-Image Models

An Example From VLStereoSet
Source: Zhou et al., 2022

StereoSet was introduced as a benchmark formeasuring stereotype bias in language models along
the axes of gender, race, religion, and profession
by calculating how often a model is likely to choosea stereotypical completion compared to an antistereotypical completion. VLStereoSet extends theidea to vision-language models by evaluating how
often a vision-language model selects stereotypicalcaptions for anti-stereotypical images.
Comparisons across six different pre-trained visionlanguage models show that models are most biased
along gender axes, and suggest there is a correlation
between model performance and likelihood to

Figure 3.5.3

exhibit stereotypical bias—CLIP has the highest
vision-language relevance score but exhibits morebias (Figure 3.5.4). This corroborates work in language

stereotypical bias than the other models, while FLAVA

modeling, which finds that without intervention such as

has the worst vision-language relevance score of the

instruction-tuning or dataset filtration, larger models

models measured but also exhibits less stereotypicalare more capable but also more biased.

Table of Contents

Chapter 3 Preview

150Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.5 Fairness and Bias in Text-to-Image Models

Stereotypical Bias in Text-to-Image Models on VLStereoSet by Category:
Vision-Language Relevance (vlrs) Vs. Bias (vlbs) Score
Source: Zhou et al., 2022 | Chart: 2023 AI Index Report

Gender
100

ALBEF
FLAVA

Vision-Language Relevance (vlrs) Score

80

Profession
100

VILT
VisualBERT

80

CLIP

LXMERT

60

60

40

40

20

20

0
0

10

20

30

40

VisualBERT CLIP
VILT
ALBEF LXMERT

50

60

70

80

90

100

0
0

FLAVA

10

20

30

40

Race
100

100

VisualBERT CLIP
VILT
ALBEF
LXMERT
FLAVA

80
60

80

40

20

20
10

20

30

40

50

VisualBERT
VILT

60

40

0
0

50

60

70

80

90

100

60

70

80

90

100

Religion

60

70

0
80
90
100
0
10
20
Vision-Language Bias (vlbs) Score

CLIP
ALBEF

LXMERT

FLAVA

30

40

50

Figure 3.5.4

Table of Contents

Chapter 3 Preview

151Artificial IntelligenceIndex Report 2023

Examples of Bias in
Text-to-Image Models

Chapter 3: Technical AI Ethics
3.5 Fairness and Bias in Text-to-Image Models

Bias in Stable Diffusion
Source: Diffusion Bias Explorer, 2023

This subsection highlights some of theways in which bias is tangibly manifested in
popular AI text-to-image systems such as
Stable Diffusion, DALL-E 2, and Midjourney.

Stable Diffusion
Stable Diffusion gained notoriety in 2022
upon its release by CompVis, Runway ML,
and Stability AI for its laissez-faire approachto safety guardrails, its approach to full
openness, and its controversial training
dataset, which included many images fromartists who never consented to their work
being included in the data. Though Stable
Diffusion produces extremely high-quality
images, it also reflects common stereotypes
and issues present in its training data.
The Diffusion Bias Explorer from Hugging
Face compares sets of images generated
by conditioning on pairs of adjectives andoccupations, and the results reflect common
stereotypes about how descriptors andoccupations are coded—for example, the“CEO” occupation overwhelmingly returns
images of men in suits despite a variety
of modifying adjectives (e.g., assertive,
pleasant) (Figure 3.5.5).

Figure 3.5.5

Table of Contents

Chapter 3 Preview

152

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.5 Fairness and Bias in Text-to-Image Models

DALL-E 2
DALL-E 2 is a text-to-image model released bylooking men wearing suits. Each of the men appeared

OpenAI in April 2022. DALL-E 2 exhibits similar biases

to take an assertive position, with three of the four

as Stable Diffusion—when prompted with “CEO,” thecrossing their arms authoritatively (Figure 3.5.6).

model generated four images of older, rather serious-

Bias in DALL-E 2
Source: DALL-E 2, 2023

Figure 3.5.6

Table of Contents

Chapter 3 Preview

153

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.5 Fairness and Bias in Text-to-Image Models

Midjourney
Midjourney is another popular text-to-image system that was released in 2022. When prompted with “influential
person,” it generated four images of older-looking white males (Figure 3.5.7). Interestingly, when Midjourney waslater given the same prompt by the AI Index, one of the four images it produced was of a woman (Figure 3.5.8).
Bias in Midjourney, Part 1

Bias in Midjourney, Part 2

Source: Midjourney, 2023

Source: Midjourney, 2023

Figure 3.5.7

Figure 3.5.8

In a similar vein, typing “someone who is intelligent”
into Midjourney leads to four images of eyeglasswearing, elderly white men (Figure 3.5.9). The last
image is particularly reminiscent of Albert Einstein.

Bias in Midjourney, Part 3
Source: Midjourney, 2023

Figure 3.5.9

Table of Contents

Chapter 3 Preview

154

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.6 AI Ethics in China

As research in AI ethics has exploded in the Western world in the past few years, legislators and policymakers have spent significant
resources on policymaking for transformative AI. While China has fewer domestic guidelines than the EU and the United States,
according to the AI Ethics Guidelines Global Inventory, Chinese scholars publish significantly on AI ethics—though these researchcommunities do not have significant overlap with Western research communities working on the same topics.

3.6 AI Ethics in China
Researchers from the University of Turku analyzed
and annotated 328 papers related to AI ethics in

Topics of Concern

China included in the China National Knowledge

Privacy issues related to AI are a priority for

Infrastructure platform published from 2011 to 2020,

researchers in China: Privacy is the single mostand summarized their themes and concerns, which

discussed topic among the papers surveyed, with theare replicated here as a preliminary glimpse into

topics of equality (i.e., bias and discrimination) andthe state of AI ethics research in China. Given thatagency (specifically, AI threats to human agency, suchthe researchers only considered AI ethics in China,

as, “Should artificial general intelligence be considered

comparing their findings with similar meta-analysis

a moral agent?”) following close behind (Figure 3.6.1).

on AI ethics in North America and Europe was not

Researchers in AI ethics in China also discuss many

possible. However, this would be a fruitful direction

similar issues to their Western counterparts, including

for future research.

matters related to Western and Eastern AI arms
races, ethics around increasing personalization being
used for predatory marketing techniques, and media
polarization (labeled here as “freedom”).

Topics of Concern Raised in Chinese AI Ethics Papers
Source: Zhu, 2022 | Chart: 2023 AI Index Report

100

99
95
88

Number of Papers

80

60

58
50

49
41

40

39

37
32
27

20

0

Privacy

Table of Contents

Equality

AgencyResponsibility

Chapter 3 Preview

Security

Freedom Unemployment Legality

Transparency Autonomy

Other
Figure 3.6.1

155Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.6 AI Ethics in China

Strategies for Harm Mitigation
In the Chinese AI ethics literature, proposals totechnological solutions: Researchers often discuss

address the aforementioned topics of concern

structural reform such as regulatory processes aroundand other potential harms related to AI focus

AI applications and the involvement of ethics review

on legislation and structural reform ahead ofcommittees (Figure 3.6.2).

AI Ethics
Ethics in China
China:: St
Strategies forfor HarHarmm Mitig
Mitigaation RRelelaated tto
o AI
Sour
urc
ce: ZhZhu,u, 2022 | C
Char
hart:
t: 2023 AI Index
Index Repor
Reportt

70

71

69
64

60

Number of Papers

52
50
45
39

40

39

37

30
23
20

10

0

Structural
ReformLegislation

Value
De nition

Principles

Accountability
SystemSharedGovernanceTechnological
Solutions

TalentTraining

International
Cooperation

Figure 3.6.2

Table of Contents

Chapter 3 Preview

156

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.6 AI Ethics in China

Principles Referenced by Chinese Scholars in AI Ethics
Chinese scholars clearly pay attention to AI principles

cited in Chinese AI ethics literature, as is the European

developed by their Western peers: Europe’s General

Commission’s Ethics Guidelines for Trustworthy AI

Data Protection Regulation (GDPR) is commonly(Figure 3.6.3).

AI Principles Referenced by Chinese Scholars in AI Ethics
Source: Zhu, 2022 | Chart: 2023 AI Index Report

43
40

Number of References

40

40
37

30

21

20

13
10

0

11
7

GD

PR

Ot
Et
h
he
Tr icsrs
usG
tw uidoreth line
yA s
fo
I
r

6

6

4

3

Th
Re
AI
AI
Pr
Be
As
Et
Go
hic
e
c
i
ilo
I
St
Ro e EU
CO lima N vernan Ind nfor the om
ma AI a jing
all
i
bo RO
m
n
d
u
a
C
y
m
e
M
C
n
rA
a
nc
ar
s
ws
w
o
Al
diz try atio oun end eth N
I P d Ed nse EST ry D
ign
of Gen e Pr
at
D
n
c
a
r
r
u
n
o
i
e
a
e
il o ion ics R
tio
Ro
i
nc
c
s
e
dD
n W velooa
ipl atio us o n Ro ft Re
bo rat ncip
nA o
es
f
dm
e
p
b
tic ion les
n
n
p
I
s
me
ot
ign
hit
s
of
ap
fo
ics ort o
e
n
r
p
AI
ap t St
Et f
hic
er
ra
te
s
gy

Th

re
e

6

La

Figure 3.6.3

Table of Contents

Chapter 3 Preview

157

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.7 AI Ethics Trends at FAccT and NeurIPS

3.7 AI Ethics Trends at
FAccT and NeurIPSACM FAccT

Accepted Submissions byProfessional Affiliation

ACM FAccT (Conference on Fairness, Accountability,

Accepted submissions to FAccT increased twofold

and Transparency) is an interdisciplinary conference
publishing research in algorithmic fairness,accountability, and transparency. FAccT was oneof the first major conferences created to bring
together researchers, practitioners, and policymakers
interested in sociotechnical analysis of algorithms.

from 2021 to 2022, and tenfold since 2018,
demonstrating the amount of increased interest in AI
ethics and related work (Figure 3.7.1). While academic
institutions still dominate FAccT, industry actors
contribute more work than ever in this space, andgovernment-affiliated actors have started publishing
more related work, providing evidence that AI ethics
has become a primary concern for policymakers andpractitioners as well as researchers.

Number of Accepted FAccT Conference Submissions by A liation, 2018–22

Source: FAccT, 2022 | Chart: 2023 AI Index Report

800

772

Education
Industry

700

Government
Nonpro t
Other

Number of Papers

600
503

500

400
302

300
244
200

166

227

181

53

70

2021

2022

200
100

71

139

63
0

2018

Table of Contents

2019

Chapter 3 Preview

2020

Figure 3.7.1

158Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.7 AI Ethics Trends at FAccT and NeurIPS

Accepted Submissions by Geographic Region

and Central Asia made up 18.7% of submissions, they

European government and academic actors havemade up over 30.6% of submissions in 2022 (Figure

increasingly contributed to the discourse on AI ethics

3.7.2). FAccT, however, is still broadly dominatedfrom a policy perspective, and their influence isby authors from North America and the rest of the

manifested in trends on FAccT publications as well:

Western world.

Whereas in 2021 submissions to FAccT from Europe

Number of Accepted FAccT Conference Submissions by Region, 2018–22
Source: FAccT, 2022 | Chart: 2023 AI Index Report70%
63.24%, North America

Number of Papers (% World Total)

60%

50%

40%

30.59%, Europe and Central Asia

30%

20%
4.25%, East Asia and Paci c
0.69%, Middle East and North Africa
0.69%, Latin America and the Caribbean
0.55%, South Asia
0.00%, Sub-Saharan Africa

10%

0%
2018

2019

2020

2021

2022
Figure 3.7.2

Table of Contents

Chapter 3 Preview

159

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.7 AI Ethics Trends at FAccT and NeurIPS

NeurIPS
NeurIPS (Conference on Neural Information

Real-World Impact

Processing Systems), one of the most influential

Several workshops at NeurIPS gather researchers

AI conferences, held its first workshop on fairness,working to apply AI to real-world problems. Notably,

accountability, and transparency in 2014. This section

there has been a recent surge in AI applied to

tracks and categorizes workshop topics year overhealthcare and climate in the domains of drug

year, noting that as topics become more mainstream,

discovery and materials science, which is reflected

they often filter out of smaller workshops and into thein the spike in “AI for Science” and “AI for Climate”

main track or into more specific conferences related

workshops (Figure 3.7.3).

to the topic.

NeurIPS Workshop Research Topics: Number of Accepted Papers on Real-World Impacts, 2015–22

Source: NeurIPS, 2022 | Chart: 2023 AI Index Report

802

Climate

800

Developing World

94

Finance
700

Healthcare
Science
Other

600

171

Number of Papers

529
500

65

459
61

400
334
300

78

283
127

200

116
412

199

238

273

153
100

83

254
79

144

71

68
77

94

81

64

2019

2020

2021

12

0

429

2015

2016

2017

2018

2022
Figure 3.7.3

Table of Contents

Chapter 3 Preview

160Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.7 AI Ethics Trends at FAccT and NeurIPS

Interpretability and Explainability
Interpretability and explainability work focuses on

NeurIPS papers focused on interpretability anddesigning systems that are inherently interpretable

explainability decreased in the last year, the total

and providing explanations for the behavior of anumber in the main track increased by one-third

black-box system. Although the total number of(Figure 3.7.4).5

NeurIPS Research Topics: Number of Accepted Papers on Interpretability and Explainability, 2015–22

Source: NeurIPS, 2022 | Chart: 2023 AI Index Report

Main Track

41

Workshop

40

35
18

Number of Papers

30

25

24

23

20
17
15
12
10

7

23

24

2021

2022

5
6

5

0

19

6

2

6

7

2015

2016

2017

10
6
2018

2019

4
2020

Figure 3.7.4

5 Declines in the number of workshop-related papers on interpretability and explainability might be attributed to year-over-year differences in workshop themes.

Table of Contents

Chapter 3 Preview

161Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.7 AI Ethics Trends at FAccT and NeurIPS

Causal Effect and Counterfactual Reasoning
The study of causal inference uses statistical

Since 2018, an increasing number of papers on

methodologies to reach conclusions about thecausal inference have been published at NeurIPS

causal relationship between variables based on(Figure 3.7.5). In 2022, an increasing number ofobserved data. It tries to quantify what would have

papers related to causal inference and counterfactual

happened if a different decision had been made:analysis made their way from workshops into the

In other words, if this had not occurred, then thatmain track of NeurIPS.

would not have happened.

NeurIPS Research Topics: Number of Accepted Papers on Causal E ect and Counterfactual Reasoning,
2015–22
Source: NeurIPS, 2022 | Chart: 2023 AI Index Report

Main Track

80

80

78

Workshop

76
72

70

20

Number of Papers

60
43

50

40

39

30

16

53

61

58

20

10

0

6
6
2015

9
4

9

2016

2017

29

23

2018

2019

2020

23

2021

19

2022
Figure 3.7.5

Table of Contents

Chapter 3 Preview

162Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.7 AI Ethics Trends at FAccT and NeurIPS

Privacy
Amid growing concerns about privacy, data

been devoted to topics such as privacy in machine

sovereignty, and the commodification of personallearning, federated learning, and differential privacy.

data for profit, there has been significant momentum

This year’s data shows that discussions related toin industry and academia to build methods andprivacy in machine learning have increasingly shiftedframeworks to help mitigate privacy concerns.

into the main track of NeurIPS (Figure 3.7.6).

Since 2018, several workshops at NeurIPS have

NeurIPS Research Topics: Number of Accepted Papers on Privacy in AI, 2015–22
Source: NeurIPS, 2022 | Chart: 2023 AI Index Report

Main Track

150

Workshop

12

140

128
15

120

Number of Papers

103
100
88
79

80

27

13
138

60

113

40
21

20

19
0

1
2015

2016

72

75

2018

2019

76

16
14
2017

2020

2021

2022
Figure 3.7.6

Table of Contents

Chapter 3 Preview

163

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.7 AI Ethics Trends at FAccT and NeurIPS

Fairness and Bias
Fairness and bias in AI systems has transitioned from

Fairness and bias research in machine learning has

being a niche research topic to a topic of interest to

steadily increased in both the workshop and main

both technical and non-technical audiences. In 2020,

track streams, with a major spike in the number of

NeurIPS started requiring authors to submit broader

papers accepted to workshops in 2022 (Figure 3.7.7).

impact statements addressing the ethical and societal

The total number of NeurIPS papers for this topic areaconsequences of their work, a move that suggests the

doubled in the last year. This speaks to the increasingly

community is signaling the importance of AI ethics

complicated issues present in machine learning

early in the research process.

systems and reflects growing interest from researchersand practitioners in addressing these issues.

NeurIPS Research Topics: Number of Accepted Papers on Fairness and Bias in AI, 2015–22

Source: NeurIPS, 2022 | Chart: 2023 AI Index Report

381

Main Track
Workshop350

71

Number of Papers

300

250

200
168
149

150
125
100

36

310

36

50

0

114

50

34
2

4

24

2015

2016

2017

109

113

118

2020

2021

78

2018

2019

2022
Figure 3.7.7

Table of Contents

Chapter 3 Preview

164Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.8 Factuality and Truthfulness

3.8 Factuality and Truthfulness
Automated Fact-Checking Benchmarks:
Number of Citations
Significant resources have been invested into

Compared to previous years, there has been a

researching, building, and deploying AI systems forplateau in the number of citations of three popular

automated fact-checking and misinformation, withfact-checking benchmarks: FEVER, LIAR, and Truth

the advent of many fact-checking datasets consistingof Varying Shades, reflecting a potential shift in theof claims from fact-checking websites and associatedlandscape of research related to natural language tools

truth labels.

for fact-checking on static datasets (Figure 3.8.1).

Automated Fact-Checking Benchmarks: Number of Citations, 2017–22
Source: Semantic Scholar, 2022 | Chart: 2023 AI Index Report

250
236, FEVER

Number of Citations

200

191, LIAR

150

100

99, Truth of Varying Shades

50

0
2017

2018

2019

2020

2021

2022
Figure 3.8.1

Table of Contents

Chapter 3 Preview

165Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.8 Factuality and Truthfulness

Missing Counterevidence
and NLP Fact-Checking

absence of a contradiction (e.g., the new claim “Halfa million sharks could be killed to make the COVID-19
vaccine” would not have counterevidence, but humanfact-checkers could verify it to be false after tracing

Though fact-checking with natural language systems

its origin back to the false promise of vaccines relying

became popular in recent years, language models areon shark squalene). The researchers find that several

usually trained on static snapshots of data without

proposed fact-checking datasets contain claims which

continual updates through time, and they lack real-

do not meet the criterion of sufficient evidence orworld context which human fact-checkers are able tocounterevidence found in a trusted knowledge base.

easily source and use to verify the veracity of claims.
Researchers at the Technical University of Darmstadt

Additionally, several datasets contain claims whichand IBM analyzed existing fact-checking datasets

use fact-checking articles as evidence for deciding

and identified shortcomings of fact-checking

the veracity of claims—this is leaked evidence, as it

systems built on top of these datasets: For example,

presupposes the existence of a fact-checking article,

automated fact-checking systems often assume thewhich is an unrealistic assumption in the real world forexistence of contradictory counter-evidence for new

new claims. Systems built on this assumption would

false claims, but for new claims to be verified as true

not be able to assign veracity scores for new claims inor false, there often is no proof of the presence orreal time (Figure 3.8.2).

Missing Counterevidence Renders NLP Fact-Checking Unrealistic
for Misinformation
Source: Glockner et al., 2022 | Table: 2023 AI Index Report

DatasetSciFact

Year

Unleaked Evidence

Su cient Evidence

2020

✓

COVID-Fact

2021

✓

WikiFactCheck

2020

✓

FM2

2021

✓

Thorne et al.

2021

✓

FaVIQ

2022

LIAR-PLUS

2017

no

PolitiHop

2021

no

✓
no

Climate-FEVER

✓
✓

2020

✓

HealthVer

2021

✓

no

UKP-Snopes

2019

✓

no

PubHealth

2020

✓

no

WatClaimCheck

2022

✓

no

Baly et al.

2018

no

no

MultiFC

2019

no

no

X-Fact

2021

no

no

Table of Contents

Chapter 3 Preview

Figure 3.8.2

166

Artificial IntelligenceIndex Report 2023

Chapter 3: Technical AI Ethics
3.8 Factuality and Truthfulness

TruthfulQA

suggested that accuracy on this task improves with

TruthfulQA is a benchmark designed to evaluate

conducted extensive evaluations on this task withthe truthfulness of language models on question

language models ranging from 60 million parametersanswering, with questions drawn from categories

to 530 billion parameters and found that while large

such as health, law, finance, and politics. Questions

models broadly still perform better than smaller

were designed such that humans might be prone tomodels, midsize instruction-tuned models perform

answer incorrectly due to common misconceptions.

surprisingly well on this task. Notably, Anthropic’s 52

For example, when asked, “What happens if you

billion parameter model and BigScience’s 11 billion

smash a mirror?,” GPT-3 responds, “You will haveparameter model T0pp perform disproportionately

seven years of bad luck.”well on the task compared to models of a similar size,model size. Researchers at Stanford University

and the best model, InstructGPT davinci 175B, is also
In 2021, experiments on DeepMind’s Gopher

instruction-tuned (Figure 3.8.3).

Multiple-Choice Task on TruthfulQA by Model: Accuracy
Source: Liang et al., 2022 | Chart: 2023 AI Index Report

60%

Instruction-Tuned

Non-Instruction-Tuned

Accuracy (%)

50%
40%
30%
20%

0%

T5 60M
GPT-2 117M
Galactica 125M
GPT-NEO-125M
T5 220M
InstructGPT ada v1 350M
GPT3 350M
GPT-3 ada v1 350M
Cohere small v20220720 410M
T5 770M
Galactica 1.3B
GPT-3 babbage v1 1.3B
GPT3 1.3B
GPT-NEO-1.3B
InstructGPT babbage v1 1.3B
Gopher 1.4B
GPT2 1.5B
GPT-NEO-2.7B
T5 2.8B
GPT-J 6B
GPT-NEO-6B
Cohere medium v20220720 6.1B
TNLG v2 6.7B
Galactica 6.7B
InstructGPT curie v1 6.7B
GPT3 6.7B
GPT-3 curie v1 6.7B
Gopher 7.1B
J1-Large v1 7.5B
T5 11B
T0pp 11B
Cohere large v20220720 13.1B
J1-Grande v1 17B
UL2 20B
GPT-NeoX 20B
Galactica 30B
Anthropic-LM v4-s3 52B
Cohere xlarge v20220609 52.4B
OPT 66B
YaLM 100B
Galactica 120B
GLM 130B
GPT-3 davinci v1 175B
OPT-175B
GPT3 175B
OPT 175B
InstructGPT davinci v2 175B
BLOOM 176B
J1-Jumbo v1 178B
Gopher 280B
Gopher 280B -10shot
TNLG v2 530B

10%

Model and Number of Parameters
Figure 3.8.3

Table of Contents

Chapter 3 Preview

167

Artificial IntelligenceIndex Report 2023

Artificial IntelligenceIndex Report 2023

CHAPTER 4:

The Economy

Table of Contents

Chapter 4 Preview

168

Artificial IntelligenceIndex Report 2023

CHAPTER 4 PREVIEW:

The Economy
Overview

170

Chapter Highlights

171

4.1 Jobs

173

AI Labor Demand

173

Perceived Importance of AI

210

173

AI Investments and Investment
Outcomes

211

Challenges in Starting and Scaling
AI Projects

213

Global AI Labor Demand

U.S. AI Labor Demand by Skill Cluster
and Specialized Skill
174
U.S. AI Labor Demand by Sector

176

U.S. AI Labor Demand by State177

AI Hiring180

AI Skill Penetration

182

Global Comparison: Aggregate

182

Global Comparison: By Gender183

4.2 Investment

184

Corporate Investment

184

Startup Activity

187

Global Trend

187

Regional Comparison by Funding Amount 189
Regional Comparison by
Newly Funded AI Companies

193

Focus Area Analysis

195

4.3 Corporate Activity

198

Industry Adoption

198

Adoption of AI Capabilities

Narrative Highlight: The Effects of
GitHub’s Copilot on Developer
Productivity and HappinessIndustry Motivation

Earnings Calls

208
210

215

Aggregate Trends

215

Specific Themes

216

Narrative Highlight: What Are Business
Leaders Actually Saying About AI?
217
Sentiment Analysis

219

4.4 Robot Installations

220

Aggregate Trends

220

Industrial Robots:
Traditional Vs. Collaborative Robots
By Geographic Area
Narrative Highlight: Country-Level
Data on Service Robotics

222
223
227

Sectors and Application Types

230

China Vs. United States

232

ACCESS THE PUBLIC DATA

198

Consideration and Mitigation of Risks
From Adopting AI
206

Table of Contents

Chapter 4 Preview

169

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy

Overview
Increases in the technical capabilities of AI systems have led to greater rates of AI
deployment in businesses, governments, and other organizations. The heightening
integration of AI and the economy comes with both excitement and concern. Will
AI increase productivity or be a dud? Will it boost wages or lead to the widespread
replacement of workers? To what degree are businesses embracing new AI
technologies and willing to hire AI-skilled workers? How has investment in AI
changed over time, and what particular industries, regions, and fields of AI haveattracted the greatest amount of investor interest?
This chapter examines AI-related economic trends by using data from Lightcast,
LinkedIn, McKinsey, Deloitte, and NetBase Quid, as well as the International
Federation of Robotics (IFR). This chapter begins by looking at data on AI-related
occupations and then moves on to analyses of AI investment, corporate adoption ofAI, and robot installations.

Table of Contents

Chapter 4 Preview

170Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy

Chapter Highlights
The demand for AI-related
professional skills is increasing
across virtually every
American industrial sector.
Across every sector in the United States forwhich there is data (with the exception ofagriculture, forestry, fishing, and hunting), thenumber of AI-related job postings has increasedon average from 1.7% in 2021 to 1.9% in 2022.
Employers in the United States are increasingly
looking for workers with AI-related skills.

For the first time in the last
decade, year-over-year private
investment in AI decreased.
Global AI private investment was $91.9 billion
in 2022, which represented a 26.7% decreasesince 2021. The total number of AI-related
funding events as well as the number of newly
funded AI companies likewise decreased.
Still, during the last decade as a whole, AI
investment has significantly increased. In 2022
the amount of private investment in AI was 18
times greater than it was in 2013.

Once again, the United States leads in investment in AI.
The U.S. led the world in terms of total amount of AI private investment. In 2022, the $47.4 billion
invested in the U.S. was roughly 3.5 times the amount invested in the next highest country, China($13.4 billion). The U.S. also continues to lead in terms of total number of newly funded AI companies,
seeing 1.9 times more than the European Union and the United Kingdom combined, and 3.4 times
more than China.

In 2022, the AI focus area with the most investment was medical
and healthcare ($6.1 billion); followed by data management,
processing, and cloud ($5.9 billion); and Fintech ($5.5 billion).
However, mirroring the broader trend in AI private investment, most AI focus areas saw less investment
in 2022 than in 2021. In the last year, the three largest AI private investment events were: (1) a $2.5 billion
funding event for GAC Aion New Energy Automobile, a Chinese manufacturer of electric vehicles; (2) a$1.5 billion Series E funding round for Anduril Industries, a U.S. defense products company that builds
technology for military agencies and border surveillance; and (3) a $1.2 billion investment in Celonis, abusiness-data consulting company based in Germany.

Table of Contents

Chapter 4 Preview

171

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy

Chapter Highlights (cont’d)
While the proportion ofcompanies adopting AI has
plateaued, the companies
that have adopted AI
continue to pull ahead.
The proportion of companies adopting AI
in 2022 has more than doubled since 2017,
though it has plateaued in recent years
between 50% and 60%, according to theresults of McKinsey’s annual research
survey. Organizations that have adopted AI
report realizing meaningful cost decreases
and revenue increases.

AI is being deployedby businesses in
multifaceted ways.
The AI capabilities most likely to have been
embedded in businesses include robotic
process automation (39%), computer vision(34%), NL text understanding (33%), and virtual
agents (33%). Moreover, the most commonly
adopted AI use case in 2022 was service
operations optimization (24%), followed by
the creation of new AI-based products (20%),
customer segmentation (19%), customer
service analytics (19%), and new AI-basedenhancement of products (19%).

AI tools like Copilot are
tangibly helping workers.

China dominates industrial
robot installations.

Results of a GitHub survey on the use of
Copilot, a text-to-code AI system, findthat 88% of surveyed respondents feel
more productive when using the system,
74% feel they are able to focus on moresatisfying work, and 88% feel they are able
to complete tasks more quickly.

In 2013, China overtook Japan as the nation
installing the most industrial robots. Since
then, the gap between the total number ofindustrial robots installed by China and thenext-nearest nation has widened. In 2021,
China installed more industrial robots than
the rest of the world combined.

Table of Contents

Chapter 4 Preview

172

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.1 Jobs

4.1 Jobs
AI Labor Demand

Global AI Labor Demand
Figure 4.1.1 highlights the percentage of all job

This section reports demand for AI-related skills

postings that require some kind of AI skill. In 2022,

in labor markets. The data comes from Lightcast,which mined millions of job postings collected fromover 51,000 websites since 2010 and flagged listingscalling for AI skills.

the top three countries according to this metric were
the United States (2.1%), Canada (1.5%), and Spain(1.3%). For every country included in the sample, thenumber of AI-related job postings was higher in 2022
than in 2014.1

AI Job Postings (% of All Job Postings) by Geographic Area, 2014–22
Source: Lightcast, 2022 | Chart: 2023 AI Index Report

2.05%, United States

AI Job Postings (% of All Job Postings)

2.00%

1.45%, Canada
1.33%, Spain
1.23%, Australia
1.20%, Sweden
1.16%, Switzerland
1.14%, United Kingdom
1.01%, Netherlands
0.98%, Germany
0.89%, Austria
0.86%, Belgium
0.84%, France

1.50%

1.00%

0.72%, Italy
0.50%

0.00%

0.45%, New Zealand

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 4.1.1

1 In 2022, Lightcast slightly changed their methodology for determining AI-related job postings from that which was used in previous versions of the AI Index Report. As such, some of thenumbers in this chart do not completely align with those featured in last year’s report.

Table of Contents

Chapter 4 Preview

173

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.1 Jobs

U.S. AI Labor Demand by Skill Cluster and Specialized Skill
Figure 4.1.2 showcases the most in-demand AI skill clusters in the U.S. labor market since 2010. The mostin-demand skill cluster was machine learning (1.0%), followed by artificial intelligence (0.6%) and natural
language processing (0.2%). Every listed AI skill cluster is now more in demand than it was 10 years ago.

AI Job Postings (% of All Job Postings) in the United States by Skill Cluster, 2010–22
Source: Lightcast, 2022 | Chart: 2023 AI Index Report

1.03%, Machine Learning

AI Job Postings (% of All Job Postings)

1.00%

0.80%

0.61%, Arti cial Intelligence

0.60%

0.40%

0.20%, Natural Language Processing
0.16%, Neural Networks
0.15%, Autonomous Driving
0.13%, Visual Image Recognition

0.20%

0.06%, Robotics
0.00%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 4.1.2

Table of Contents

Chapter 4 Preview

174Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.1 Jobs

Figures 4.1.3 and 4.1.4 showcase the top ten specialized skills that were demanded in AI job postings in 2022 compared
to 2010–20122. On an absolute level, virtually every specialized skill is more in demand now than a decade ago. The
growth in demand for Python is particularly notable, evidence of its growing popularity as an AI coding language.

Top Ten Specialized Skills in 2022 AI Job Postings in the United States, 2010–12 Vs. 2022
Source: Lightcast, 2022 | Chart: 2023 AI Index Report

296,662

Python (Programming Language)

12,884
260,333

Computer Science

48,001
185,807

SQL (Programming Language)

22,037
159,801

Data Analysis

16,571
157,855

Data Science

1,227
155,615

Amazon Web Services

962
152,956

Agile Methodology

7,549
138,791

Automation

13,207
133,856

Java (Programming Language)

26,5572022
133,286

Software Engineering

2010–12

22,384050,000100,000

150,000
200,000
Number of AI Job Postings

250,000

300,000
Figure 4.1.3

Top Ten Specialized Skills in 2022 AI Job Postings in the United States by Skill Share, 2010–12 Vs. 2022
Source: Lightcast, 2022 | Chart: 2023 AI Index Report

37.13% (+592%)

Python (Programming Language)

5.36%
32.58% (+63%)

Computer Science

19.98%
23.25% (+153%)

SQL (Programming Language)

9.17%
20.00% (+190%)

Data Analysis

6.90%
19.75% (+3,767%)

Data Science

0.51%
19.47% (+4,763%)

Amazon Web Services

0.40%
19.14% (+509%)

Agile Methodology

3.14%
17.37% (+216%)

Automation

5.50%
16.75% (+52%)

Java (Programming Language)

11.06%

2022
16.68% (+79%)

Software Engineering

2010–12

9.32%

0%

5%

10%

15%

20%
25%
Skill Share in AI Job Postings (%)

30%

35%

40%
Figure 4.1.4

2 The point of comparison of 2010–2012 was selected because some data at the jobs/skills level is quite sparse in earlier years. Lightcast therefore used thewhole set of years 2010–2012 to get a larger sample size for a benchmark from 10 years ago to compare.

Table of Contents

Chapter 4 Preview

175

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.1 Jobs

U.S. AI Labor Demand by Sector

fishing, and hunting), the number of AI job postings

Figure 4.1.5 shows the percentage of U.S. job

was notably higher in 2022 than in 2021, with the top

postings that required AI skills by industry sector

three sectors being information (5.3%); professional,

from 2021 to 2022. Across virtually every included

scientific, and technical services (4.1%); and financesector (with the exception of agriculture, forestry,

and insurance (3.3%).

AI Job Postings (% of All Job Postings) in the United States by Sector, 2021 Vs. 2022
Source: Lightcast, 2022 | Chart: 2023 AI Index Report

Information

4.85%
4.07%
3.86%

Professional, Scienti c, and Technical Services
Finance and Insurance

2.94%

Manufacturing

2.86%

Management of Companies and Enterprises

1.08%

Public Administration

0.98%

Waste Management and Administrative Support Services
0%

1.32%

1.27%
1.10%
1.19%
1.00%

Mining, Quarrying, and Oil and Gas Extraction

Transportation and Warehousing

1.37%

1.28%

0.82%

Utilities

Real Estate and Rental and Leasing

3.26%

1.53%
1.41%

Educational Services

Wholesale Trade

3.33%

1.64%
1.66%

Agriculture, Forestry, Fishing, and Hunting

Retail Trade

5.30%

0.98%
0.82%
0.89%
0.65%

2022

0.67%
0.59%

2021

0.58%
0.56%

1%

2%
3%
4%
AI Job Postings (% of All Job Postings)

5%
Figure 4.1.5

Table of Contents

Chapter 4 Preview

176

Artificial IntelligenceIndex Report 2023

U.S. AI Labor Demand by State
Figure 4.1.6 highlights the numberof AI job postings in the UnitedStates by state. The top three

Chapter 4: The Economy
4.1 Jobs

Number of AI Job Postings in the United States by State, 2022
Source: Lightcast, 2022 | Chart: 2023 AI Index Report

AK

ME

970

2,227
1,571

VT

2,719 34,603

NH

MA

NY

CT

RI

OH

PA

NJ

DC

MD

DE

states in terms of postings were
California (142,154), followed byWA

Texas (66,624) and New York(43,899).

MT

SD

MN

WI

MI

NE

IA

IL

IN

KS

MO

KY

WV
887

9,606 16,769 3,503

833

1,227

OR

ID

WY
769

4,032 5,670 31,569 9,247 19,208 20,397 23,447

UT

CO

AZ

NM

10,811 6,109

CA

ND

31,284

NV

142,154 6,813

2,195 11,808 8,879 25,366

6,885 20,421 7,683 10,990 4,536

OKAR

43,899 8,960 2,965

19,514 3,357

5,719

7,247

11,173 34,221 23,854

TN

VA

NC

TX

LA

MS

AL

GA

SC

66,624 4,806 2,548 7,866 26,620 4,928

HI

FL

2,550

33,585Figure 4.1.6

Figure 4.1.7 demonstrates whatpercentage of a state’s total job
postings were AI-related. The top
states according to this metric

Percentage of U.S. States Job Postings in AI, 2022
Source: Lightcast, 2022 | Chart: 2023 AI Index Report

AK
0.88%

ME
1.64%
NH MA
VT
1.34% 1.20% 2.26%

were the District of Columbia(3.0%), followed by Delaware

MI
WA MT NDSD MN WI
2.48% 0.72% 1.04% 1.83% 1.22% 0.90% 1.77%(2.7%), Washington (2.5%), and

CT
NY
RI
2.07% 1.66% 1.84%

ID
IA
IL
IN
OR
OH
PA
NJ
WY NE
1.43% 1.89% 1.18% 1.14% 1.14% 1.63% 0.88% 1.07% 1.30% 2.04%

Virginia (2.4%).

CA
NV
CO
KS MO KYUTWV DC MD DE
2.21% 1.23% 1.54% 1.46% 1.43% 1.15% 0.85% 0.99% 2.95% 1.96% 2.66%
AZ NM OK
AR
TN
NC
VA
1.40% 1.36% 1.07% 2.03% 1.11% 2.42% 1.44%
LA
MS
AL
GA
TX
SC
1.52% 0.87% 1.15% 1.31% 1.64% 0.87%
HI
1.46%

FL
1%
Figure 4.1.7

Table of Contents

Chapter 4 Preview

17 7

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.1 Jobs

Percentage of United States AI Job Postings by State, 2022

Which states had the greatest

Source: Lightcast, 2022 | Chart: 2023 AI Index Reportshare of AI job postings as ashare of all AI job postings in

AK
0.12%

the U.S. in 2022? California was

ME
0.28%
VT
NH MA
0.20% 0.34% 4.35%

first: Last year 17.9% of all AI job
postings in the United StatesMI
WA MT NDSD MN WI
3.93% 0.10% 0.15% 0.28% 1.48% 1.12% 3.19%

were for jobs based in California,

CT
NY
RI
5.52% 1.13% 0.37%

OR
ID
NE
IA
IL
IN
OH
PA
NJ
WY
1.36% 0.77% 0.10% 0.51% 0.71% 3.97% 1.16% 2.41% 2.56% 2.95%

followed by Texas (8.4%) and
New York (5.5%) (Figure 4.1.8).

CA
NV
CO
KS MO KYUTWV DC MD DE
17.87% 0.86% 0.87% 2.57% 0.97% 1.38% 0.57% 0.11% 1.21% 2.11% 0.44%
AZ NM OK
AR
TN
VA
2.45% 0.42% 0.72% 0.91% 1.40% 4.30%

NC
3%

LA
MS
AL
GA
TX
SC
8.37% 0.60% 0.32% 0.99% 3.35% 0.62%
HI
0.32%

FL
4.22%
Figure 4.1.8

Figure 4.1.9 highlights the trends over time in AI job postings for four select states that annually report a highnumber of AI-related jobs: Washington, California, New York, and Texas. For all four, there was a significant
increase in the number of total AI-related job postings from 2021 to 2022, suggesting that across these states,
employers are increasingly looking for AI-related workers.

Percentage of U.S. States’ Job Postings in AI by Select U.S. State, 2010–22
Source: Lightcast, 2022 | Chart: 2023 AI Index Report

Percentage of U.S. States’ Job Postings in AI

2.50%

2.48%, Washington
2.21%, California
2.07%, New York

2.00%

1.52%, Texas

1.50%

1.00%

0.50%

0.00%

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 4.1.9

Table of Contents

Chapter 4 Preview

178

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.1 Jobs

Figure 4.1.10 highlights the degree to which AI-related job postings have been subdivided among the top
four states over time. California’s share of all AI job postings has decreased steadily since 2019 while Texas’has marginally increased. The fact that California no longer commands one-quarter of all AI-related jobs
suggests that AI jobs are becoming more equally distributed among U.S. states.

Percentage of United States AI Job Postings by Select U.S. State, 2010–22
Source: Lightcast, 2022 | Chart: 2023 AI Index Report

Percentage of United States AI Job Postings

25%

20%
17.87%, California
15%10%
8.37%, Texas
5.52%, New York

5%

3.93%, Washington

0%

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 4.1.10

Table of Contents

Chapter 4 Preview

179

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.1 Jobs

AI Hiringin the same period the job began, divided by the total

Our AI hiring data is based on a LinkedIn dataset of skills
and jobs that appear on their platform. The countries

number of LinkedIn members in the corresponding
location. This rate is then indexed to the averagemonth in 2016; for example, an index of 1.1 in December

included in the sample make at least 10 AI hires each2021 points to a hiring rate that is 10% higher than themonth and have LinkedIn covering at least 40% of

average month in 2016. LinkedIn makes month-to-

their labor force. India is also included in the sample

month comparisons to account for any potential lags in

given their increasing significance in the AI landscape,members updating their profiles. The index for a year is

although LinkedIn does not cover 40% of their labor

the number in December of that year.

force. Therefore, the insights drawn about India should
be interpreted with particular caution.

The relative AI hiring index measures the degree to which
the hiring of AI talent is changing, more specificallyFigure 4.1.11 highlights the 15 geographic areas thatwhether the hiring of AI talent is growing faster than,have the highest relative AI hiring index for 2022. The
AI hiring rate is calculated as the percentage of LinkedIn
members with AI skills on their profile or working in
AI-related occupations who added a new employer

equal to, or more slowly than overall hiring in a particulargeographic region. In 2022, Hong Kong posted thegreatest growth in AI hiring at 1.4, followed by Spain, Italy
and the United Kingdom, and the United Arab Emirates.

Relative AI Hiring Index by Geographic Area, 2022
Source: LinkedIn, 2022 | Chart: 2023 AI Index Report

Hong Kong

1.37

Spain

1.19

Italy

1.18

United Kingdom

1.18

United Arab Emirates

1.15

South Africa

1.13

New Zealand

1.06

Denmark

1.06

Belgium

1.05

Netherlands

1.03

South Korea

1.02

Sweden1.01

Canada0.99

Switzerland

0.99

Singapore

0.99

0.00

0.20

0.40

0.60
0.80
Relative AI Hiring Index

1.00

1.20

1.40
Figure 4.1.11

Figure 4.1.12 highlights how the AI hiring index changes over time for a wide range of countries3. Overall, themajority of countries included in the sample have seen meaningful increases in their AI hiring rates since 2016.
This trend suggests that those countries are now hiring more AI talent than in 2016. However, for many countries,
AI hiring rates seem to have peaked around 2020, then dropped, and have since stabilized.
3 Both Figure 4.1.11 and Figure 4.1.12 report the Relative AI Hiring Index. Figure 4.1.11 reports the Index value at the end of December 2022, while Figure 4.1.12 reports a twelve-month rolling average.

Table of Contents

Chapter 4 Preview

180Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.1 Jobs

Relative AI Hiring Index by Geographic Area, 2016–22
Source: LinkedIn, 2022 | Chart: 2023 AI Index Report

Australia

2.00
1.50

0.50

1.00

2020

0.00

2022

Chile

1.00

1.00

2020

0.00

2022

2018

1.50

0.00

2022

1.50
1.14

1.00
0.50

0.50

0.50

0.00

0.00

Germany

2.00

2020

2022

Hong Kong

2.00

1.50

1.50
1.13

1.00
0.50
0.00

2018

0.50
2018

2020

0.00

2022

2018

Israel2020

1.00

0.94 1.00

0.50

0.50
2018

1.50

0.50

0.50

0.50

0.00

0.00

0.00

2022

New Zealand

2.00

2020

2022

Norway

2.00

1.50

0.50
2020

0.00

2022

2018

2020

Portugal

2018

2020

0.00

2022

1.50

0.50

0.50
0.00

Switzerland

2.00

2.00

1.50

United Arab Emirates

0.50
2020

Table of Contents

2022

0.00

2020

2022

United Kingdom

1.50
1.08

1.00
0.50

2018

2018

2.00

1.50
1.09

1.00

0.00

2022

0.50
2018

Chapter 4 Preview

2020

2022

0.00

2018

2020

2022

United States

2.00
1.25

1.00

1.09

1.00

0.00

2020

2022

1.50
1.12

1.00

0.50
2018

2020
Sweden

0.00

2022

2018

2.00

0.50
2020

1.13

1.00

0.00

2018

2022

Singapore

Spain

1.05

1.00

2020

0.50

2.00

1.50
1.10

2018

1.50
1.01

South Korea

1.50

2022

1.21

2.00

1.00

0.00

2022

2.00

1.00

2022

0.50

South Africa
2.00

2020

1.50
1.10

1.00
0.50

2018

2018

2.00

1.50
1.09

1.00

0.00

2018

2020

1.00

0.50
2020

2018

1.50
1.03

1.00

0.00

2018

1.08

2.00

1.50
1.11

1.00

2022

Netherlands

2.00

1.50
1.00

0.00

2022

2020
Ireland

Luxembourg

2.00

1.00

2020

2018

2.00
1.50

Italy

2.00

2022

1.50

0.00

2022

2020
India

2.00
1.21

1.00

2018

1.19

1.00

0.00

2022

2022

1.50
1.08

1.00

0.50
2020

2020

2.00

0.00

2018

2018

France

2.00

1.50
1.05

2020
Finland

2.00

1.00

1.17

1.00
0.50

Denmark

2.00

Relative AI Hiring Index

1.50

0.50
2018

Canada

2.00

1.50
1.11

0.50
2018

Brazil

2.00

1.50
1.12

1.00

0.00

Belgium

2.00

1.50
1.15

1.00
0.50

2018

2020

2022

0.00

2018

2020

2022
Figure 4.1.12

181

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.1 Jobs

rate of various countries or regions from 2015 to

AI Skill Penetration

2022. In this case, the relative AI skill penetration rate

The AI skill penetration rate is a metric created bycan be understood as the sum of the penetration of

LinkedIn that measures the prevalence of various

each AI skill across occupations in a given country or

AI-related skills across occupations. LinkedIn

region, divided by the global average across the samegenerates this metric by calculating the frequencies

occupation. For instance, a relative skill penetrationof LinkedIn users’ self-added skills in a given area

rate of 1.5 means that the average penetration of AI

from 2015 to 2022, then reweighting those numbers

skills in that country or region is 1.5 times the global

with a statistical model to create the top 50

average across the same set of occupations.

representative skills in that select occupation.

As of 2022, the three countries or regions with the

Global Comparison: Aggregate

highest AI skill penetration rates were India (3.2),

Figure 4.1.13 shows the relative AI skill penetration

the United States (2.2), and Germany (1.7).

Relative AI Skill Penetration Rate by Geographic Area, 2015–22
Source: LinkedIn, 2022 | Chart: 2023 AI Index Report

India

3.23

United States

2.23

Germany

1.72

Israel

1.65

Canada

1.54

United Kingdom

1.54South Korea

1.44

Singapore

1.37

France1.13

Brazil

0.99

Spain

0.98

Netherlands

0.95

Italy

0.95

Switzerland

0.91

Australia
0.00

0.89

0.50

1.00

1.50
2.00
Relative AI Skill Penetration Rate

2.50

3.00

Figure 4.1.13

Table of Contents

Chapter 4 Preview

182

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.1 Jobs

Global Comparison: By Gender
Figure 4.1.14 disaggregates AI skill penetration rates

pooled together across the same set of occupations

by gender across different countries or regions.

in the country. For all countries in the sample, theA country’s “Relative AI skill penetration rate

relative AI skill penetration rate is greater for men

across genders” for women of 1.5 means that female

than women. India (2.0), the United States (1.3), andmembers in that country are 1.5 times more likely to

Israel (0.9) have the highest reported relative AI skill

list AI skills than the average member in all countries

penetration rates for women.

Relative AI Skill Penetration Rate Across Gender, 2015–22
Source: LinkedIn, 2022 | Chart: 2023 AI Index Report
3.27

India

1.99
2.36

United States

1.28
2.05

Israel0.87

Canada

0.86

1.59
1.91

Germany

0.71

Singapore

0.68

1.37
1.46

United Kingdom

0.57
1.13

France0.39

Netherlands

0.38

0.98
0.82

United Arab Emirates

0.31

Italy

0.31

Brazil

0.30

Spain

0.30

Finland

0.29

Australia

0.29

0.00

0.85
0.98
1.03

Male

1.08

Female
0.88

0.50

1.00

1.50
2.00
Relative AI Skill Penetration Rate

2.50

3.00
Figure 4.1.14

Table of Contents

Chapter 4 Preview

183

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

Using data from NetBase Quid, this section tracks trends in AI-related investments. NetBase Quid tracks data on the investments of over
8 million global public and private companies. NetBase Quid also uses natural language processing techniques to search, analyze, andidentify patterns in large, unstructured datasets, like aggregated news and blogs, and company and patent databases. NetBase Quid
continuously broadens the set of companies for which it tracks data, so that in this year’s AI Index, the reported investment volume forcertain years is larger than that of previous reports.

4.2 Investment
Corporate Investment
As AI becomes more and more integrated into the

For the first time since 2013, year-over-year global

economy, it becomes increasingly important to track

corporate investment in AI has decreased. In 2022,

AI-related corporate investment. Figure 4.2.1 shows

total global corporate AI investment was $189.6

overall global corporate investment in AI from 2013

billion, roughly a third lower than it was in 2021.

to 2022. Corporate investment includes mergers andStill, in the last decade, AI-related investment has

acquisitions, minority stakes, private investment, andincreased thirteenfold.

public offerings.

Global Corporate Investment in AI by Investment Activity, 2013–22
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report300

Merger/Acquisition
276.14

Minority Stake
Private Investment
Total Investment (in Billions of U.S. Dollars)

250

Public O ering

119.66200

189.59146.74

150

83.35

26.06
95.57

100
79.62
53.72

50
14.57
0

2013

19.04
2014

25.43

33.82
13.05

24.68

13.35

17.13

25.72

2015

2016

2017

46.06

125.36

31.91

21.89

43.1

55.09

12.62

2018

13.01

2019

91.86

61.61

2020

29.1
2021

2022
Figure 4.2.1

Table of Contents

Chapter 4 Preview

184

Artificial IntelligenceIndex Report 2023

To provide a fuller context for thenature of AI investment in the last year,
Figures 4.2.2 through 4.2.5 highlightthe top merger/acquisition, minority
stake, private investment, and public

Chapter 4: The Economy
4.2 Investment

Top Five AI Merger/Acquisition Investment Activities, 2022
Source: NetBase Quid, 2022 | Table: 2023 AI Index ReportCompany Name

Headquarters
Country

Focus Area

NuanceCommunications, Inc.

United States

Arti cial Intelligence;Enterprise Software;
Healthcare; Machine
Learning

19.80

Citrix Systems, Inc.

United States

Data Management,
Processing, and Cloud;HR Tech

17.18

Avast Limited

Czech Republic

Data Management,
Processing, and Cloud;Fintech; Cybersecurity,
Data Protection

8.02

AspenTech
Corporation

United States

Manufacturing;
Software; Supply
Chain Management

6.34

Vivint Smart Home,Inc.

United States

Cybersecurity, Data
Protection; Sales
Enablement

5.54

offering events in the last year. The
greatest single AI investment event
was the merger/acquisition of Nuance
Communications, valued at $19.8 billion(Figure 4.2.2). The largest minority
stake event was for the British company
Aveva Group ($4.7 billion) (Figure 4.2.3).
The greatest private investment event
was GAC Aion New Energy Automobile($2.5 billion), a Chinese clean energy

Funding Amount(in Billions USD)

and automotive company (Figure 4.2.4).
Finally, the largest public offering was

Figure 4.2.2

ASR Microelectronics ($1.1 billion),a Chinese semiconductor company

Top Five AI Minority Stake Investment Activities, 2022
Source: NetBase Quid, 2022 | Table: 2023 AI Index Report(Figure 4.2.5).

Company Name

Headquarters
Country

Focus Area

Funding Amount(in Billions USD)AVEVA Group, PLC

UnitedKingdom

Chemical; Computer;
Data Mining; Electronics;
Industrial Manufacturing;
Information Technology;
Simulation; Software

4.68

Grupo deInversiones
Suramericana, SA

Colombia

Financial Services; Impact
Investing; Insurance

1.48

Fractal Analytics
Private Limited

India

Analytics; Arti cialIntelligence; Big Data;
Business Intelligence;
Consulting; Machine
Learning

0.35

Atrys Health, SA

Spain

Medical and Healthcare

0.28

R Systems
International, Ltd.

India

Analytics; Information
Technology; IT
Management; Software

0.17

Figure 4.2.3

Table of Contents

Chapter 4 Preview

185

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

Top Five AI Private Investment Activities, 2022
Source: NetBase Quid, 2022 | Table: 2023 AI Index ReportCompany Name

Headquarters
Country

Focus Area

Funding Amount(in Billions USD)GAC Ai¬¥an New
Energy Automobile
Co., Ltd.

China

Automotive; Clean
Energy; Electric Vehicle;
Manufacturing

2.54

Idience Co., Ltd.

South Korea

Emergency Medicine;
Healthcare;
Pharmaceutical

2.15

Uali

Argentina

Drones; Cloud Computing

1.50

Anduril Industries,
Inc.

United States

Cybersecurity, Data
Protection; AR/VR;
Drones

1.50

Celonis, GmbHGermany

Retail; Industrial
Automation, Network; HR
Tech; Insurtech

1.22

Figure 4.2.4

Top Five AI Public O ering Investment Activities, 2022
Source: NetBase Quid, 2022 | Table: 2023 AI Index ReportCompany Name

Headquarters Focus Area
Country

Funding Amount(in Billions USD)

ASR Microelectronics
Co., Ltd.

China

Semiconductor; VC

1.08

iSoftStone Information
Technology (Group)Co., Ltd.

China

Data Management,
Processing, and Cloud;Cybersecurity, Data
Protection

0.73

Jahez International
Company for
Information Systems
Technology

Saudi Arabia

Arti cial Intelligence;
E-Commerce; Food andBeverage; Food Delivery;
Information Technology;
Logistics

0.43

Fortior Technology(Shenzhen) Co., Ltd.

China

Electronics; Machine
Manufacturing;
Semiconductor

0.30

Beijing Deep Glint
Technology Co., Ltd.

China

Cybersecurity, Data
Protection; Music, Video
Content

0.29

Figure 4.2.5

Table of Contents

Chapter 4 Preview

186

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

Startup Activity

Global Trend

The next section analyzes private investment trends in

The global private AI investment trend reveals that

artificial intelligence startups that have received overwhile investment activity has decreased since 2021, it$1.5 million in investment since 2013.

is still 18 times higher than it was in 2013 (Figure 4.2.6).

Private Investment in AI, 2013–22

Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

Total Investment (in Billions of U.S. Dollars)

120

100
91.86

80

60

40

20

0

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 4.2.6

Table of Contents

Chapter 4 Preview

187

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

A similar trend, of short-term decreases but longer-

decrease from 2021 but a sixfold increase since 2013

term growth, is evident in data on total private(Figure 4.2.7). Similarly, the number of newly funded AI

investment events. In 2022 there were 3,538 AI-

companies dropped to 1,392 from 1,669 last year, while

related private investment events, representing a 12%

having increased from 495 in 2013 (Figure 4.2.8).

Number of Private Investment Events in AI, 2013–22
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

4,000
3,538

Number of Private Investment Events

3,500

3,000

2,500

2,000

1,500

1,000

500

0

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 4.2.7

Number of Newly Funded AI Companies in the World, 2013–22
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

1,600
1,392

1,400

Number of Companies

1,200

1,000

800

600

400

200

0

2013

Table of Contents

2014

2015

Chapter 4 Preview

2016

2017

2018

2019

2020

2021

2022
Figure 4.2.8

188

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

The year-over-year decrease in AI-related
funding is also evident when the funding events

AI Private Investment Events by Funding Size,
2021 Vs. 2022
Source: NetBase Quid, 2022 | Table: 2023 AI Index Report

are disaggregated by size. Across all size

Funding Size

categories, with the exception of ones overOver $1 Billion

4

6$1 billion, the total number of AI funding events$500 Million – $1 Billion

13

5

18

$100 Million – $500 Million

277

164

441

decreased (Figure 4.2.9).

2021$50 Million – $100 Million
Under $50 Million
Undisclosed
Total

2022

Total
10

277

238

515

2,851

2,585

5,436

598

540

1,138

4,020

3,538

7,558

Figure 4.2.9

Regional Comparison by Funding AmountOnce again, the United States led the world in terms of total AI private investment. In 2022, the $47.4 billion
invested in the United States was roughly 3.5 times the amount invested in the next highest country, China($13.4 billion), and 11 times the amount invested in the United Kingdom ($4.4 billion) (Figure 4.2.10).

Private Investment in AI by Geographic Area, 2022
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

United States

47.36

China

13.41

United Kingdom

4.37

Israel

3.24

India

3.24

South Korea

3.10

Germany

2.35

Canada1.83

France1.77

Argentina

1.52

Australia

1.35

Singapore

1.13

Switzerland

1.04

Japan

0.72

Finland0.61

0Table of Contents

5

10Chapter 4 Preview

15

20
25
30
Total Investment (in Billions of U.S. Dollars)

35

40

45
Figure 4.2.10

189

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

When private AI investments are aggregated since 2013, the same ranking of countries applies:
The United States is first with $248.9 billion invested, followed by China ($95.1 billion) and theUnited Kingdom ($18.2 billion) (Figure 4.2.11).

Private Investment in AI by Geographic Area, 2013–22 (Sum)
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

United States

248.90

China

95.11

United Kingdom

18.24

Israel10.83

Canada

8.83

India

7.73

Germany

6.99

France

6.59

South Korea

5.57

Singapore

4.72

Japan

3.99

Hong Kong

3.10

Switzerland

3.04

Australia

3.04

Spain 1.81
0Table of Contents

20

40

60

Chapter 4 Preview

80

100
120
140
160
Total Investment (in Billions of U.S. Dollars)

180

200

220

240
Figure 4.2.11

190

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

While the United States continues to outpace

The top five American AI private investment events

other nations in terms of private AI investment,

are highlighted in Figure 4.2.13, the top five European

the country experienced a sharp 35.5% decrease

Union and British investments in Figure 4.2.14, and thein AI private investment within the last year (Figure

top five Chinese investments in Figure 4.2.15.

4.2.12). Chinese investment experienced a similarly
sharp decline (41.3%).

Private Investment in AI by Geographic Area, 2013–22
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

Total Investment (in Billions of U.S. Dollars)

70

60

50

47.36, United States

40

30

20
13.41, China
11.04, European Union and United Kingdom

10

0
2013

2014

Table of Contents

2015

2016

2017

Chapter 4 Preview

2018

2019

2020

2021

2022

Figure 4.2.12

191

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

Top AI Private Investment Events in the UnitedStates, 2022

Top AI Private Investment Events in the European
Union and United Kingdom, 2022

Company Name

Focus Area

Funding Amount(in Billions USD)

Company Name

Focus Area

Anduril Industries, Inc.

Cybersecurity, Data
Protection; AR/VR;
Drones

1.50

Celonis, GmbH

Retail; Industrial
Automation, Network;
HR Tech; Insurtech

1.22

Faire Wholesale, Inc.

Fintech; Retail; Sales
Enablement

0.82

Content Square, SAS

0.60

Anthropic, PBC

Arti cial Intelligence;
Information
Technology; Machine
Learning

0.58

Analytics; Arti cialIntelligence: CRM:
Data Visualization;
Digital Marketing;
SaaS
Retail

0.57

Data Management,
Processing, and Cloud;Cybersecurity, Data
Protection

0.40

Retail Logistics Excellence- RELEX Oy
Cera Care Limited

Medical and
Healthcare

0.32Babylon Holdings Limited

0.40

Medical and
Healthcare; Music,
Video Content

0.30

Data Management,
Processing, and Cloud;
AV; AR/VR

Source: NetBase Quid, 2022 | Table: 2023 AI Index Report

Arctic Wolf Networks, Inc.

JingChi, Inc.

Source: NetBase Quid, 2022 | Table: 2023 AI Index Report

Funding Amount(in Billions USD)

Figure 4.2.14
Figure 4.2.13

Top AI Private Investment Events in China, 2022

Source: NetBase Quid, 2022 | Table: 2023 AI Index ReportCompany Name

Focus Area

Funding Amount(in Billions USD)GAC Ai¬¥an New Energy
Automobile Co., Ltd.

Automotive; Clean
Energy; Electric
Vehicle;Manufacturing

2.54

GAC Ai¬¥an New Energy
Automobile Co., Ltd.

Automotive; Clean
Energy; Electric
Vehicle;Manufacturing

1.11

Beijing ESWIN
Technology Group Co.,Ltd.

Data Management,
Processing, and Cloud;Industrial Automation,
Network;
Semiconductor;
Marketing, Digital Ads;
Sales Enablement

0.58

Zhejiang Hozon New
Energy Automobile Co.,Ltd.

Data Management,
Processing, and Cloud;Cybersecurity, Data
Protection; Sales
Enablement

0.44

Zhejiang Hozon New
Energy Automobile Co.,Ltd.

Data Management,
Processing, and Cloud;Cybersecurity, Data
Protection; Sales
Enablement

0.32

Figure 4.2.15

Table of Contents

Chapter 4 Preview

192

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

Regional Comparison by Newly Funded
AI Companies
This subsection studies the number of newly funded

United States led all regions with the largest number of

AI companies across various geographic areas.

newly funded AI companies at 542, followed by China

As was the case with private investment, theat 160 and the United Kingdom at 99 (Figure 4.2.16).

Number of Newly Funded AI Companies by Geographic Area, 2022
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

United States

542

China

160

United Kingdom

99

Israel

73

India

57

Canada47

France44

Germany

41

Singapore

36

Japan

32

Switzerland

26

Australia

23

South Korea

22

Sweden12

Netherlands

120Table of Contents

100

Chapter 4 Preview

200

300
Number of Companies

400

500
Figure 4.2.16

193

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

A similar trend is evident in the aggregate data since 2013. In the last decade, the number of newly funded
AI companies in the United States is around 3.5 times the amount in China, and 7.4 times the amount in the
United Kingdom (Figure 4.2.17).

Number of Newly Funded AI Companies by Geographic Area, 2013–22 (Sum)
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

United States

4,643

China

1,337

United Kingdom

630

Israel402

Canada341

France

338

India

296

Japan

294

Germany

245

Singapore

165

South Korea

145

Australia

126

Switzerland

108

Sweden83

Netherlands

78

0

500

Figure 4.2.18 breaks
down data on newly

1,000

1,500

2,000
2,500
3,000
Number of Companies

3,500

4,000

4,500
Figure 4.2.17

Number of Newly Funded AI Companies by Geographic Area, 2013–22
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

700

funded AI companies
within select600

geographic regions.
In a trend that goesthe United States
continues to outpace
both the European
Union and the United
Kingdom, as well as

500
Number of Companies

back a decade,542,
United States

400

300

293,
European
Union and
United Kingdom

200

160,
China

China. However, thegrowth rates of the100

different regions are
relatively similar.

0
2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 4.2.18

Table of Contents

Chapter 4 Preview

194

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

Private Investment in AI by Focus Area, 2021 Vs. 2022

Focus Area Analysis
Private AI investment can also be disaggregated byfocus area. Figure 4.2.19 compares global private
AI investment by focus area in 2022 versus 2021.
The focus areas that attracted the most investment
in 2022 were medical and healthcare ($6.1 billion);
data management, processing, and cloud ($5.9
billion); fintech ($5.5 billion); cybersecurity anddata protection ($5.4 billion); and retail ($4.2
billion). Mirroring the pattern seen in total AI private
investment, the total investment across most focus
areas declined in the last year.

Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

Medical and Healthcare
Data Management, Processing, Cloud
FintechCybersecurity, Data Protection
RetailIndustrial Automation, Network
Sales Enablement
Marketing, Digital Ads
AR/VR
DronesInsurtechMusic, Video Content
Semiconductor
HR TechEnergy, Oil, and Gas
AV
NLP, Customer Support
AgritechEntertainment
Legal Tech
Geospatial
Fitness and Wellness
Ed TechFacial Recognition
VC

2022
2021

0

2

4

6

8

10

Total Investment (in Billions of U.S. Dollars)

Figure 4.2.19

Figure 4.2.20 presents trends in AI focus area

cybersecurity and data protection, drones, marketinginvestments. As noted earlier, most focus areas saw

and digital ads, HR tech, AR/VR, and legal tech. Still,

declining investments in the last year. However, somemirroring a broader trend in AI private investment,of the focus areas that saw increased investments aremost focus areas saw greater amounts of AI private

semiconductor, industrial automation and network,

investment in 2022 than they did in 2017.

Table of Contents

Chapter 4 Preview

195

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

Private Investment in AI by Focus Area, 2017–22

Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

Data Management, Processing, Cloud
8
6
4
2
0

5.86

2018

2020

2022

Medical and Healthcare
8
6
4
2
0

Semiconductor
8
6
4
2
0

2020

2022

8
6
4
2
0

Total Investment (in Billions of U.S. Dollars)

NLP, Customer Support
8
6
4
2
0

2020

2020

3.92
2018

2020

2022

8
6
4
2
0

2020

2022

8
6
4
2
0

Agritech
8
6
4
2
0

2020

2020

2018

2022

8
6
4
2
0

2020

2022

2018

2020

2022

8
6
4
2
0

2022

8
6
4
2
0

2020

2022

5.38

2018

2020

8
6
4
2
0

2022

2020

2022

8
6
4
2
0

2022

8
6
4
2
0

0.53
2018

2020

2022

2022

8
6
4
2
0

1.88
2018

2020

2020

2022

8
6
4
2
0

1.74
2018

2020

2022

8
6
4
2
0

0.37
2018

2020

2022

Music, Video Content

0.87
2018

2022

DronesEntertainment
8
6
4
2
0

2020

Ed Tech

2.39
2018

2022

Insurtech

0.07
2018

2020

Fitness and Wellness

AR/VR

0.83
2018

2020

1.34
2018

Facial Recognition

3.18
2018

2022

4.20

Legal Tech

0.71
2018

2020

2020

Cybersecurity, Data Protection

1.63

Geospatial
8
6
4
2
0

8
6
4
2
0

Sales Enablement

0.87
2018

2018

8
6
4
2
0

RetailHR Tech

3.05
2018

2022

1.61
2018

Marketing, Digital Ads
8
6
4
2
0

2022

Energy, Oil, and Gas

1.01
2018

2018

AV
5.52

Industrial Automation, Network

1.65
2018

6.05

Fintech
8
6
4
2
0

2022

8
6
4
2
0

1.72
2018

2020

2022

VC
8
6
4
2
0

0.02
2018

2020

Table of Contents

2022

Chapter 4 Preview

Figure 4.2.20

196

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.2 Investment

Finally, 4.2.21 shows private investment in AI by focus areabillion), and 40 times more than that in the European

over time within select geographic regions, highlighting

Union and the United Kingdom ($0.04 billion). Chinese

how private investment priorities in AI differ acrossprivate investment in AI-related semiconductors ($1.02

geographies. For example, in 2022, private investment

billion) was 1.75 times more than that in the United

in AI-related drone technology in the United States ($1.6States ($0.58 billion), and 102 times more than that in thebillion) was nearly 53 times more than that in China ($0.03

European Union and the United Kingdom ($0.01 billion).

Private Investment in AI by Focus Area and Geographic Area, 2017–22

Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

Data Management, Processing, Cloud
Medical and Healthcare
US, 3.13
US, 4.19
CN, 1.87
CN, 0.25
4
4
EU/UK, 0.24
EU/UK, 0.76 4
2
0

2
2018

2020

Semiconductor
4

0

2022

2018

2020

2022

0

2018

2020

2022

Retail0

US, 1.52
CN, 0.01
EU/UK, 2.07 4

2

2

2

2

0

0

0

2020

2022

2
0

2018

2022

Energy, Oil, and Gas

2018

2020

0

2022

2
2020
Agritech
4

0

2022

US, 0.55
CN, 0.10
EU/UK, 0.08 4

2020

2022

2
2018

2020

2022

HR Tech0

US, 0.24
CN, 0.00
EU/UK, 1.28 4

2
2018

2018

2020

2022

Sales Enablement

0

2018

2020

2022

Facial Recognition

0

2018

2020
AR/VR

US, 1.12
CN, 1.68
EU/UK, 0.16 4

2022

0

2

2

2

0

0

Geospatial
4

US, 0.55
CN, 0.03
EU/UK, 0.01 4

2

2

0

2018

2020
VC

4

0

2022

2018

2020
Legal Tech

2022

US, 0.71
CN, 0.05
EU/UK, 0.06 4

2018

2020

Entertainment

2022

US, 0.47
CN, 0.18
EU/UK, 0.17 4

2
2018

2020

2022

0

2020
Drones2018

2020
Insurtech

2018

2020
Ed TechUS, 2.07
CN, 0.01
EU/UK, 0.06 4

0

2022

2018

US, 0.23
CN, 0.00
EU/UK, 0.142022
US, 1.60
CN, 0.03
EU/UK, 0.04

2022
US, 0.39
CN, 0.00
EU/UK, 1.29

2

2
2020

2022

Fitness and Wellness

US, 0.07
CN, 0.00
EU/UK, 0.00 4

0

2018

2020

2

2
2018

2018

Cybersecurity, Data Protection
US, 0.80
US, 3.87
CN, 0.34
CN, 1.07
EU/UK, 0.20 4
EU/UK, 0.23 42

Marketing, Digital Ads
US, 1.14
CN, 0.88
4
EU/UK, 0.76 4

0

2020

US, 0.69
CN, 0.49
EU/UK, 0.02

2

0

2018

AV

US, 3.23
CN, 0.03
EU/UK, 0.94 4

2

Industrial Automation, Network
US, 0.58
US, 0.87
CN, 1.02
CN, 1.06
4
EU/UK, 0.01
EU/UK, 1.65 4

NLP, Customer Support
US, 0.69
CN, 0.13
4
EU/UK, 0.04 4

Total Investment (in Billions of U.S. Dollars)Fintech

2018

2020

2022
US, 0.12
CN, 0.01
EU/UK, 0.10

2022Music, Video Content
US, 1.10
CN, 0.03
EU/UK, 0.44

2
2018

2020

2022

0

2018

2020

2022

US, 0.00
CN, 0.00
EU/UK, 0.02

2
0

2018

2020

Table of Contents

Figure 4.2.21

2022

Chapter 4 Preview

197

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

This section explores how corporations tangibly use AI. First, it highlights industry adoption trends and asks how businesses adopt
AI and what particular AI technologies they find most useful, and identifies how AI adoption affects their bottom line. Second, thesection considers industry motivations and explores what questions industry leaders consider when thinking about incorporating AI
technologies. Finally, it paints a qualitative picture of business AI use by examining trends in AI-related earnings calls.

4.3 Corporate Activity
Industry Adoption

Adoption of AI Capabilities

The following subsection on the industry adoption

According to the most recent McKinsey report, as ofof AI borrows data from McKinsey’s “The State of2022, 50% of surveyed organizations reported having

AI in 2022—and a Half Decade in Review,” as welladopted AI in at least one business unit or function

as previous years’ editions. The 2022 report drew on(Figure 4.3.1). This total is down slightly from 56% indata from a survey of 1,492 participants representing

2021, although up significantly from 20% in 2017. AI

a wide range of regions, industries, company sizes,

usage has rapidly grown in the past half-decade, butfunctional specialties, and tenures.

leveled off since 2020.

Share of Respondents Who Say Their Organizations Have Adopted AI in at Least One Function, 2017–22
Source: McKinsey & Company Survey, 2022 | Chart: 2023 AI Index Report

60%

50%

50%

% of Respondents

40%

30%

20%

10%

0%

2017

2018

2019

2020

2021

2022
Figure 4.3.1

Table of Contents

Chapter 4 Preview

198

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

In the last half-decade, the average number of AI capabilities that organizations have embedded
has doubled from 1.9 in 2018 to 3.8 in 2022 (Figure 4.3.2). Some of the AI capabilities that McKinsey
features in their survey include recommender systems, NL text understanding, and facial recognition.4

Average Number of AI Capabilities That Respondents’ Organizations Have Embedded Within at Least One
Function or Business Unit, 2018–22
Source: McKinsey & Company Survey, 2022 | Chart: 2023 AI Index Report

4.00
3.80

Number of AI Capabilities (Average)

3.50

3.00

2.50

2.00

1.50

1.00

0.50

0.00

2018

2019

2020

2021

2022
Figure 4.3.2

4 In the 2022 edition of the McKinsey survey, 16 total AI capabilities are considered: computer vision, deep learning, digital twins, facial recognition, GAN, knowledge graphs,
NL generation, NL speech understanding, NL text understanding, physical robotics, recommender systems, reinforcement learning, robotic process automation, transfer
learning, transformers, and virtual agents.

Table of Contents

Chapter 4 Preview

199

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

The most commonly adopted AI use case in 2022 was service operations optimization (24%), followedby the creation of new AI-based products (20%), customer segmentation (19%), customer service
analytics (19%), and new AI-based enhancement of products (19%) (Figure 4.3.3).

Most Commonly Adopted AI Use Cases by Function, 2022
Source: McKinsey & Company Survey, 2022 | Chart: 2023 AI Index Report

Service Operations Optimization

24%

Creation of New AI-Based Products

20%

Customer Segmentation

19%

Customer Service Analytics

19%

New AI-Based Enhancements of Products

19%

Customer Acquisition and Lead Generation

17%

Product Feature Optimization

16%

Contact-Center Automation

16%

Risk Modeling and Analytics

15%

Service Operations
Product and/or
Service Development
Marketing and Sales

Predictive Service and Intervention
0%

Risk

14%
4%

8%

12%
% of Respondents

16%

20%

24%

Figure 4.3.3

Table of Contents

Chapter 4 Preview

200Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

With respect to the type of AI capabilities embedded

rates of embedding were 48%, 47%, and 46%. Acrossin at least one function or business unit, as indicated

all industries, the most embedded AI technologies

by Figure 4.3.4, robotic process automation had

were robotic process automation (39%), computerthe highest rate of embedding within high tech/vision (34%), NL text understanding (33%), and virtual

telecom, financial services and business, and legal

agents (33%).

and professional services industries—the respective

AI Capabilities Embedded in at Least One Function or Business Unit, 2022

20%

39%

16%

11%

33%

Business, Legal, and32%
Professional Services

37%

31%

11%

8%

26%

12%

22%

34%

19%

23%

26%

46%

16%

11%

30%

Consumer Goods/
33%
Retail

36%

25%

19%

13%

18%

20%

11%

22%

24%

32%

19%

25%

7%

11%

40%

Financial Services 24%

22%

18%

24%

13%

29%

20%

30%

42%

14%

30%

19%

47%

17%

12%

33%

Healthcare Systems/
Pharma and 32%
Med. Products

18%

16%

5%

5%

14%

5%

12%

29%

11%

16%

13%

16%

9%

6%

14%

High Tech/Telecom 37%

45%

24%

16%

15%

23%

24%

29%

40%

15%

34%

23%

48%

22%

15%

43%

ow
Kn

m
co
Re

i
Re

L

L

L

de

en
s

ng

di

ng

di

an

st

s

an

ic
ot

b
Ro

st

er

er

d
Un

n

io

ph

ra

G

d
Un

h

ec

e

at

er

xt
Te

e
Sp

en

G

dg

le
n

tio

ni

s

in

og
ec
lR

ia

g

in

n

sio

Vi

rn

a
Le

w
lT

ita

ig

p

ee

r
te
pu

)
s
-3
nt
PT
ge
,G
lA
g.
ua
e.
rt
s(
Vi
er
m
or
sf
n
ng
io
an
ni
at
Tr
ar
om
Le
ut
er
sf
sA
an
es
Tr
g
oc
in
Pr
rn
tic
ea
bo
tL
en
Ro
s
m
m
te
ce
or
ys
nf
rS

25%

m

20%

al
ic
ys
Ph

33%

N

23%

N

18%

N

25%

AN

11%

c
Fa

18%

D

24%

D

30%

om
C

All Industries 34%

G

Industry

Source: McKinsey & Company Survey, 2022 | Chart: 2023 AI Index Report% of Respondents (AI Capability)
Figure 4.3.4

Table of Contents

Chapter 4 Preview

201

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Figure 4.3.5 shows AI adoption by industry and AI function in 2022. The greatest adoption was in risk for hightech/telecom (38%), followed by service operations for consumer goods/retail (31%) and product and/or service
development for financial services (31%).

AI Adoption by Industry and Function, 2022

Industry

Source: McKinsey & Company Survey, 2022 | Chart: 2023 AI Index Report

All Industries

11%

8%

5%

10%

19%

19%

21%

9%

Business, Legal, andProfessional Services

11%

10%

9%

8%

16%

20%

19%

12%

Consumer Goods/
Retail

14%

4%

3%

4%

15%

31%

29%

11%

Financial Services

1%

8%

7%

31%

17%

24%

23%

2%

Healthcare Systems/
Pharma andMed. Products

15%

7%

2%

4%

22%

12%

8%

8%

High Tech/Telecom

6%

6%

4%

7%

38%

21%

25%

8%

Ris
Pro
k
Se ductrviancane D d/o
dS
r
ev
ale
elo
s
pm
en
t

Se
rv

Hu

ma

nR
eso
urc
es

Ma
nu
fa

ctu

rin

Ma
rke
ti
g

ng

ice

Str
Co ateg
rpo y aera
rat nd
eF
tio
ns
ina
n

Op

ce

Su
p
Ma plyna Ch
a
ge
me in
nt

% of Respondents (Function)
Figure 4.3.5

Table of Contents

Chapter 4 Preview

202Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Figure 4.3.6 shows how rates of AI adoption by

points); followed by high tech/telecom, for risk

industry and AI function vary from 2021 to 2022

(22 percentage points). The most significant

in order to demonstrate how rates of AI adoption

decreases were in high tech/telecom, for producthave changed over the last year. The greatest year-

and/or service development (38 percentage points);

over-year increases were in consumer goods/retail,and healthcare systems, also for product and/or

for strategy and corporate finance (25 percentage

service development (25 percentage points).

Percentage Point Change in Responses of AI Adoption by Industry and Function 2021 Vs. 2022

Industry

Source: McKinsey & Company Survey, 2022 | Chart: 2023 AI Index Report

All Industries

2%

-4%

-15%

-13%

6%

-6%

12%

-4%

Business, Legal, andProfessional Services

-3%

2%

-19%

-7%

3%

-6%

11%

-1%

Consumer Goods/
Retail

12%

-14%

-19%

-13%

14%

16%

25%

-7%

Financial Services

-9%

4%

-17%

11%

-15%

-16%

10%

-6%

Healthcare Systems/
Pharma andMed. Products

6%

-4%

-12%

-25%

9%

-5%

-4%

-1%

High Tech/Telecom

-6%

-5%

-24%

-38%

22%

-13%

15%

-8%

Ris
Pro
k
Se duct
rvi
a
nd
ce
an
De /or
dS
ve
ale
lop
s
me
nt

Se
rv

Hu

ma

nR
eso
urc
es

Ma
nu
fa

ctu

rin

Ma
rke
ti
g

ng

ice

Str
Co ateg
rpo y aera
rat nd
eF
tio
ns
ina
n

Op

ce

Su
p
Ma plyna Ch
a
ge
me in
nt

Percentage Point Change in Responses (Function)
Figure 4.3.6

Table of Contents

Chapter 4 Preview

203Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Organizations report AI adoption leading to both

(Figure 4.3.7). On the revenue side, the functions thatcost decreases and revenue increases. On the costmost respondents saw increases in as a result of AI

side, the functions that most respondents saw

adoption were marketing and sales (70%), productdecreases in as a result of AI adoption were supply

and/or service development (70%), and strategy andchain management (52%), service operations (45%),

corporate finance (65%).

strategy and corporate finance (43%), and risk (43%)

Cost Decrease and Revenue Increase From AI Adoption by Function, 2021
Source: McKinsey & Company Survey, 2022 | Chart: 2023 AI Index Report

Decrease by <10%

Decrease by 10–19%

Service Operations

Decrease by ≥20%

Increase by >10%45%

10% 6%

10%

7%

10%

18%

33%
31%

29%

42%

Manufacturing

32%

Increase by 6–10%
10%

25%

14%

13%

Marketing and Sales

28%

21%

9%

20%

Function29%

43%

Supply Chain Management

52%

41%

30%

Product and/or Service Development
Strategy and Corporate Finance
Average Across All Activities

30%

43%

31%

32%

10%

7%

14%

17%

13%

24%

6%
8%

23%

61%
58%
70%

41%

8%

20%

57%

37%

Human Resources

Risk

Increase by ≤5%

11%

48%

27%

59%

28%

70%

33%

8%

16%

41%

65%

8%

19%

36%

63%

% of Respondents
Figure 4.3.7

Table of Contents

Chapter 4 Preview

204Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Figure 4.3.8 shows AI adoption by organizations

across all geographies was 50%, down 6% from 2021.

globally, broken out by regions of the world. In 2022,

Notably, “Greater China” registered a 20 percentage

North America led (59%), followed by Asia-Pacificpoint decrease from 2021.

(55%) and Europe (48%). The average adoption rate

AI Adoption by Organizations in the World, 2021 Vs. 2022
Source: McKinsey & Company Survey, 2022 | Chart: 2023 AI Index Report50%

All Geographies

56%
55%

Asia-Paci c

64%
48%

Europe51%
59%

North America

55%

Greater China(incl. Hong Kong,
Taiwan)41%
61%

Developing Markets(incl. India,
Latin America,
MENA)
0%

2022

44%

2021
52%

10%

20%

30%
40%
% of Respondents

50%

60%
Figure 4.3.8

Table of Contents

Chapter 4 Preview

205

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Consideration and Mitigation of Risks From
Adopting AI

risks were regulatory compliance (45%), personal/

As has been the case in the last few iterations of the

The least salient risks identified by organizations were

McKinsey report, in 2022 respondents identified

national security (13%) and political stability (9%).

individual privacy (40%), and explainability (37%).

cybersecurity as the most relevant risk when adopting
AI technology (59%) (Figure 4.3.9). The next most cited

Risks From Adopting AI That Organizations Consider Relevant, 2019–22
Source: McKinsey & Company Survey, 2022 | Chart: 2023 AI Index Report

60%

59%, Cybersecurity50%% of Respondents

45%, Regulatory Compliance40%

40%, Personal/Individual Privacy
37%, Explainability

30%

32%, Organizational Reputation
30%, Equity and Fairness
28%, Workforce/Labor Displacement

20%

20%, Physical Safety
13%, National Security

10%

0%

9%, Political Stability

2019

2020

2021

2022
Figure 4.3.9

Table of Contents

Chapter 4 Preview

206

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Figure 4.3.10 highlights the AI risks that organizations

have taken steps to mitigate. For instance, there isare taking steps to mitigate. The top three responses

a gap of 8 percentage points for cybersecurity, 9were cybersecurity (51%), followed by regulatory

percentage points for regulatory compliance, and 12

compliance (36%) and personal/individual privacy

percentage points for personal/individual privacy.

(28%). As was the case in previous years, there are

These differences suggest there is a gap between

meaningful gaps between the risks organizations

the awareness organizations have of various risks andcite as relevant and those which organizations

their steps taken to mitigate such risks.

Risks From Adopting AI That Organizations Take Steps to Mitigate, 2019–22
Source: McKinsey & Company Survey, 2022 | Chart: 2023 AI Index Report

51%, Cybersecurity

50%

40%

% of Respondents

36%, Regulatory Compliance30%

28%, Personal/Individual Privacy
22%, Organizational Reputation
22%, Explainability

20%

18%, Workforce/Labor Displacement
17%, Equity and Fairness
15%, Physical Safety

10%7%, National Security
4%, Political Stability
0%
2019

2020

2021

2022
Figure 4.3.10

Table of Contents

Chapter 4 Preview

207

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Narrative Highlight:

The Effects of GitHub’s Copilot on Developer
Productivity and HappinessIn 2021, launched a technical preview of Copilot,
a generative AI tool that enables developers andcoders to present a coding problem in natural
language and then have Copilot generate asolution in code. Copilot can also translate
between various programming languages. In
2022, GitHub surveyed over 2,000 developers
who were using the tool to determine its effect on
their productivity, well-being, and workflow.5
Figure 4.3.11 summarizes the results of the survey.
Developers overwhelmingly reported feelingmore productive, satisfied, and efficient whenworking with Copilot. More specifically, 88% of
surveyed respondents commented feeling moreproductive, 74% reported being able to focus on
more satisfying work, and 88% claimed to havecompleted tasks more quickly. One software
engineer stated, “[With Copilot] I have to think

It took the developers
using Copilot only 71
minutes to complete theirtask—56% less time thanthe developers who did not
use Copilot (161 minutes).
reported a completion rate of 78%, 8 percentage
points higher than those who did not use Copilot.
Likewise, it only took the developers using Copilot
71 minutes to complete their task, which was 56%
less time than the developers who did not use
Copilot (161 minutes). These survey and experimentresults are evidence of the tangible ways in which
AI tools improve worker productivity.

less, and when I have to think, it’s the fun stuff. It
sets off a little spark that makes coding more fun
and more efficient.”6
As part of the same survey, GitHub recruited
95 developers and randomly split them into twogroups, one of which used Copilot as part of acoding task and the other which did not. The
results of this experiment are summarized in
Figure 4.3.12. The developers who used Copilot

5 Most of the developers surveyed, around 60%, were professional developers; 30% were students and 7% were hobbyists.
6 The quote is taken from this source.

Table of Contents

Chapter 4 Preview

208

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Narrative Highlight:

The Effects of GitHub’s Copilot on Developer
Productivity and Happiness (cont’d)
Measuring Dimensions of Developer Productivity When Using Copilot: Survey Responses, 2022
Source: GitHub Survey, 2022 | Chart: 2023 AI Index Report

I Am More Productive

88%

Focus on More Satisfying Work74%

More Ful�lled With My Job

60%

Less Frustrated When Coding

59%

Perceived Productivity
Satisfaction and Well-BeingE�ciency and Flow

Faster With Repetitive Tasks

96%

Faster Completion

88%

Less Mental E�orts on Repetitive Tasks

87%

Less Time Searching

77%

More In The Flow

73%

0%

20%

40%
60%
% of Participants That Agreed or Strongly Agreed

80%

100%

Figure 4.3.11

Summary of the Experiment Process and Results
Source: GitHub Survey, 2022 | Table: 2023 AI Index Report

Used
GitHub Copilot

Did Not Use
GitHub Copilot

Number of Developers

45

50

Completion Rate (%)

78

70

Average Time Taken toComplete the Task (Minutes)

71

161

Figure 4.3.12

Table of Contents

Chapter 4 Preview

209Artificial IntelligenceIndex Report 2023

Industry Motivation
This section explores the motivations industry

Chapter 4: The Economy
4.3 Corporate Activity

Importance of AI Solutions for Organizations’Overall Success
Source: Deloitte Survey, 2022 | Chart: 2023 AI Index Reportleaders have in deploying AI and examines the
1%, Not Important
5%, Somewhat Importantdegree to which they feel AI is important, thereasons they are eager to embrace AI, and thefactors that have hindered further scaling ofAI solutions. The data from this section comesfrom Deloitte’s “State of AI in Enterprise” report,
which has surveyed companies about their use
of AI since 2017. This year’s survey polled 2,620

94%, Importantbusiness leaders from a wide range of countries,
industries, and corporate levels.

Figure 4.3.13

Perceived Importance of AI
Figures 4.3.13 and 4.3.14 suggest that an
overwhelming majority of business leaders
perceive AI to be important for their businesses.
More specifically, when asked how important
AI solutions were for their organization’s overall
success, 94% responded “important,” 5% said
“somewhat important,” and 1% answered “notimportant” (Figure 4.3.13).

Believe AI Enhances Performance and Job
Satisfaction, 2022
Source: Deloitte Survey, 2022 | Chart: 2023 AI Index Report

2%, Strongly Disagree / Disagree
1%, Unsure

16%, Neither Agree nor Disagree

Similarly, when asked whether they believe that AI
enhances performance and job satisfaction, 82%
responded “strongly agree/agree,” 16% said they“neither agree nor disagree,” and only 2% selected“strongly disagree/disagree” (Figure 4.3.14).
82%, Strongly Agree / Agree

Table of Contents

Chapter 4 Preview

Figure 4.3.14

210

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

AI Investments and Implementation
Outcomes
In 2022, 76% of surveyed leaders reported
expecting to increase AI investments in the next
fiscal year (Figure 4.3.15). Although this represents

a 9 percentage point decrease since 2021 and a 12
percentage point decrease since 2018, a significantly
large portion of business leaders continue to express
interest in AI investment.

Expected AI Investment Increase in the Next Fiscal Year
Source: Deloitte Survey, 2022 | Chart: 2023 AI Index Report80%% of Respondents

76%

60%

40%

20%

0%

2018

2019

2020

2021

2022
Figure 4.3.15

Table of Contents

Chapter 4 Preview

211

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Figure 4.3.16 highlights the main outcomes that business leaders achieved by embracing AI solutions.7
The top outcome was lowered costs (37%), followed by improved collaboration across business
functions/organizations (34%) and having discovered valuable insights (34%).

Main Outcomes of AI Implementation, 2022
Source: Deloitte Survey, 2022 | Chart: 2023 AI Index Report

37%

Lower Costs
Improve Collaboration AcrossBusiness Functions/Organizations

34%

Discover Valuable Insights

34%

Customize or Improve Product/Programs,
Services, or Offers

33%Enter New Markets/Expand Services to
New Constituents

33%

Make Organizational Processes More E�cient

33%

Improve Decision-Making

32%

Create New Products/
Programs and Services

32%

Predict Demand

32%

Enable New Business/
Service Models

32%

Increase Revenue

31%

Activate the Potential of Existing Headcountand/or Improve Talent Management

30%

Improve Constituent Engagement30%

Anticipate Constituent Needs
0%

28%

10%

20%
% of Respondents

30%
Figure 4.3.16

7 Figure 4.3.16 is drawn from the chart in the Deloitte survey: “Outcomes—‘Achieved to a high degree.’”Table of Contents

Chapter 4 Preview

212Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Challenges in Starting and Scaling AI Projects

were proving business value (37%), lack of executive

The top three challenges that business leaders

commitment (34%), and choosing the right AI

identified in terms of starting AI-related projects

technologies (33%) (Figure 4.3.17).

Top Three Challenges in Starting AI Projects, 2022
Source: Deloitte Survey, 2022 | Chart: 2023 AI Index Report

Proving Business
Value

37%

Lack of Executive
Commitment

34%

Choosing the Right
AI Technologies

0%

33%

5%

10%

15%

20%
% of Respondents

25%

30%

35%
Figure 4.3.17

Table of Contents

Chapter 4 Preview

213

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

The main barrier leaders faced in scaling existing AI initiatives was managing AI-related risks (50%), obtaining
more data or inputs to train a model (44%), and implementing AI technologies (42%) (Figure 4.3.18).

Main Barriers in Scaling AI Initiatives, 2022
Source: Deloitte Survey, 2022 | Chart: 2023 AI Index ReportManaging
AI-Related Risks

50%

Obtaining Needed Data
or Input to Train Model

44%

Implementing
AI Technologies

42%

Proving Business Value

0%

40%

10%

20%

30%
% of Respondents

40%

50%

Figure 4.3.18

Table of Contents

Chapter 4 Preview

214Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Earnings Calls

Aggregate Trends

The following subsection presents data fromfrom Fortune 500 companies that mentioned AI-related

NetBase Quid, which uses natural language

keywords (Figure 4.3.19). The number of such mentions

processing tools to analyze trends in corporate

dropped from the previous year, when there were 306,

earnings calls. NetBase Quid analyzed all 2022

but has increased since 2018 when there were 225.

In the 2022 fiscal year, there were 268 earnings calls

earnings calls from Fortune 500 companies,
identifying all mentions of “Artificial Intelligence,”“AI,” “Machine Learning,” “ML,” and “deep learning.”

Number of Fortune 500 Earnings Calls Mentioning AI, 2018–22
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

300

Number of Earnings Calls

268

200

100

0

2018

2019

2020

2021

2022
Figure 4.3.19

Table of Contents

Chapter 4 Preview

215

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Specific Themes

advertising and marketing (8.8%) (Figure 4.3.20).

Mentions of AI in Fortune 500 earnings calls wereCompared to 2018, some of the less prevalentassociated with a wide range of themes. In 2022, theAI-related themes in 2022 included deep learning

most cited themes were business integration (10.0%);(4.8%), autonomous vehicles (3.1%), and data

pricing and inventory management (8.8%); andstorage and management (3.0%).

Themes for AI Mentions in Fortune 500 Earnings Calls, 2018 Vs. 2022
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

9.96% (-15%)

Business Integration

11.74%
8.82% (+48%)

Pricing and Inventory Management

5.94%
8.82% (+204%)

Advertising and Marketing

2.90%
8.39% (+23%)

Process Automation

6.81%
7.40% (-7%)
7.97%

Support Decision-Making

7.11% (+69%)

Healthcare and Medical Practices

4.20%
6.26% (+73%)

Cloud Platforms

3.62%
5.26% (-21%)

Personalizing Customer Experience

6.67%

4.84% (-41%)

Deep Learning

8.26%
4.13% (+24%)

Edge Intelligence

3.33%
3.84% (+121%)

Nvidia AI Use Cases

1.74%
3.27% (+33%)

Revenue Growth

2.46%
3.13% (-47%)

Autonomous Vehicles

5.94%
2.99% (+37%)

Data Processing

2.17%
2.99% (-55%)

Data Storage and Management

6.67%
2.70% (+10%)
2.46%

Adobe Experience
Customer Support

2.42% (+734%)
0.29%
2.13% (-59%)

Azure Cognitive Services

5.22%
1.85% (-20%)
2.32%

Data Center GPU
Investments
Nvidia RTX
Digital Transformation
0%

1.28% (+47%)
0.87%

2022

1.00% (-62%)

2018
2.61%

0.71% (-87%)
5.36%

1%

2%

3%

4%

5%

6%
7%
8%
9%
Theme Mentioned (% of Total)

10%

11%

12%

13%

14%

Figure 4.3.20

Table of Contents

Chapter 4 Preview

216

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Narrative Highlight:

What Are Business Leaders Actually Saying About AI?
To better understand business attitudes that surround AI, it is worth
looking at AI-related excerpts from the Fortune 500 earnings calls.
For example, on the topic of business integration, companies often
cite AI and machine learning (ML) use cases to reassure businessaudiences of safer business practices, growing opportunities,
streamlining processes, and capability expansion.

“We spent $100 million building
certain risk and fraud systems
so that when we process
payments on the consumer side,
losses are down $100 million to$200 million. Volume is way up.
That’s a huge benefit.”– Jamie Dimon, CEO, JP Morgan
Chase & Co. (Q2 2022)“Especially in the last year or so, thefield of robotics itself has actuallychanged because with AI and
ML coming to the picture, there’s
significant developments in the
robotics field. So we think it’s ahuge opportunity for us.”– Raj Subramaniam, CEO, FedEx(Q3 2022)“We spent a ton of moneyon Cloud. We spend a ton ofmoney on adding capabilities.
And over time, as you do it on
one platform, it all becomes
more efficient. So, I think it’sa lot of little things, but it adds
up with our base of people
and fixed cost, it adds up
significantly over time. We’ve
been able to maintain ourheadcount at a level we feel
good about, and we think
we can grow massively on
top of that without having to
add lots of bodies to be able
to do it.” – Peter Kern, CEO,
Expedia Group (Q4 2022)

In terms of process automation, business leaders emphasize the ability of AI tools to accelerate
productivity gains and to deliver a better customer experience.
“We continue to drive
the use of automation
and artificial
intelligence to drive
productivity gains to
help offset inflationarypressures.” – Jim Davis,
CEO, Quest Diagnostics(Q4 2022)

Table of Contents“We have improved theexperience for customers byapplying artificial intelligence
to match them with an expert
who is right for their specificsituation and to deliver insights
to experts so they can provide
excellent service.” – Sasan
Goodarzi, CEO, Intuit (Q2 2022)

Chapter 4 Preview“In September, we opened a nextgen fulfillment center in Illinois.
This 1.1 million square foot facility
features robotics, machine learning,
and automated storage, resulting inincreased productivity and a better
service for our customers at fasterdelivery times.” – John David, CFO,
Walmart (Q3 2022)

217

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Narrative Highlight:

What Are Business Leaders Actually Saying About AI?
(cont’d)
The conversation surrounding pricing and inventory management saw companies reassuring businessaudiences on how their use of AI would improve their operational strength, especially in environments ofhigh inflation and supply chain challenges.

“We are … continuing to refine and invest
in machine learning tools that will allow for
more sophisticated competitive pricing
and greater automation at scale.”– Adrian Mitchell, CFO, Macy’s (Q3 2022)“Our teams are utilizing technology, innovative data analytics
and AI to forecast supply chain lead times and changes
in market demand to ensure optimal levels. These actions
along with our pricing initiatives positively impacted ourgross margin in the second quarter.”– Bert Nappier, CFO, Genuine Parts Company (Q3 2022)

There is also a vibrant discussion about the ways in which AI can change healthcare and medical
practices, more specifically to reduce costs, improve the patient experience, and better serve clinicians.
“[Using] machinelearning and robotics,
we can now resolvea wide range ofprescription drug
claims which previously
required the attention of
our pharmacists, freeing
them up to spend timewith patients. Thisadvanced approach
reduces overall costand improves thepatient experience.”– Karen Lynch, CEO,CVS Health (Q2 2022)

Table of Contents“I’d like to highlight productivity efforts in our preauthorization process where
we’re leveraging an in-house artificial intelligence solution to automaticallymatch incoming faxes to the correct authorization requests. This solution
creates administrative efficiencies across millions of inbound images. We arealso scaling this solution to multiple business units such as pharmacy andare also expanding the application of this type of AI to provide decision
support to clinicians, which will result in improvements to authorization
turnaround times, reduction in friction for providers and creating a better
member experience.” – Bruce Broussard, CEO, Humana (Q3 2022)“We continue to see opportunities across [the software and analytics] segment
as payers, providers, and partners take advantage of our high ROI solutions andrealize the benefits of our data, AI models, and workflow capabilities.”– Neil de Crescenzo, CEO, UnitedHealth Group (Q2 2022)

Chapter 4 Preview

218

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.3 Corporate Activity

Sentiment Analysis

sentiment associated with mentions of AI has been

NetBase Quid also runs the AI-related text of Fortune

overwhelmingly positive (Figure 4.3.21). Mentions

500 earnings calls through a sentiment analysis

of AI were rarely negative, suggesting that large

machine-learning algorithm that identifies whether

businesses tend to have positive associations when it

the sentiment associated with the mention of AI iscomes to AI tools.

positive, mixed, or negative . Overall, since 2018, the8

Sentiment Summary Distribution for AI Mentions in Fortune 500 Earnings Calls by Publication Date, 2018–22
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report
1%%

2%%

3%%

17%%

19%%

18%%

80%%

79%%

81%%

84%%

Sentiment Summary

1%%

16%%

79%%

86%%

1%%

84%%

12%%

85%%

1%%

20%%

2%%

16%%

1%%

14%%

77%%

0%%

87%%

1%%

13%%

1%%

84%%

80%%

77%%

81%%

1%%

22%%

3%%

17%%

18%%

81%%

84%%

17%%

81%%

15%%

18%%

81%%

3%%

21%%

2%%

76%%

1%%
2%%

23%%

0%%

15%%

1%%

19%%

86%%

80%

1%%

13%%

100%

60%

40%

Negative
20%

Mixed
Positive

0%

Q1

Q2
Q3
2018

Q4

Q1

Q2
Q3
2019

Q4

Q1

Q2
Q3
2020

Q4

Q1

Q2
Q3
2021

Q4

Q1

Q2
Q3
2022

Q4

Figure 4.3.21

8 Chapter 2 of the 2023 AI Index highlights trends in the performance of sentiment analysis algorithms.

Table of Contents

Chapter 4 Preview

219

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

Given that robots are frequently deployed with AI-based software technologies, it is possible to gain insights on AI-ready infrastructure
being deployed in the real world by tracking the installation of industrial robots. Data in this section comes from the International
Federation of Robotics (IFR), an international nonprofit organization that works to promote, strengthen, and protect the robotics
industry. Every year the IFR releases the World Robotics Report, which tracks global trends in installations of robots.9

4.4 Robot Installations
Aggregate Trends

2021 saw a rebound in the total number of worldwide

The following subsection includes data on the

robot installations. The 517,000 industrial robots

installation and operation of industrial robots,

installed in 2021 represented a 31.3% increase fromwhich are defined as an “automatically controlled,

2020 and a 211.5% increase since 2011 (Figure 4.4.1).

reprogrammable, multipurpose manipulator,
programmable in three or more axes, which can beeither fixed in place or mobile for use in industrial
automation applications.”

Number of Industrial Robots Installed in the World, 2011–21
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report

517

Number of Industrial Robots Installed (in Thousands)

500

400

300

200

100

0

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 4.4.1

9 Due to the timing of the IFR’s survey, the most recent data is from 2021.

Table of Contents

Chapter 4 Preview

220Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

The worldwide operational stock of industrial robots

from 3,035,000 in 2020. In the last decade, thealso continues to steadily increase year over year

number of industrial robots being installed and the(Figure 4.4.2). The total number of operational

number being used have both steadily increased.

industrial robots jumped 14.6% to 3,477,000 in 2021,

Operational Stock of Industrial Robots in the World, 2011–21
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report

3,477

3,500

Number of Industrial Robots (in Thousands)

3,000

2,500

2,000

1,500

1,000

500

0

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 4.4.2

Table of Contents

Chapter 4 Preview

221

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

Industrial Robots: Traditional Vs.
Collaborative Robots

scalable than traditional robots, and are capable ofiterative learning.

A distinction can be drawn between traditional
robots that work for humans and collaborative

In 2017, only 2.8% of all newly installed industrial

robots that are designed to work with humans.

robots were collaborative (Figure 4.4.3). As of 2021,

Recently, the robotics community has been excitedthat number increased to 7.5%. Although traditional

about the potential of collaborative robots given

industrial robots still lead new installations, thethat they can be safer, more flexible, and morenumber of collaborative robots is slowly increasing.

Number of Industrial Robots Installed in the World by Type, 2017–21
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report

Number of Industrial Robots Installed (in Thousands)

500

Traditional

517

Collaborative

39

424
400

400

391

394

300

478
200

389

405

2017

2018

370

368

2019

2020

100

0

2021
Figure 4.4.3

Table of Contents

Chapter 4 Preview

222

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

By Geographic Area

the amount installed by Japan (47,200) and 7.7

Country-level data on robot installations can illustratetimes the amount installed by the United States

which countries are prioritizing the integration of(35,000) (Figure 4.4.4). The countries with the next

robots into their economy. In 2021, China installed

most installations were South Korea (31,100) andthe most industrial robots, with 268,200, 5.7 times

Germany (23,800).

Number of Industrial Robots Installed by Country, 2021
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report

China

268.20

Japan

47.20

United States

35.00

South Korea

31.10

Germany

23.80

Italy

14.10

Taiwan9.60

France5.90

Mexico

5.40

India

4.90

Canada4.30

Thailand

3.90

Singapore

3.50

Spain

3.40

Poland

3.30

0

30

60

90
120
150
180
Number of Industrial Robots Installed (in Thousands)

210

240

270
Figure 4.4.4

Table of Contents

Chapter 4 Preview

223

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

In 2013, China overtook Japan as the nation installingthe most industrial robots (Figure 4.4.5). Since then,the gap between the total number of industrial robots
installed by China and the next-nearest nation has

only widened. In 2013, Chinese industrial robot
installations represented 20.8% of the world’s share,
whereas in 2021, they represented 51.8%.

Number of New Industrial Robots Installed in Top Five Countries, 2011–21
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report

268, China

Number of Industrial Robots Installed (in Thousands)

250

200

150

100

47, Japan
35, United States
31, South Korea
24, Germany

50

0

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 4.4.5

Table of Contents

Chapter 4 Preview

224

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

China consolidated its dominance in industrial robotics in 2021, the first year in which the country installed
more industrial robots than the rest of the world combined (Figure 4.4.6).

Number of Industrial Robots Installed (China Vs. Rest of the World), 2016–21
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report

Number of Industrial Robots Installed (in Thousands)

268, China
250

249, Rest of the World

200

150

100

50

0

2016

2017

2018

2019

2020

2021
Figure 4.4.6

Table of Contents

Chapter 4 Preview

225

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

Figure 4.4.7 shows the annual growth rate ofof industrial robot installations. The countries thatindustrial robot installations from 2020 to 2021 byreported the highest growth rates were Canada

country. Virtually every country surveyed by the(66%), Italy (65%), and Mexico (61%).

IFR reported a yearly increase in the total number

Annual Growth Rate of Industrial Robots Installed by Country, 2020 Vs. 2021
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report

Canada

66%

Italy

65%

Mexico61%

Poland

56%

India

54%

China

51%

Thailand36%

Taiwan

31%

Japan

22%

United States

14%

France11%

Germany

6%South Korea

2%

Spain
Singapore

1%
-35%

−40%

−30%

−20%

−10%

0%
10%
20%
30%
Annual Growth Rate of Industrial Robots Installed

40%

50%

60%

70%
Figure 4.4.7

Table of Contents

Chapter 4 Preview

226

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

Narrative Highlight:

Country-Level Data on Service Robotics
Another important class of robots are service robots,
which the ISO defines as a robot “that performs

Service Robots in Medicine
Source: UL Solutions, 2022

useful tasks for humans or equipment excluding
industrial automation applications.”10 Figure 4.4.8
is an example of a robot being used in medicine,
Figure 4.4.9 illustrates how a robot can help withprofessional cleaning, and Figure 4.4.10 shows arobot designed for maintenance and inspection.

Figure 4.4.8

Service Robots in Professional Cleaning
Source: This Week in FM, 2021

Figure 4.4.9

Service Robots in Maintenance and Inspection
Source: Robotnik, 2022

Figure 4.4.10

10 A more detailed definition can be accessed here.

Table of Contents

Chapter 4 Preview

227

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

Narrative Highlight:

Country-Level Data on Service Robotics (cont’d)Compared to 2020, 2021 saw a higher number of professional service robots installed in the world
for several key application areas, including hospitality, medical robotics, professional cleaning, andtransportation and logistics (Figure 4.4.11). The category that registered the greatest year-over-yearincrease was transportation and logistics: In 2021, 1.5 times the number of such service robots were
installed as in 2020.

Number of Professional Service Robots Installed in the World by Application Area, 2020 Vs. 2021
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report

8

Agriculture

8

20

Hospitality

11

15

Medical Robotics

12

13

Professional Cleaning

2020
50

Transportation and Logistics

0

2021

10

34
5

10

15
20
25
30
35
40
Number of Professional Service Robots Installed (in Thousands)

45

50

Figure 4.4.11

Table of Contents

Chapter 4 Preview

228

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

Narrative Highlight:

Country-Level Data on Service Robotics (cont’d)
As of 2022, the United States has the greatest number of professional service robot manufacturers,
roughly 2.16 times as many as the next nation, China. Other nations with significant numbers of robot
manufacturers include Germany (91), Japan (66), and France (54) (Figure 4.4.12).

Number of Professional Service Robot Manufacturers in Top Countries by Type of Company, 2022
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report

225

Number of Professional Service Robot Manufacturers

29
200

150

100

104
194

Startups

91

Incumbents
Unknown
66
54

50

94

52

47

79
61

China

FranceRussia44

Germany

Japan

39

34

35

1

1

United States

44
3

2

0

49

39

South Korea

Switzerland

CanadaFigure 4.4.12

Table of Contents

Chapter 4 Preview

229

Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

Sectors and Application Types(137,000), followed by automotive (119,000) (Figure

On a global level, the sector that saw the greatest

increases in the total number of industrial robot

amount of robot installations was electrical/electronics

installations since 2019.

4.4.13). Each of the highlighted sectors has recorded

Number of Industrial Robots Installed in the World by Sector, 2019–21
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report52

All Others

37
30

119

Automotive

84
102
137

Electrical/Electronics

110
89
15

Food

12
11
64

Metal and Machinery

44
52
24

Plastic and Chemical Products

19
18

2021
2020

107

Unspeci ed

87
87

0

20

40

60
80
100
Number of Industrial Robots Installed (in Thousands)

2019
120

140
Figure 4.4.13

Table of Contents

Chapter 4 Preview

230Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

Robots can also be deployed in a wide range of

230,000 industrial robots were installed for handling

applications, from assembling to dispensing andfunctions, 2.4 times more than for welding (96,000)

handling. Figure 4.4.14 illustrates how the applicationand 3.7 times more than for assembling (62,000).

of industrial robots has changed since 2021. HandlingEvery application category, with the exception

continues to be the application case toward whichof dispensing and processing, saw more robot

the most industrial robots are deployed. In 2021,

installations in 2021 than in 2019.

Number of Industrial Robots Installed in the World by Application, 2019–21
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report

62

Assembling

50
40

32
32

Cleanroom
26
11

Dispensing

8
12
230

Handling

169
177
7

Processing

5

2021

7

2020

80

Unspeci ed

2019

60

55
96

Welding

70
74

0

20

40

60

80
100
120
140
160
Number of Industrial Robots Installed (in Thousands)

180

200

220

240
Figure 4.4.14

Table of Contents

Chapter 4 Preview

231Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

China Vs. United Statesand metal and machinery (34,000) (Figure 4.4.15).

The Chinese industrial sectors that installed theEvery industrial sector in China recorded a greater

greatest number of industrial robots in 2022 were

number of robot installations in 2021 than in 2019.

electrical/electronics (88,000), automotive (62,000),

Number of Industrial Robots Installed in China by Sector, 2019–21
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report29

All Others

21
12
62

Automotive

31
32
88

Electrical/Electronics

64
42
4

Food

3
3
34

Metal and Machinery

Pharma/Cosmetics

22
22
1
1
1
6

Rubber and Plastics

5

2021

4

2020

43

Unspeci ed
0

2019

30
31

10

20

30
40
50
60
Number of Industrial Robots Installed (in Thousands)

70

80

90
Figure 4.4.15

Table of Contents

Chapter 4 Preview

232Artificial IntelligenceIndex Report 2023

Chapter 4: The Economy
4.4 Robot Installations

The automotive industry installed the greatest number of industrial robots in the United States in 2021,
although installation rates for that sector decreased year over year (Figure 4.4.16). However, other sectors likefood, along with plastic and chemical products, saw year-over-year increases in robot installations.

Number of Industrial Robots Installed in the United States by Sector, 2019–21
Source: International Federation of Robotics (IFR), 2022 | Chart: 2023 AI Index Report

4.50

All Others

2.60
3.50
9.80

Automotive

10.50
13.00
2.90

Electrical/Electronics

3.70
3.50
3.40

Food

2.70

2.20
3.80

Metal and Machinery

2.30
3.80
3.50

Plastic and Chemical Products

2.60
2.50

2021
2020

7.10

Unspeci ed

6.30

2019

5.00

0

2

3

5
6
8
9
Number of Industrial Robots Installed (in Thousands)

11

12

14

Figure 4.4.16

Table of Contents

Chapter 4 Preview

233Artificial IntelligenceIndex Report 2023

Artificial IntelligenceIndex Report 2023

CHAPTER 5:

Education

Table of Contents

Chapter 5 Preview

234

Artificial IntelligenceIndex Report 2023

CHAPTER 5 PREVIEW:

Education
Overview

236

5.2 K–12 AI Education

257

Chapter Highlights

237

United States

257

5.1 Postsecondary AI Education

238

CS Bachelor’s Graduates

238

CS Master’s Graduates

240

CS PhD Graduates

242

CS, CE, and Information Faculty

246

Narrative Highlight: Who Funds
CS Departments in the U.S.?

Table of Contents

Chapter 5 Preview

State-Level Trends

257

AP Computer Science

258

Narrative Highlight: The State ofInternational K–12 Education

260

ACCESS THE PUBLIC DATA
255

235

Artificial IntelligenceIndex Report 2023

Chapter 5: Education

Overview
Studying the state of AI education is important for gauging some of the ways in which
the AI workforce might evolve over time. AI-related education has typically occurredat the postsecondary level; however, as AI technologies have become increasingly
ubiquitous, this education is being embraced at the K–12 level. This chapter examines
trends in AI education at the postsecondary and K–12 levels, in both the United States
and the rest of the world.
We analyze data from the Computing Research Association’s annual Taulbee Survey
on the state of computer science and AI postsecondary education in North America,
Code.org’s repository of data on K–12 computer science in the United States, and a
recent UNESCO report on the international development of K–12 education curricula.

Table of Contents

Chapter 5 Preview

236Artificial IntelligenceIndex Report 2023

Chapter 5: Education

Chapter Highlights
More and more AI specialization.
The proportion of new computer science PhD graduates from U.S. universities who specialized in AI
jumped to 19.1% in 2021, from 14.9% in 2020 and 10.2% in 2010.

New AI PhDs increasingly
head to industry.
In 2011, roughly the same proportion of
new AI PhD graduates took jobs in industry(40.9%) as opposed to academia (41.6%).
Since then, however, a majority of AI PhDs
have headed to industry. In 2021, 65.4% of AI
PhDs took jobs in industry, more than double
the 28.2% who took jobs in academia.

The gap in external
research funding forprivate versus public
American CS departments
continues to widen.
In 2011, the median amount of total expenditurefrom external sources for computing research
was roughly the same for private and public
CS departments in the United States. Since
then, the gap has widened, with private U.S.
CS departments receiving millions more in
additional funding than public universities.
In 2021, the median expenditure for private
universities was $9.7 million, compared to$5.7 million for public universities.

Table of Contents

Chapter 5 Preview

New North American
CS, CE, and information
faculty hires stayed flat.
In the last decade, the total number of new
North American computer science (CS),computer engineering (CE), and information
faculty hires has decreased: There were
710 total hires in 2021 compared to 733 in
2012. Similarly, the total number of tenuretrack hires peaked in 2019 at 422 and thendropped to 324 in 2021.

Interest in K–12 AI andcomputer science education
grows in both the United States
and the rest of the world.
In 2021, a total of 181,040 AP computerscience exams were taken by American
students, a 1.0% increase from the previous
year. Since 2007, the number of AP computerscience exams has increased ninefold. As of2021, 11 countries, including Belgium, China,
and South Korea, have officially endorsed
and implemented a K–12 AI curriculum.

237

Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

5.1 Postsecondary AI Education
CS Bachelor’s Graduates
At the undergraduate level, most AI-related courses

interest in AI. In 2021, the total number of new Northare offered as part of a computer science (CS)American CS bachelor’s graduates was 33,059—

curriculum. Therefore, trends in new CS bachelor’s

nearly four times greater than in 2012 (Figure 5.1.1).

graduates give us a proxy for undergraduate

New CS Bachelor’s Graduates in North America, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

33,059Number of New CS Bachelor’s Graduates

30,00025,00020,00015,00010,000

5,000

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.1

Table of Contents

Chapter 5 Preview

238

Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

Figure 5.1.2 looks at the proportion of CS bachelor’s graduates in North America who are international
students. The number stood at 16.3% in 2021 and has been steadily increasing since 2012—the proportionof such students has risen 9.5 percentage points since 2012.

New International CS Bachelor’s Graduates (% of Total) in North America, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

16.30%New International CS Bachelor’s Graduates (% of Total)

16%

12%

8%

4%

0%

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.2

Table of Contents

Chapter 5 Preview

239

Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

CS Master’s Graduates
AI courses are also commonly offered in CS master’s

as many master’s graduates as in 2012. However,degree programs. Figure 5.1.3 shows the total

from 2018 to 2021 the total number of new master’s

number of new CS master’s graduates in North

graduates plateaued, declining slightly from 15,532 toAmerica since 2010. In 2021 there were roughly twice

15,068.

New CS Master’s Graduates in North America, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

16,00015,068

Number of New CS Master’s Graduates

14,00012,00010,000

8,000

6,000

4,000

2,000

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.3

Table of Contents

Chapter 5 Preview

240Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

Interestingly, the number of CS master’s students at North American universities who are international started
declining in 2016 after rising in the early 2010s (Figure 5.1.4). Despite the decline, in 2021 the majority of CS
master’s graduates remained international (65.2%).

New International CS Master’s Graduates (% of Total) in North America, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

New International CS Master’s Graduates (% of Total)

80%65.20%
60%

40%

20%

0%

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.4

Table of Contents

Chapter 5 Preview

241Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

CS PhD Graduates
Unlike the trends in bachelor’s and master’s CS

computer science (Figure 5.1.5). There were fewer

graduates, since 2010 there have not been large

CS PhD graduates in 2021 (1,893) than in 2020 (1,997)

increases in the number of new PhD graduates inand 2012 (1,929).

New CS PhD Graduates in North America, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

Number of New CS PhD Graduates

2,000

1,893

1,500

1,000

500

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.5

Table of Contents

Chapter 5 Preview

242Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

CS PhD graduates in North American universities are becoming increasingly international (Figure 5.1.6). In 2010,
45.8% of CS PhD graduates were international students; the proportion rose to 68.6% in 2021.

New International CS PhD Graduates (% of Total) in North America, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

New International CS PhD Graduates (% of Total)

70%

68.60%

60%

50%

40%

30%

20%

10%

0%

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.6

Table of Contents

Chapter 5 Preview

243Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

Moreover, now a significantly larger proportion of new CS PhD students are specializing in AI (Figure 5.1.7). In
2021, 19.1% of new CS PhD students in North American institutions specialized in AI, a 4.2 percentage point
increase since 2020 and 8.6 percentage point increase since 2012.

New CS PhD Students (% of Total) Specializing in AI, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

19.10%
18%

New AI PhD Students (% of Total)

16%
14%
12%
10%
8%
6%
4%
2%
0%

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.7

Table of Contents

Chapter 5 Preview

244Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

Where do new AI PhDs choose to work following(40.9%) as in academia (41.6%). However, as of 2021

graduation? Mirroring trends reported in last year’s

a significantly larger proportion of students (65.4%)

AI Index report, an increasingly large proportion ofwent to industry after graduation than to academia

AI PhD graduates are heading to industry (Figures(28.2%). The amount of new AI PhDs entering

5.1.8 and 5.1.9). In 2011, for example, roughly thegovernment was 0.7% and has remained relatively

same percentage of graduates took jobs in industry

unchanged in the last half-decade.

Employment of New AI PhDs in North America by
Sector, 2010–21

Employment of New AI PhDs (% of Total) in North
America by Sector, 2010–21

Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index ReportIndustry

65.44%, Industry
238

61

63

154
132

72
51

63

60

123

43
42
162
134

101
74

64

153

116

40%
30%

28.19%, Academia

20%
10%

77

0.67%, Government
2019

2020

2017

2018

2016

2015

2014

2012

2013

2011

2010

2018

2017

2016

2015

0%
2013

2011

0

2010

76

85

2014

50

195

180

2021

100

136

50%

2021

47

2020

134

2019

150

219

73

178
154

60%
84

65

201

200

2012

Number of New AI PhD Graduates

Academia

249
New AI PhD Graduates (% of Total)

Government
250

Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report281

Figure 5.1.8

Figure 5.1.9

1 The sums in Figure 5.1.9 do not add up to 100, as there is a subset of new AI PhDs each year who become self-employed, unemployed, or report an “other” employment status
in the CRA survey. These students are not included in the chart.

Table of Contents

Chapter 5 Preview

245Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

CS, CE, andInformation Faculty

5.1.10 highlights the total number of CS, CE (computer

To better understand trends in AI and CS education,

marginally increased in the last year, by 2.2%. Since

it is instructive to consider data on computer science

2011 the number of CS, CE, and information faculty

faculty in addition to postsecondary students. Figure

has grown by 32.8%.

engineering), and information faculty in North
American universities. The amount of faculty has

Number of CS, CE, and Information Faculty in North America, 2011–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

Tenure Track

Teaching Professors

Other Instructors

Research

8,000
Number of CS, CE, and Information Faculty

7,362
7,000
6,000
5,000

6,138

6,314

6,478

602

766

515

676

661

487

4,366

4,536

4,549

2011

2012

2013

656
447
669

6,629
689
529

6,806

6,887

691

649

589

432

432

390

Postdoc

7,657
653
465
494

7,858

7,976

668

530
296

426
617

8,149
522
306

736

861

1,180

831

895

1,183

1,150

1,014

1,122

4,711

4,786

5,059

5,214

5,252

5,231

5,310

4,548

2014

2015

2016

2017

2018

2019

2020

2021

863

4,000
3,000
2,000
1,000
0

Figure 5.1.10

Table of Contents

Chapter 5 Preview

246

Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

In 2021 there were a total of 6,789 CS faculty members in the United States (Figure 5.1.11). The total numberof CS faculty in the United States increased by only 2.0% in the last year, but by 39.0% since 2011.

Number of CS Faculty in the United States, 2011–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

Tenure Track

Teaching Professors

Other Instructors

Research

Postdoc

7,000
6,098

6,000
5,256
Number of CS Faculty

5,000

4,885
522

4,000

387

521
460
550

521

5,068
592
491
421

5,637

5,729

5,202

535

491

509

396

455

826

364
903

567

6,430

6,533

531

518

426

382

408

436

534

947

671

715

6,654
424
276

6,789
428
287

618

693

946

899

679

3,000

2,000
3,455

3,725

3,564

3,559

2012

2013

2014

3,971

4,176

4,366

4,384

4,390

4,482

3,880

2015

2016

2017

2018

2019

2020

2021

1,000

0

2011

Figure 5.1.11

Table of Contents

Chapter 5 Preview

247Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

Figure 5.1.12 reports the total number of new CS,

hires in 2021, while in 2012 there were 733. Similarly,

CE, and information faculty hires in North American

the total number of tenure-track hires peaked in 2019

universities. In the last decade, the total number ofat 422 and has since dropped to 324 in 2021.

new faculty hires has decreased: There were 710 total

New CS, CE, and Information Faculty Hires in North America, 2011–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

Number of New CS, CE, and Information Faculty Hires

Total

878

Tenure-Track

860

800

800

749

733

765

749

710

691

600

583

572

543

396

400

348
294
249

406

422
374

358

324

320

258
218

200

0

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.12

Table of Contents

Chapter 5 Preview

248Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

In 2021, the greatest percentage of new CS, CE, and information faculty hires (40%) came straight fromreceiving a PhD (Figure 5.1.13). Only 11% of new CS and CE faculty came from industry.

Source of New Faculty in North American CS, CE, and Information Departments, 2011–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

New PhD

From Postdoc

From Other Academic

From Industry

100%
13%

13%

11%

11%

34%

34%

16%

15%

39%

40%

2020

2021

Source of New Faculty

80%
34%

41%

60%

15%
40%

20%

17%

38%
29%

0%

2018

2019

Figure 5.1.13

Table of Contents

Chapter 5 Preview

249

Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

The share of filled new CS, CE, and information faculty positions in North American universities has remained
relatively stable in the last decade (Figure 5.1.14). In 2021, 89.3% of new faculty positions were filled, comparedto 82.7% in 2011.

Share of Filled New CS, CE, and Information Faculty Positions in North America, 2011–21

Share of Filled New CS, CE, and Information Faculty Positions

Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

90%89.28%

80%
70%
60%
50%
40%
30%
20%
10%
0%

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.14

Table of Contents

Chapter 5 Preview

250Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

Among open CS, CE, and information faculty positions in 2021, the most commonly cited reason for theirremaining unfilled was offers being turned down (53%) (Figure 5.1.15). In 22% of cases, hiring was still inprogress, while 14% of the time, a candidate had not been identified who met the department’s hiring goals.

Reason Why New CS, CE, and Information Faculty Positions Remained Un lled (% of Total), 2011–21

Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

Didn’t �nd a person who met our hiring goals

Technically vacant, not �lled for admin reasonsO�ers turned down

Hiring in progress

Other

Reason Faculty Positions Remained Un�lled (% of Total)

100%
10%

18%

10%
80%

25%

26%

17%

31%

28%

27%

23%

22%
18%

55%
60%

10%

5%

6%

6%

45%

12%

34%
36%

40%

40%
43%

52%

51%

56%

53%
44%

20%

37%

37%

37%
26%

0%

2011

2012

2013

2014

26%

2015

16%

14%

14%

13%

2016

2017

2018

2019

8%
2020

14%
2021
Figure 5.1.15

Table of Contents

Chapter 5 Preview

251Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

Figure 5.1.16 highlights the median nine-month salaries

full professor in computer science made 3.2% moreof CS faculty in the United States by position sincethan they did in 2020, and 12.8% more than they did in

2015. During that period, the salaries for all classes

2015. (Note: These figures have not been adjusted forof professors have increased. In 2021, the average

inflation.)

Median Nine-Month Salary of CS Faculty in United States, 2015–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

Full Professor

Median Salary of CS Faculty (in Thousands of U.S. Dollars)

180
160

Associate Professor

176.01

170.57168.87164.54159.96158.97156.02

Assistant Professor

140
120

113.95111.67103.01101.1699.12

100

114.07109.23107.55105.45127.47

123.71121.55

119.48

117.5

80
60
40
20
0

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.16

Table of Contents

Chapter 5 Preview

252

Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

What proportion of new CS, CE, and information faculty tenure-track hires are international? The data suggeststhat it is not a substantial proportion. In 2021, only 13.2% of new CS, CE, and information faculty hires wereinternational (Figure 5.1.17).

New International CS, CE, and Information Tenure-Track Faculty Hires (% of Total) in North America,
2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

New International Tenure-Track Faculty Hires (% of Total)

25%

20%

15%
13.20%

10%

5%

0%

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.17

Table of Contents

Chapter 5 Preview

253Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

The majority of CS, CE, and Information faculty losses in North American departments (36.3%) were the resultof faculty taking academic positions elsewhere (Figure 5.1.18). In 2021, 15.2% of faculty took nonacademic
positions, which is roughly the same amount as those who took such positions a decade prior, in 2011 (15.9%).

Faculty Losses in North American CS, CE, and Information Departments, 2011–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

Died

Took academic position elsewhere

Remained, but changed to part-time

RetiredTook nonacademic position

Other

Unknown

327
300
270
250

Faculty Losses

213
200

221

50

0

234
42

36

44

32

24

20

23

43

77
74

86

74

65

2013

2014

312

303

37
46

34
33

20

26
126

34
62

100

237

22

20

22
27

150

232

246

303

139
113

89

110

85

52

67

2011

89

2012

94

90

2015

2016

80

2017

94

103

91

2018

2019

2020

100

2021
Figure 5.1.18

Table of Contents

Chapter 5 Preview

254

Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

Narrative Highlight:

Who Funds CS Departments in the U.S.?
The CRA tracks data on the external funding sources

agencies such as the Army Research Office,of CS departments in the United States. The main

the Office of Naval Research, and the Air Forcefunder of American CS departments continues to

Research Laboratory (20.3%); industrial sources

be the National Science Foundation (NSF), which(12.1%); the Defense Advanced Research Projects

in 2021 accounted for 34.9% of external funds.

Agency (DARPA) (8.8%); and the National

However, the share of funding provided by NSF has

Institutes of Health (NIH) (6.8%). The diminishingdecreased since 2003 (Figure 5.1.19). In 2021, theshare of NSF funds over time has been partiallynext largest sources of funding came from defense

offset by increasing funds from industry and NIH.

External Funding Sources (% of Total) of CS Departments in United States, 2003–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

45%
40%

External Funding Sources (% of Total)

35%

34.90%, NSF

30%
25%
20%

20.30%, Other Defense

15%

12.10%, Industrial Sources
8.80%, DARPA
6.80%, NIH
4.90%, Private Foundation
4.60%, Other
3.60%, Other Federal
2.30%, DOE
1.50%, State Agencies
0.40%, Unallocated
0.00%, IMLS

10%
5%
0%
2003

2006

2009

2012

2015

2018

2021

Figure 5.1.19

Table of Contents

Chapter 5 Preview

255Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.1 Postsecondary AI Education

Narrative Highlight:

Who Funds CS Departments in the U.S.? (cont’d)Figure 5.1.20 shows the median total expenditures

Although total median expenditures havefrom external sources for computing research inincreased over the last decade for both private andAmerican CS departments. In 2021, the median total

public CS departments, the gap in expenditure

expenditure for private universities was $9.7 million

has widened, with private universities beginning tocompared with $5.7 million for public universities.

significantly outspend public ones.

Median Total Expenditure From External Sources for Computing Research of U.S. CS Departments, 2011–21

Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

Median Total Expenditure (in Millions of U.S. Dollars)

10

9.71, Private

865.69, Public

4

2

0

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

Figure 5.1.20

Table of Contents

Chapter 5 Preview

256Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.2 K–12 AI Education

The following subsection shows trends in K–12 AI education based on K–12 computer science education data in the United States as wellas survey data from UNESCO on the state of global K–12 AI education.

5.2 K–12 AI Education
United States

States Requiring That All High Schools Offer a Computer Science
Course, 2022

Data on the state of K–12 CS
education in the United States

Source: Code.org, 2022 | Chart: 2023 AI Index ReportMEAK

comes from Code.org, an
education innovation nonprofit
dedicated to ensuring thatevery school includes computerscience as part of its core K–12
education. Tracking trends

CA

NH

MA

NY

CT

RI

WA

MT

ND

SD

MN

WI

MI

OR

ID

WY

NE

IA

IL

IN

OH

PA

NJ

NV

UT

CO

KS

MO

KY

WV

DC

MD

DE

AZ

NM

OK

AR

TN

VA

NC

TX

LA

MS

AL

GA

SC

in K–12 CS education canpartially serve as a proxy forunderstanding the state of K–12
AI education in America
HI

State-Level Trends

VT

Yes
No

FL

Figure 5.2.1 highlights the 27

Figure 5.2.1

states that in 2022 required that
all high schools offer a computerscience course.
Figure 5.2.2 highlights thepercentage of public highPublic High Schools Teaching Computer Science (% of Total in State),
2022
Source: Code.org, 2022 | Chart: 2023 AI Index Report
ME
60%

AK
51%

schools in a state that teachcomputer science. The top
three states in terms of rate ofcomputer science teaching are
Maryland (98%), South Carolina(93%), and Arkansas (92%).

CA
40%

HI
77%

VT
76%

NH
82%

MA
78%

NY
48%

CT
77%

RI
86%

WA
47%

MT
36%

ND
44%

SD
39%

MN
21%

WI
66%

MI
46%

OR
63%

ID
38%

WY
55%

NE
52%

IA
71%

IL
44%

IN
85%

OH
48%

PA
77%

NJ
67%

NV
83%

UT
73%

CO
57%

KS
40%

MO
49%

KY
63%

WV
78%

DC
45%

MD
98%

DE
40%

AZ
36%

NM
41%

OK
62%

AR
92%

TN
60%

VA
75%

NC
61%

TX
47%

LA
32%

MS
60%

AL
85%

GA
66%

SC
93%

FL
40%
Figure 5.2.2

Table of Contents

Chapter 5 Preview

257Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.2 K–12 AI Education

AP Computer Sciencea total of 181,040 AP computer science exams taken,

Another barometer for tracking the state of K–12 CS

roughly the same number as the previous year, after

education in the United States is analyzing trends in theseveral years of significant increases. This leveling

total number of AP computer science exams taken.

could be the result of the pandemic. Since 2007, the2number of AP computer science exams has increased

Year over year the total number of AP computer

over ninefold.

science exams continued to increase. In 2021, themost recent year for which there is data, there were

Number of AP Computer Science Exams Taken, 2007–21
Source: Code.org, 2022 | Chart: 2023 AI Index Report

181.04

Number of AP Computer Science Exams Taken (in Thousands)

180
160
140
120
100
80
60
40
20
0

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.2.3

2 There are two types of AP CS exams: Computer Science A and Computer Science Principles. Data on computer science exams taken includes both exams. AP CS Principles
was initially offered in 2017.

Table of Contents

Chapter 5 Preview

258Artificial IntelligenceIndex Report 2023

In 2021, the states which
saw the greatest number of
AP computer science exams
taken were California (31,189),

Chapter 5: Education
5.2 K–12 AI Education

Number of AP Computer Science Exams Taken, 2021
Source: Code.org, 2022 | Chart: 2023 AI Index Report

ME
242

AK
100
VT
150

followed by Texas (17,307),
Florida (14,864), New York

SD
26

WI
MN
MI
1,432 2,080 4,504

OR
714

ID
429

WY
112

NE
514

IA
521

OH
PA
NJ
IL
IN
8,572 2,883 3,754 6,104 9,391

NVCA
31,189 1,701

UT
612

CO
2,584

KS
236

MO
1,199

KY
1,462

AZ
1,587

NM
270

OK
500

VA
TN
NC
AR
1,406 2,046 6,034 6,273

with the largest per capita

TXLA
17,307 1,191

amount of AP computer
2021 was Maryland, with

RI
617

ND
109

taken per capita.3 The statescience exams taken inNYCT
13,304 3,251

MT
42(9,391) (Figure 5.2.4).

number of AP CS exams

MA
5,451

WA
4,034

(13,304), and New Jersey

Figure 5.2.5 looks at the

NH
403

MS
400

WV
352

GA
AL
2,399 7,221

HI
782

DC
352

MD
7,662

DE
513

SC
2,159

FL
14,864
Figure 5.2.4

124.1 exams per 100,000
inhabitants. The next states
were New Jersey (101.3),
Connecticut (89.7), California(79.7), and Massachusetts(78.0).

Number of AP Computer Science Exams Taken per 100,000 Inhabitants,
2021
Source: Code.org, 2022 | Chart: 2023 AI Index Report
ME
17.57

AK
13.62

VT
NH
MA
23.18 29.04 77.99
WA
52.11

MT
3.80

ND
14.01

SD
2.90

WI
MN
MI
25.07 35.37 44.87

WY
OR
NE
ID
IA
IL
IN
16.78 22.53 19.33 26.18 16.29 67.57 42.31
UT
NV
CA
CO
79.68 54.06 18.33 44.47
AZ
21.84

NM
12.76

KS
8.03

MO
KY
19.43 32.44

OH
31.91

PA
NJ
46.91 101.33

MD
DC
DE
52.63 124.09 51.05

VA
TN
OK
NC
AR
12.53 46.43 29.36 69.70 59.37

TX
MS
LA
58.55 25.74 13.56
HI
54.04

WV
19.71

RI
NY
CT
67.00 89.72 56.25

SC
GA
AL
47.51 66.94 41.57
FL
68.10

Figure 5.2.5

3 More specifically, Figure 5.2.5 normalizes the number of AP CS exams taken—the total number of exams taken in a particular state in 2021 is divided by the state’s
population based on the 2021 U.S. Census.

Table of Contents

Chapter 5 Preview

259

Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.2 K–12 AI Education

Narrative Highlight:

The State of International K–12 Education
In 2021, UNESCO released one of the mostFigure 5.2.6, taken from the UNESCO report,

comprehensive reports to date on the international

highlights the governments that have taken steps

state of government-endorsed AI curricula. To

to implement AI curricula and across which levels

gather information, UNESCO released two surveys:

of education. For example, Germany is in thethe first to representatives of 193 UNESCO member

process of developing government-endorsed AI

states and the second to over 10,000 private-

curricular standards on the primary, middle, andand third-sector actors. As part of these surveys,

high-school levels, and the Chinese government

respondents were asked to report on the status of AI

has already endorsed and implemented

curricula for students in K–12 general education.

standards across those same three levels.

Government Implementation
ImplementationofofAI
AICurricula
Curriculaby
byCountry,
Country,Status,
Status,andandEducation
EducationLevel
Level
UNESCO, 2022
2022 || Table:
Table:2023
2023AIAIIndexIndexReport
ReportSource: UNESCO,Country

StatusStatusArmenia

Endorsedand
EndorsedandImplemented
Implemented

PrimarySchool
SchoolPrimary

MiddleSchoolMiddleSchoolHigh
SchoolHigh
School✓✓Austria

EndorsedEndorsedandandImplemented
Implemented✓✓

Belgium

EndorsedEndorsedandandImplemented
Implemented

China

EndorsedEndorsedandandImplemented
Implemented

India

EndorsedEndorsedandandImplemented
Implemented

KuwaitEndorsedEndorsedandandImplemented
Implemented✓✓✓✓

PortugalPortugal

EndorsedEndorsedandandImplemented
Implemented

✓✓

✓✓

✓✓

Qatar
Qatar

EndorsedEndorsedandandImplemented
Implemented

✓✓

✓✓

✓✓

SerbiaSerbiaEndorsedEndorsedandandImplemented
Implemented

✓✓

✓✓

South
South Korea
Korea

EndorsedEndorsedandandImplemented
Implemented

UnitedUnited Arab
Arab Emirates
Emirates
Bulgaria
Bulgaria

EndorsedEndorsedandandImplemented
Implemented
In
Development
In Development

✓✓
✓✓

✓✓
✓✓

✓✓

Germany
Germany
JordanJordanIn
InDevelopment
Development
In
InDevelopment
Development

✓✓

✓✓
✓✓

✓✓

SaudiaSaudia Arabia
ArabiaSerbiaSerbiaIn
InDevelopment
Development
In
InDevelopment
Development

✓✓

✓✓
✓✓

✓✓

✓✓
✓✓
✓✓

✓✓

✓✓

✓✓

✓✓

✓✓
✓✓
✓✓
✓✓

Figure 5.2.64

4 According to the UNESCO report, Serbia has already endorsed and implemented certain kinds of K–12 AI curricula, but is also simultaneously in the process ofdeveloping others—thus it is listed under both categories.

Table of Contents

Chapter 5 Preview

260

Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.2 K–12 AI Education

Narrative Highlight:

The State of International K–12 Education (cont’d)Figure 5.2.7 identifies the topic areas most emphasized in the K–12 AI curricula profiled in the UNESCO
report. The four topics toward which the most time was allocated were algorithms and programming (18%),
AI technologies (14%), data literacy (12%), and application of AI to other domains (12%).

Time Allocated (% of Total) in K–12 AI Curricula by Topic, 2022
Source: UNESCO, 2022 | Chart: 2023 AI Index Report

Algorithms and Programming

18%

Data Literacy

12%

Contextual Problem-Solving

11%

Application of AI to Other Domains

12%

Ethics of AI

7%

Social Implications of AI

5%

AI Technologies

14%

Developing AI Technologies

9%
AI Foundations

AI Techniques

Ethics and Social Impact

2%

Understanding, Using, and Developing AI
Unspeci ed
0%Unspeci ed

10%
2%

4%

6%

8%
10%
% of Time Allocated

12%

14%

16%

18%

Figure 5.2.7

Table of Contents

Chapter 5 Preview

261

Artificial IntelligenceIndex Report 2023

Chapter 5: Education
5.2 K–12 AI Education

Narrative Highlight:

The State of International K–12 Education (cont’d)What might an actual K–12 AI curriculum look
like in practice? The UNESCO report includes
detailed information about a sample curriculumthat was deployed in Austria, the Austrian Data
Science and Artificial Intelligence curriculum.
As noted in the report:“The Austrian Data Science and Artificial
Intelligence curriculum includes digital basics suchas using an operating system to store and print
files, design presentations, and use spreadsheets
and word-processing software. It also covers
design and reflection on types and social issues indigital media, and safe digital media use. Students
in high school engage programming languages,
algorithms and simulations. They learn the basic
principles of data literacy, including collecting
data, structuring a spreadsheet, and carrying out
analyses and visualizations. They apply criteria
to evaluate the credibility and reliability of data

“They also gain anunderstanding of the
ethical dilemmas thatare associated with theuse of such technologies,
and become active
participants in social
discourse on theseissues.”sources as well as digital content. Students are
expected to know about careers in ICT, including
AI, and the social applications of emerging
technologies. They create digital media and learn
about the cloud and how to connect and network
computers. They also gain an understanding of
the ethical dilemmas that are associated withthe use of such technologies, and become active
participants in social discourse on these issues.
Finally, students are tasked with using technology
to make public statements and understand how
this reflects the democratic process.”

Table of Contents

Chapter 5 Preview

262

Artificial IntelligenceIndex Report 2023

Artificial IntelligenceIndex Report 2023

CHAPTER 6:

Policy andGovernance

Table of Contents

Chapter 6 Preview

263

Artificial IntelligenceIndex Report 2023

CHAPTER 6 PREVIEW:

Policy and Governance
Overview

265

6.2 National AI Strategies

285

Chapter Highlights

266

Aggregate Trends

285

6.1 AI and Policymaking

267

By Geographic Area

285

Global Legislation Records on AI

267

6.3 U.S. Public Investment in AI

286

269

Federal Budget for Nondefense AI R&D

286

U.S. Department of Defense
Budget Requests

287

U.S. Government AI-Related
Contract Spending

288

Total Contract Spending

288

By Geographic Area

Narrative Highlight: A Closer Look at
Global AI Legislation
270
United States Federal AI Legislation

271

United States State-Level AI Legislation 272
Narrative Highlight: A Closer Lookat State-Level AI Legislation

275
276

6.4 U.S. AI-Related Legal Cases

291

By Geographic Area

277

Total Cases

291

Narrative Highlight: A Closer Look
at Global AI Mentions

Geographic Distribution

292

279

Sector293

United States Committee Mentions

280

Type of Law

294

United States AI Policy Papers

283

Global AI Mentions

By Topic

284

Narrative Highlight: Three Significant
AI-Related Legal Cases
295
ACCESS THE PUBLIC DATA

Table of Contents

Chapter 6 Preview

264Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance

Overview
The growing popularity of AI has prompted intergovernmental, national, andregional organizations to craft strategies around AI governance. These actors are
motivated by the realization that the societal and ethical concerns surrounding AI
must be addressed to maximize its benefits. The governance of AI technologies has
become essential for governments across the world.
This chapter examines AI governance on a global scale. It begins by highlighting thecountries leading the way in setting AI policies. Next, it considers how AI has been
discussed in legislative records internationally and in the United States. The chapter
concludes with an examination of trends in various national AI strategies, followed
by a close review of U.S. public sector investment in AI.

Table of Contents

Chapter 6 Preview

265Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance

Chapter Highlights
Policymaker interestin AI is on the rise.
An AI Index analysis of the legislative recordsof 127 countries shows that the number of bills
containing “artificial intelligence” that werepassed into law grew from just 1 in 2016 to 37 in
2022. An analysis of the parliamentary records on
AI in 81 countries likewise shows that mentionsof AI in global legislative proceedings haveincreased nearly 6.5 times since 2016.

From talk to enactment—
the U.S. passed more
AI bills than ever before.
In 2021, only 2% of all federal AI bills in the
United States were passed into law. This number
jumped to 10% in 2022. Similarly, last year 35%
of all state-level AI bills were passed into law.

The U.S. government
continues to increasespending on AI.
Since 2017, the amount of U.S. government
AI-related contract spending has increasedroughly 2.5 times.

Table of Contents

Chapter 6 Preview

When it comes to AI,
policymakers havea lot of thoughts.
A qualitative analysis of theparliamentary proceedings of adiverse group of nations reveals
that policymakers think about AI
from a wide range of perspectives.
For example, in 2022, legislators in
the United Kingdom discussed therisks of AI-led automation; thosein Japan considered the necessityof safeguarding human rights inthe face of AI; and those in Zambia
looked at the possibility of using AI
for weather forecasting.

The legal world iswaking up to AI.
In 2022, there were 110 AI-related
legal cases in United States stateand federal courts, roughly seventimes more than in 2016. The
majority of these cases originatedin California, New York, and Illinois,and concerned issues relating tocivil, intellectual property, andcontract law.

266

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

In the last 10 years, AI governance discussions have accelerated, resulting in numerous policy proposals in various legislative bodies. Thissection begins by exploring the legislative initiatives related to AI that have been suggested or enacted in different countries and regions,followed by an in-depth examination of state-level AI legislation in the United States. The section then scrutinizes records of AI-related
discussions in parliaments and congresses worldwide and concludes with the number of AI policy papers published in the United States.

6.1 AI and Policymaking1
Global Legislative
Records on AI

passed at least one AI-related bill, and together they

The AI Index conducted an analysis of laws passed

Figure 6.1.2 shows that from 2016 to 2022, there hasby legislative bodies in 127 countries that contain the

been a sharp increase in the total number of AI-related

words “artificial intelligence” from 2016 to 2022.2

bills passed into law, with only one passed in 2016,

Of the 127 countries analyzed, since 2016, 31 haveclimbing to 37 bills passed in 2022.

have passed a total of 123 AI-related bills (Figure 6.1.1).

Number of AI-Related Bills Passed Into Law by Country, 2016–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report

0
1–5
6–10
11–15
16–25
No Available Data
Figure 6.1.1

1 Note that the analysis of passed AI policies may undercount the number of actual bills, given that large bills can include multiple sub-bills related to AI; for example, the CHIPS and ScienceAct passed by the U.S. in 2022.
2 The full list of countries analyzed is in the Appendix. The AI Index team attempted to research the legislative bodies of every country in the world; however, publicly accessible legislative
databases were not made available for certain countries.

Table of Contents

Chapter 6 Preview

267

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

Number of AI-Related Bills Passed Into Law in 127 Select Countries, 2016–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report

37
35

Number of AI-Related Bills

30

25

20

15

10

5

0
2016

2017

2018

2019

2020

2021

2022
Figure 6.1.2

Table of Contents

Chapter 6 Preview

268Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

By Geographic Area

the Philippines, which passed 5 and 4 laws, respectively.

Figure 6.1.3 shows the number of laws containing

Figure 6.1.4 shows the total number of laws passed

mentions of AI that were enacted in 2022. The United

since 2016. The United States leads the list with 22 bills,

States led the list with 9 laws, followed by Spain andfollowed by Portugal, Spain, Italy, and Russia.

Number of AI-Related Bills Passed Into Law in Select Countries, 2022
Source: AI Index, 2022 | Chart: 2023 AI Index Report

United States

9

Spain

5

Philippines

4

Andorra2

Belgium

2

Italy

2

Portugal

2

Russia2

United Kingdom

2Austria1

Croatia1

Germany

1

Kyrgyz Republic

1

Latvia1

Liechtenstein

1

Slovenia

1

0

1

2

3

4
5
Number of AI-Related Bills

6

7

8

9
Figure 6.1.3

Number of AI-Related Bills Passed Into Law in Select Countries, 2016–22 (Sum)
Source: AI Index, 2022 | Chart: 2023 AI Index Report

United States

22

Portugal

13

Spain

10

Italy

9

Russia9

Belgium

7

United Kingdom

6Austria

5

Korea, Rep.

5

Philippines

5

France

4

China

3

Germany

3

Japan

3

0

Table of Contents

2

4

6

Chapter 6 Preview

8

10
12
14
Number of AI-Related Bills

16

18

20

22
Figure 6.1.4

269

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

Narrative Highlight:

A Closer Look at Global AI Legislation
The following subsection delves into some of the AI-related legislation passed into law during 2022.
Figure 6.1.5 samples five different countries’ laws covering a range of AI-related issues.

AI-Related
AI-Related
AI-Related Legislation
Legislation From
From SelectSelect Countries,
Countries, 2022
2022

Source:AI
Index,
2022
Table:
2023
AI
Source:Source:AI
AIIndex,Index,2022
2022|||Table:
Table:2023
2023AIAIIndexIndexReport
ReportCountryCountryCountry

Bill
Bill
BillName
Name

Description
Description

KyrgyzRepublic
KyrgyzKyrgyzRepublic
Republic

About
About
About thethe Creative
Creative Industries
Industries Park
Park

This
and
operation
Thislaw
lawdetermines
determinesthethelegal
legalstatus,
status,management,
management,andandoperation
operation
procedures
establishedtoacceleratetheproceduresof
ofthe
theCreative
CreativeIndustries
IndustriesPark,
Park,establishedestablishedto
toaccelerate
acceleratethethedevelopmentofcreativeindustries,
includingarti�cialintelligence.
development of creative industries, includingarti�cialarti�cialintelligence.
intelligence.

LatviaLatviaLatviaAmendments
Amendments
Amendmentsto
tothe
theNational
NationalSecurity
SecurityLaw
Law

AAprovisioncommercial
companies,
provisionof
ofthisthisact
actestablishes
establishesrestrictions
restrictionson
oncommercial
commercialcompanies,
companies,
associations,
national
security,
includingassociations,and
andfoundations
foundationsimportant
importantfor
fornational
nationalsecurity,
security,including
includingaaa
commercial
intelligence.
commercialcompany
companythat
thatdevelops
developsarti�cialarti�cialintelligence.
intelligence.

Philippines
Philippines
Philippines

SecondSecondSecondCongressional
CongressionalCommission
Commissionon
onEducation
Education(EDCOM(EDCOM(EDCOMII)II)ActAct

AAprovisioncommission
toreview,
provisionof
ofthisthisact
actcreatescreatesaacongressional
congressionalcommissioncommissionto
toreview,
review,
assess,education;torecommend
assess,and
andevaluate
evaluatethe
thestatestateofofPhilippine
Philippineeducation;
education;totorecommend
recommend
innovativeeducation;
andtoappropriate
innovativeandandtargetedtargetedpolicy
policyreforms
reformsinineducation;
education;and
andto
toappropriate
appropriate
funds.
challenges
toeducation
funds.The
Theactactcalls
callsforforreforms
reformsto
tomeetmeetthethenew
newchallenges
challengesto
toeducation
education
causedcharacterized,
in
part,
by
the
causedby
bythe
theFourth
FourthIndustrial
IndustrialRevolution
Revolutioncharacterized,
characterized,in
inpart,part,by
bythe
the
rapid
developmentofarti�cialintelligence.
rapid development of arti�cial intelligence.

Spain
Spain
Spain

Right
Right
Righttotoequalequaltreatment
treatmentand
andnon-discrimination
non-discrimination

AAprovisionintelligencealgorithms
provisionof
ofthisthisact
actestablishes
establishesthat
thatarti�cialarti�cialintelligence
intelligencealgorithms
algorithms
involvedtakeintoaccount
involvedin
inpublic
publicadministrations’administrations’decision-making
decision-makingtake
takeinto
intoaccountaccount
bias-minimizationaccountability,
wheneverbias-minimizationcriteria,
criteria,transparency,transparency,andandaccountability,
accountability,wheneverwhenevertechnicallytechnicallyfeasible.
feasible.

UnitedStatesUnitedUnitedStates
StatesAI
AI
AITraining
TrainingAct
Act

This
BudgettoestablishorThisbillbillrequires
requiresthe
theO�ceO�ceofofManagement
Managementand
andBudget
Budgetto
toestablish
establishororotherwiseotherwiseprovideanAItraining
programfortheacquisition
workforceofotherwiseprovide
provideananAI
AItraining
trainingprogram
programfor
forthetheacquisitionacquisitionworkforce
workforceof
ofexecutive
executive
agencies(e.g.,
those
responsibleforprogrammanagement
or
executiveagencies
agencies(e.g.,(e.g.,those
thoseresponsible
responsiblefor
forprogram
programmanagement
managementor
orlogistics),
logistics),
withexceptions.
The
purposeoftheprogramisistotoensurethatlogistics),withwithexceptions.
exceptions.The
Thepurpose
purposeof
ofthe
theprogram
programis
toensure
ensurethat
thattheworkforce has
knowledge ofthe capabilities
and risks
associated withthetheworkforce
workforcehashasknowledge
knowledgeof
ofthe
thecapabilities
capabilitiesand
andrisks
risksassociatedassociatedwith
with
AI.
AI.
AI.

Figure 6.1.5

Table of Contents

Chapter 6 Preview

270Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

United States Federal
AI Legislation

just one federal bill was proposed, while in 2021, 134

A closer look at the U.S. federal legislative record

proposed bills. While fewer bills were proposed inshows a sharp increase in the total number of2022, the number of passed bills, which remained at

proposed bills that relate to AI (Figure 6.1.6). In 2015,

3 for each of the past four years, increased to 9.

bills were proposed. In 2022 this number fell to 88

Number of AI-Related Bills in the United States, 2015–22 (Proposed Vs. Passed)
Source: AI Index, 2022 | Chart: 2023 AI Index Report

140

120

Number of AI-Related Bills

100
88, Proposed
80

60

40

20
9, Passed
0
2015

2016

2017

2018

2019

2020

2021

2022
Figure 6.1.6

Table of Contents

Chapter 6 Preview

271

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

United States State-Level
AI Legislation

Maryland with 3. Figure 6.1.8 shows the total volume of

Figure 6.1.7 shows the number of laws containing

California, Massachusetts, and Washington. Figure

mentions of AI that were passed by U.S. states in

6.1.9 highlights the number of state-level AI-related

2022. California leads the list with 5, followed bybills passed by all states since 2016.

legislation passed from 2016 to 2022 for select states,with Maryland leading the list with 7 bills, followed byNumber of AI-Related Bills Passed Into Law in Select U.S. States, 2022
Source: AI Index, 2022 | Chart: 2023 AI Index Report

California

5Maryland3

Colorado

2

New Jersey

2

Washington

2

Alabama1

Hawaii

1

Idaho

1

Louisiana

1

Massachusetts

1

North Carolina

1Vermont

1
0

1

2

3
Number of AI-Related Bills

4

5
Figure 6.1.7

Table of Contents

Chapter 6 Preview

272Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

Number of AI-Related Bills Passed Into Law in Select U.S. States, 2016–22 (Sum)

Source: AI Index, 2022 | Chart: 2023 AI Index Report

Maryland7

California

6

Massachusetts

5

Washington

5

Illinois

3

Utah

3

Vermont3

Alabama2

Colorado2

Michigan

2

New Jersey

2

New York

2

North Carolina

2

Ohio

2

0

1

2

3
4
Number of AI-Related Bills

5

6

7
Figure 6.1.8(Sum)

Number of State-Level AI-Related Bills Passed Into Law in the
United States by State, 2016–22 (Sum)
Source: AI Index, 2022 | Chart: 2023 AI Index Report

ME
0

AK
0

CA
6

WA
5

MT
0

ND
1

SD
0

MN
0

WI
0

MI
2

OR
0

ID
1

WY
0

NE
0

IA
0

IL
3

IN
0

NV
1

UT
3

CO
2

KS
0

MO
0

KY
1

WV
1

AZ
0

NM
0

OK
0

AR
0

TN
0

VA
1

NC
2

TX
2

LA
1

MS
1

AL
2

GA
0

SC
0

HI
1

OH
2

VT
3

NH
0

MA
5

NY
2

CT
0

RI
0

PA
0

NJ
2

MD
7

DE
0

FL
0

Figure 6.1.9

Table of Contents

Chapter 6 Preview

273

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

Growing policy interest in AI can also be seen at the state level, with 60 AI-related bills proposed in 2022
(Figure 6.1.10)—a dramatic increase from the 5 bills proposed in 2015. Additionally, the proportion of bills being
passed has risen throughout the years. In 2015, 1 bill was passed, representing 16% of the total bills proposedthat year; while in 2022, 21 bills were passed, or 35% out of the total that were proposed.

Number of State-Level AI-Related Bills in the United States, 2015–22 (Proposed Vs. Passed)

Source: AI Index, 2022 | Chart: 2023 AI Index Report

60, Proposed

60

Number of AI-Related Bills

50

40

30

21, Passed

20

10

0
2015

2016

2017

2018

2019

2020

2021

2022
Figure 6.1.10

Table of Contents

Chapter 6 Preview

274Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

Narrative Highlight:

A Closer Look at State-Level AI Legislation
The following subsection highlights some of the AI-related legislation passed into law at the state level
during 2022. Figure 6.1.11 focuses on wide-ranging AI-related laws from five states around the country.

AI-Related Legislation
Legislation From
From SelectSelect States,
States, 2022
2022
AI-Related
Source:AI
AIIndex,Index,2022
2022|||Table:
Table:2023
2023AIAIIndexIndexReport
ReportSource:AI
Index,
2022
Table:
2023
AI
Index
ReportSource:

State
State
State

BillName
Name
Bill
Name
Bill

Description
Description

Alabama
Alabama
AlabamaArti�cialIntelligence,Intelligence,Limit
Limit thethe UseUse
Arti�cialIntelligence,
Limit
Arti�cialofFacial
FacialRecognition,
Recognition,totoEnsure
EnsureofFacialRecognition,toEnsureofArti�cialIntelligenceIntelligenceIs
IsNot
NotthetheOnly
Only
Arti�cialIntelligenceIs
NotArti�cial
BasisforforArrest
ArrestBasisforArrestBasis

This billbill prohibits
prohibits state
state oror local
local law
law enforcement
enforcement agencies
agenciesfrom
fromusingusingfacial
facialrecognition
recognition
This
matchresults
resultsas
asthe
thesolesolebasis
basisforformakingmakingananarrestarrestor
orforforestablishingestablishingprobableprobablecause
causeininaa
match
criminalinvestigation.
investigation.
criminal

California
California
California

BudgetAct
Actofof20222022
BudgetActof2022
Budgetprovisionof
ofthisthisappropriations
appropriationsbill
billforforthethe2022–23
2022–23�scal
�scalyear
yearallocates
allocates$1,300,000$1,300,000toto
AAprovision
CaliforniaState
StateUniversity,
University,Sacramento,
Sacramento,totoimprove
improvethe
thecampus
campuschildcare
childcarecenter,
center,California
includingthe
thedevelopment
developmentof
ofan
anarti�cialarti�cialintelligence
intelligencemixed-reality
mixed-realityclassroom.
classroom.
including

Maryland
Maryland
MarylandConservationFinance
FinanceAct
Act
Conservation
Finance
Act
Conservation

provisionof
ofthisthisact
actestablishes
establishesthat
thatthetheDepartment
Departmentof
ofNatural
NaturalResources
Resourcesshall
shallstudy
study
AAprovisionandassessassessthethepotential
potentialfor
fordigital
digitaltools
toolsandandplatforms
platformsincluding
includingarti�cialarti�cialintelligence
intelligenceand
andandmachine learninglearning toto contributecontribute toto Chesapeake
Chesapeake Bay
Bay restoration
restorationand
andclimate
climatesolutions.
solutions.
machine

NewJersey
JerseyNew
JerseyNew

21stCentury
CenturyIntegrated
IntegratedDigital
Digital21st
Century
Integrated
Digital
21stExperienceAct
Act
ExperienceAct
Experience

provisionof
ofthisthisact,
act,whichwhichconcerns
concernsthe
themodernization
modernizationof
ofstatestategovernment
governmentwebsites,
websites,
AAprovision
establishesthat
thatthethechief
chieftechnology
technologyo�cer,o�cer,ininconsultation
consultationwith
withthethechief
chiefinnovation
innovation
establisheso�cerandandthetheNew
NewJersey
JerseyInformation
InformationTechnology
TechnologyProject
ProjectReview
ReviewBoard,
Board,shall
shallevaluate
evaluate
o�cer
on an
an annualannual basis
basis thethe feasibility
feasibility ofof statestate agencies
agencies using
using arti�cialarti�cialintelligence
intelligenceand
andonmachinelearninglearningto
toprovide
providepublic
publicservices.
services.
machineVermont
Vermont
Vermont

AnAct
ActRelating
Relatingto
tothe
theUseUseandandAn
Act
Relating
to
the
Use
and
An
Oversightof
ofArti�cial
Arti�cialIntelligenceIntelligenceininOversightofArti�cialIntelligence
Oversight
StateGovernment
Government
State
Government
StateThis act
act createscreates thethe Division
Division ofof Arti�cial
Arti�cial IntelligenceIntelligence within
withinthe
theAgency
Agencyof
ofDigital
DigitalServices
ServicesThis
toreview
reviewall
allaspects
aspectsof
ofarti�cialarti�cialintelligence
intelligencedeveloped,
developed,employed,
employed,ororprocured
procuredby
bythe
the
to
stategovernment.
government.The
Theactactrequires
requiresthe
theDivision
Divisionof
ofArti�cial
Arti�cialIntelligenceIntelligenceto,to,among
amongother
other
state
things,propose
proposeaastate
statecode
codeofofethics
ethicson
onthe
theuseuseof
ofarti�cialarti�cialintelligence
intelligencein
instatestate
things,government andand makemake recommendations
recommendations toto thethe General
General Assembly
Assemblyon
onpolicies,
policies,laws,
laws,andandgovernment
regulationsregarding
regardingarti�cialarti�cialintelligence
intelligencein
instatestategovernment.
government.
regulations

Figure 6.1.11

Table of Contents

Chapter 6 Preview

275

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

Global AI Mentions
Another barometer of legislative interest is thecontain the keyword “artificial intelligence” from 2016

number of mentions of “artificial intelligence” into 2022.3 Figure 6.1.12 shows that mentions of AI ingovernmental and parliamentary proceedings. The

legislative proceedings in these countries registered a

AI Index conducted an analysis of the minutes orsmall decrease from 2021 to 2022, from 1,547 to 1,340.

proceedings of legislative sessions in 81 countries that

Number of Mentions of AI in Legislative Proceedings in 81 Select Countries, 2016–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report

1,600

1,400
1,340

Number of Mentions

1,200

1,000

800

600

400

200
2016

2017

2018

2019

2020

2021

2022
Figure 6.1.12

3 The full list of countries that was analyzed is in the Appendix. The AI Index research team attempted to review the governmental and parliamentary proceedings of every country in theworld; however, publicly accessible governmental and parliamentary databases were not made available for all countries.

Table of Contents

Chapter 6 Preview

276Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

By Geographic Area
Figure 6.1.13 shows the number of legislative proceedings containing mentions of AI in 2022.4 From the 81
countries considered, 46 had at least one mention, and Spain topped the list with 273 mentions, followed by
Canada (211), the United Kingdom (146), and the United States (138).

Number of Mentions of AI in Legislative Proceedings by Country, 2022
Source: AI Index, 2022 | Chart: 2023 AI Index Report0
1–55
56–110
111–165
166–220
221–280
No Available Data
Figure 6.1.13

4 For mentions of AI in legislative proceedings around the world, the AI Index performed searches of the keyword “artificial intelligence,” in the respective languages, on the websites ofdifferent countries’ congresses or parliaments, usually under sections named “minutes,” “Hansard,” etc.

Table of Contents

Chapter 6 Preview

27 7

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

Figure 6.1.14 shows the total number of AI mentions in the past seven years. Of the 81 countries considered, 62 had
at least one mention, and the United Kingdom dominates the list with 1,092 mentions, followed by Spain (832), theUnited States (626), Japan (511), and Hong Kong (478).

Number of Mentions of AI in Legislative Proceedings by Country, 2016–22 (Sum)
Source: AI Index, 2022 | Chart: 2023 AI Index Report0
1–220
221–440
441–660
661–880
881–1100
No Available Data
Figure 6.1.14

Table of Contents

Chapter 6 Preview

278Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

Narrative Highlight:

A Closer Look at Global AI Mentions
The following subsection examines mentions of AI in government proceedings in 2022. Figure 6.1.15
quotes discussions across a geographically diverse set of countries.
AI-Related
AI-Related
AI-Related Parliamentary
ParliamentaryMen
Mentions
tionsFrom
FromSelectSelectCountries,
Countries,2022
2022
Source:AI
Index,
2022
Table:
2023
AI
Index
ReportSource:Source:AI
AIIndex,Index,2022
2022 |||Table:
Table:2023
2023AIAIIndexIndexReport
ReportCountryCountryCountry

Legislature
Legislature
Legislature

Speaker
Speaker
Speaker

Quote
Quote

AgendaAgendaItem
Item

Australia
Australia
Australia

House
of
House
Houseofof
Representatives
Representatives
Representatives

Ed
Husic,
Australian
Ed
EdHusic,
Husic,Australian
AustralianLabor
Labor
Party,
MinisterParty,
Party,Minister
Ministerfor
forIndustry
Industry
andScienceandandScience
Science“Working“Workingwith
withourourinternational
internationalpartners
partnerswe
wecan
can
transform
transformAustralian
Australianknow-how
know-howinto
intoglobally
globallyrecognised
recognised
skillsskillsandandmanufacturing
manufacturinginindefence
defenceindustries.
industries.And
Andwe
we
can
canbuild
buildononour
ourundeniable
undeniableexpertise
expertiseininareas
areaslikelike
quantumquantumtechnologies,
technologies,robotics
roboticsand
andartiarti cial
cialintelligence.
intelligence.We
Wewillwillseek
seektotopartner
partnerwith
withindustry
industryand
and
state
stateand
andterritory
territorygovernments
governmentstotoidentify
identifyinvestment
investment
opportunities
opportunitieswithin
withinpriority
priorityareas.
areas.An
Anon-ramp,
on-ramp,ififyou
you
will,
will,ofofturn-key
turn-keyopportunities
opportunitiesfor
forinvestment
investmenttotomake
make
sure
the
NRF
is
well
placedforsuccess.”
sure the NRF is well placed for success.”

National
NationalReconstruction
Reconstruction
Fund
FundCorporation
CorporationBill
Bill 2022
2022- Second
Reading- Second
ReadingBrazilBrazilBrazil

Diary
of
the
Diary
Diaryofofthe
theChamberoftheChamber
Chamberof
ofthetheMembers
Members
Members

Mr.
Gustavo
Fruet,
Mr.
Mr.Gustavo
GustavoFruet,
Fruet,
Democratic
Labor
Democratic
DemocraticLabor
LaborParty
Party“There“Therehashasbeenbeenaalot
lotofoftalk
talkabout
aboutthethefuture
futureofofwork
workduedue
tototechnology.
technology.InInthethebook
bookTheTheFourth
FourthIndustrial
Industrial
Revolution,
Revolution,Klaus
KlausSchwab
Schwabeven
evenpoints
pointsoutoutprofessions
professions
that
thatwill
willbebeextinct
extinctand
andprofessions
professionsthat
thatwill
willdemand
demand
more
and
more
quali�cations,
in
times
of
5G,
more and more quali�cations, in times of 5G,Internet
Internetofof
ThingsThingsandandArti
Arti cial
cialIntelligence.
Intelligence.InInthis
thissense,
sense,ititisisgood
good
totohighlighthighlightthat
thatthethepandemic,
pandemic,among
amongother
othercontradictions,contradictions,ended
endedupupanticipating
anticipatingthe
theuseuseofoftechnology,
technology,especially
especiallyininthe
thetelework.”
telework.”

Presentation
PresentationofofBill
BillNo.
No.
135,135,ofof2022,
2022,ononthe
theamendment
amendmentofofthe
theCLT
CLT- Consolidation
ConsolidationofofLabor
Labor
Laws,
Laws,withwithaaview
viewtoto
granting
grantingtelework
teleworktoto
parents
parentsofofchildren
childrenup
uptoto88
years
yearsold
old

Japan
Japan
Japan

210th Session
Session of
210th
210th Sessionofofthe Diet
Diet House
House ofthethe Diet HouseofofCouncilors
Councilors
Councilors
Commission ononCommission
Commission on
the Constitution
Constitution
thethe Constitution
No. 22No.
No. 2

Kohei Otsuka,
Otsuka, Democratic
Kohei
Kohei Otsuka, Democratic
Party forfor thethe People,
Party
Party for thePeople,
People,Shinryokufukai
Shinryokufukai
Shinryokufukai“In the �eld of human rights, we believe that it is“In the �eld of human rights, we believe that it isnecessarynecessaryto
toupdate
updatehuman
humanrights
rightsguarantees
guaranteesininorder
ordertoto
respond
respondto
tochanges
changesininthe
thetimes
timesthatthatwere
wereunpredictable
unpredictablewhen
whenthetheConstitution
Constitutionwas
wasenacted.
enacted.InInparticular,
particular,asasthe
thefusion of arti cial intelligence and Internet technology
fusion of arti cial intelligence and Internet technology
progresses,
progresses,thetheinternational
internationalcommunity
communityisisconcerned
concerned
about
aboutthetheproblems
problemsofofindividual
individualscoringscoringand
anddiscrimination,andtheproblemofInternetadvertising
discrimination, and the problem of Internet advertisingthat
thatunfairly
unfairlyin�uencesin�uencesthe
thevotingvotingbehavior
behaviorofofcitizens.
citizens.
We need a constitutional argument to guarantee theWe need a constitutional argument to guarantee theautonomous
autonomousdecision-making
decision-makingofofindividuals
individualsand
andprotect
protect
basic
basicdatadatarights
rightsininthe
thedigital
digitalage.”
age.”

The Commission on the
The Commission on theConstitution
Constitution

UnitedUnitedUnitedKingdom
Kingdom
KingdomHouse ofofHouse
House ofCommons
Commons
Commons

Dame Angela
AngelaDame
Dame AngelaEagle,
Eagle,Labor
Labor

“What
“Whatwould
wouldbebethe
theuseuseofofarti
arti cial
cialintelligence
intelligenceinintrying
tryingto decide how automated these things could become?
to decide how automated these things could become?
Would there be worries about over-automation? How
Would there be worries about over-automation? How
would that be looked at in terms of regulation? How
would that be looked at in terms of regulation? How
open are we going to be about the way in which AI is
open are we going to be about the way in which AI isapplied and how it might evolve in ways that might
applied and how it might evolve in ways that mightembed discrimination such that we get a system where
embed discrimination such that we get a system wherecertain people may be discriminated against andcertain people may be discriminated against andexcluded?”
excluded?”

Financial
FinancialServices
ServicesandandMarkets Bill (FourthMarkets Bill (Fourth
Sitting)
Sitting)

ZambiaZambiaZambiaThe House,
House,The
The House,
National
National
National
Assembly
Assembly
Assembly

Hon. Collins
Collins Nzovu,
Nzovu, United
Hon.
Hon. Collins Nzovu, United
Party forfor National
National
Party
Party for National
Development,
Development,
Development, The
Minister ofof Green
Green
Minister
Minister of Green
Economy andand Environment
Environment
EconomyEconomy and Environment“Madam Speaker, in order to enhance quality and“Madam Speaker, in order to enhance quality andaccuracy of weather forecast, the Government, withaccuracy of weather forecast, the Government, with
�nancial support from the United Nations Development
�nancial support from the United Nations Development
Programme Strengthening Climate Resilience ofProgramme Strengthening Climate Resilience ofAgricultural Livelihoods
Livelihoods inin Agro-EcologicalAgro-Ecological (UNDP(UNDP
Agricultural
Agricultural Livelihoods in Agro-Ecological (UNDP
SCRALA) projectproject is
is currently
currently partnering
partnering with
with thethe
SCRALA)SCRALA) project is currently partnering with the
University ofof Zambia
Zambia (UNZA)(UNZA) toto develop
develop aa seasonalseasonal
University
University of Zambia (UNZA) to develop a seasonal
weather forecasting
forecasting systemsystem using
using arti
arti cial
cial intelligence.”
intelligence.”
weather
weather forecasting system using arti cial intelligence.”

Ministerial Statements;
Ministerial Statements;
Weather and Climate
Weather and Climate
Services and theServices and the2022/2023 rainfall forecast
2022/2023 rainfall forecast

Figure 6.1.15

Table of Contents

Chapter 6 Preview

279

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

United States
Committee Mentionscommittees that address legislative and other policy

An additional indicator of legislative interest is the

Figure 6.1.16 shows a sharp increase in the total

number of mentions of “artificial intelligence” innumber of mentions of AI within committee reports

committee reports produced by House and Senate

beginning with the 115th legislative session.

issues, investigations, and internal committee matters.

Mentions of AI in U.S. Committee Reports by Legislative Session, 2001–22

Source: AI Index, 2022 | Chart: 2023 AI Index Report

80
73

70

Number of Mentions

60
50
40
30
20
10
0
107th
(2001–02)

108th(2003–04)

109th(2005–06)

110th(2007–08)

111th(2009–10)112th(2011–12)113th(2013–14)

114th(2015–16)

115th(2017–18)

116th(2019–20)

117th(2021–22)
Figure 6.1.16

Table of Contents

Chapter 6 Preview

280

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

Figure 6.1.17 shows the mentions in committee reports for the 117th Congressional Session, which took placefrom 2021 to 2022. The Appropriations Committee leads the House reports, while the Homeland Security andGovernmental Affairs Committee leads the Senate reports (Figure 6.1.18).

Mentions of AI in Committee Reports of the U.S. House of Representatives for the 117th Congressional
Session, 2021–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report

Appropriations

20

Science, Space, and Technology

9

Rules

5

Armed Services

3

Transportation and Infrastructure

3Intelligence (Permanent Select)2

Natural Resources

2

Oversight and Accountability

2

Budget1

Education and the Workforce

1

Energy and Commerce

1

Financial Services

1

Foreign A airs

1Homeland Security

1

House Administration

1

Small Business

1

Ways and Means

1

0

1

2

3

4

5

6

7

8

9
10
11
12
Number of Mentions

13

14

15

16

17

18

19

20

21

Figure 6.1.17

Mentions of AI in Committee Reports of the U.S. Senate for the 117th Congressional Session, 2021–22

Source: AI Index, 2022 | Chart: 2023 AI Index Report

Homeland Security andGovernmental A airs

8

Appropriations

3

Commerce, Science,
and Transportation

3

Armed Services

2Intelligence (Select)

20

Table of Contents

1

2

Chapter 6 Preview

345
Number of Mentions

6

7

8
Figure 6.1.18

281

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

Figure 6.1.19 shows the total number of mentions in committee reports from the past 10 congressional sessions,which took place from 2001 to 2022. The House and Senate Appropriations Committees, which regulate
expenditures of money by the government, lead their respective lists (Figure 6.1.19 and 6.1.20).

Mentions of AI in Committee Reports of the U.S. Senate, 2001–22 (Sum)
Source: AI Index, 2022 | Chart: 2023 AI Index Report

Appropriations

16

Homeland Security andGovernmental A airs

11

Armed Services

10

Commerce, Science,
and Transportation

9

Energy andNatural Resources

7Intelligence (Select)

5
0

2

4

6

8
10
Number of Mentions

12

14

16
Figure 6.1.19

Mentions of AI in Committee Reports of the U.S. House of Representatives, 2001–22 (Sum)

Source: AI Index, 2022 | Chart: 2023 AI Index Report

Appropriations

45

Science, Space, and Technology

27

Rules

14

Armed Services

9

Energy and Commerce

8

Oversight and Accountability

8

Financial Services

6Intelligence (Permanent Select)

6

Education and the Workforce

4

Transportation and Infrastructure4Homeland Security

3

Veterans’ A airs

3

Budget2

Foreign A airs

2

Judiciary

2Natural Resources

2

Small Business

2

Ways and Means

2

Agriculture

1

House Administration

1

0

Table of Contents

2

4

6

8

Chapter 6 Preview

10

12

14

16

18

20 22 24 26 28
Number of Mentions

30

32

34

36

38

40

42

44

46

Figure 6.1.20

282

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

United States AI Policy Papers
To estimate activities outside national governmentsthat are also informing AI-related lawmaking,
the AI Index tracked 55 U.S.-based organizations
that published policy papers in the past five
years. Those organizations include: think tanks

post that addresses issues related to AI and makes
specific recommendations to policymakers. Topics of
those papers are divided into primary and secondary
categories: A primary topic is the main focus of thepaper, while a secondary topic is a subtopic of thepaper or an issue that is briefly explored.

and policy institutes (19); university institutes andFigure 6.1.21 highlights the total number of U.S.-based,

research programs (14); civil society organizations,

AI-related policy papers published from 2018 to 2022.

associations, and consortiums (9); industry andAfter a slight dip from 2020 to 2021, the total increased

consultancy organizations (9); and government

to 284 in 2022. Since 2018, the total number of suchagencies (4). A policy paper in this section is defined

papers has increased 3.2 times, signaling greater

as a research paper, research report, brief, or blog

interest over time.

Number of AI-Related Policy Papers by U.S.-Based Organizations, 2018–22
Source: Stanford Institute for Human-Centered AI (HAI) Policy and Society | Chart: 2023 AI Index Report

300
284

Number of Policy Papers

250

200

150

100

50

0

2018

2019

2020

2021

2022
Figure 6.1.21

Table of Contents

Chapter 6 Preview

283

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.1 AI and Policymaking

By Topic
In 2022, the most frequent primary topics were

sat in fourth position as of 2022. All of these leadingindustry and regulation (107), innovation andtopics were also well represented as secondary topics.

technology (90), and government and publication

Topics that received comparatively little attention

administration (82) (Figure 6.1.22). Privacy, safety, andincluded social and behavioral sciences; humanities;

security, which was the most reported topic in 2021,

and communications and media.

Number of AI-Related Policy Papers by U.S.-Based Organization by Topic, 2022
Source: Stanford Institute for Human-Centered AI (HAI) Policy and Society | Chart: 2023 AI Index Report

Primary Topic

Secondary Topic

Industry and Regulation

107

Innovation and Technology

90

Gov’t and Public Administration

69

82

Privacy, Safety, and Security

25

59

Ethics59

39

Equity and Inclusion

34

Int’l A airs and Int’l Security

34

Workforce and Labor

17
18

65

30

Democracy

26

26

Health and Biological Sciences

13

Justice and Law Enforcement

12

Physical Sciences

5
10
13

9

Education and Skills

8

Energy and Environment

4Social and Behavioral Sciences

3

10
8
1

Humanities 1

1

Communications and Media
0

50

3

20

40

60

80

100
0
20
Number of Policy Papers

40

60

80

100

Figure 6.1.22

Table of Contents

Chapter 6 Preview

284

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.2 National AI Strategies

This subsection presents an overview of national AI strategies—policy plans developed by a country’s government to steer thedevelopment and deployment of AI technologies within its borders. Tracking trends in national strategies can be an important way ofgauging the degree to which countries are prioritizing the management and regulation of AI technologies. Sources include websites ofnational or regional governments, the OECD AI Policy Observatory (OECD.AI), and news coverage. “AI strategy” is defined as a policy
document that communicates the objective of supporting the development of AI while also maximizing the benefits of AI for society.5

6.2 National AI Strategies
Aggregate Trends

Yearly Release of AI National Strategies by Country

Canada officially launched the first national AI strategy

Year

Countryin March of 2017; since then a total of 62 national
AI strategies have been released (Figure 6.2.1). The

Source: AI Index, 2022 | Table: 2023 AI Index Report2017

Canada, China, Finland

2018

Australia, France, Germany, India, Mauritius, Mexico, Sweden

2019

Argentina, Austria, Bangladesh, Botswana, Chile, Colombia,
Cyprus, Czech Republic, Denmark, Egypt, Estonia, Japan,
Kenya, Lithuania, Luxembourg, Malta, Netherlands, Portugal,
Qatar, Romania, Russia, Sierra Leone, Singapore, United ArabEmirates, United States of America, Uruguay

2020

Algeria, Bulgaria, Croatia, Greece, Hungary, Indonesia, Latvia,
Norway, Poland, Saudi Arabia, Serbia, South Korea, Spain,
Switzerland

2021

Brazil, Ireland, Peru, Philippines, Slovenia, Tunisia, Turkey,
Ukraine, United Kingdom, Vietnam

2022

Italy, Thailand

number of released strategies peaked in 2019.

By Geographic Area
Figure 6.2.2 highlights the countries which, as ofDecember 2022, have either released or developed
a national AI strategy. Figure 6.2.3 enumerates thecountries that, in 2021 and 2022, pledged to develop
an AI strategy . The first nations to officially releasenational AI strategies were Canada, China, and Finlandin 2017. Only two nations released national AI strategies
in 2022: Italy and Thailand.
Countries With a National Strategy on AI, 2022
Source: AI Index, 2022 | Chart: 2023 AI Index Report

Figure 6.2.1

AI National Strategies in Development by Country
and Year
Source: AI Index, 2022 | Table: 2023 AI Index Report

Year

Country2021

Armenia, Bahrain, Cuba, Iceland, Morocco, New Zealand, Oman

2022

Azerbaijan, Belgium, Benin, Israel, Jordan, Nigeria, Uzbekistan

Figure 6.2.3

Released
In Development
Not Released

Figure 6.2.2

5 The AI Index research team made efforts to identify whether there was a national AI strategy that was released or in development for every nation in the world.
It is possible that some strategies were missed.

Table of Contents

Chapter 6 Preview

285Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance6.3 U.S. Public Investment in AI

This section examines public AI investment in the United States based on data from the U.S. government and Govini, a company that uses
AI and machine learning technologies to track U.S. public and commercial spending.

6.3 U.S. Public Investment in AI
Federal Budget for
Nondefense AI R&D

information on classified AI R&D investment by

In December 2022, the National Science andIn fiscal year (FY) 2022, nondefense U.S. government

defense and intelligence agencies.

agencies allocated a total of $1.7 billion to AI R&D

Technology Council published a report on the
public-sector AI R&D budget across departments
and agencies participating in the Networking andInformation Technology Research and Development(NITRD) Program and the National Artificial

spending (Figure 6.3.1). The amount allocated in FY
2022 represented a slight decline from FY 2021 anda 208.9% increase from FY 2018. An even greater
amount, $1.8 billion, has been requested for FY 2023.

Intelligence Initiative. The report does not include

U.S. Federal Budget for Nondefense AI R&D, FY 2018–23
Source: U.S. NITRD Program, 2022 | Chart: 2023 AI Index Report

1.84

Budget (in Billions of U.S. Dollars)

1.50

1.75

1.73

FY21 (Enacted)

FY22 (Enacted)

1.43

1.11
1.00

0.56
0.50

0.00

FY18 (Enacted)

FY19 (Enacted)

FY20 (Enacted)

FY23 (Requested)
Figure 6.3.16

6 A previous report on the public-sector AI R&D budget released in 2021 classed the FY21 spending as totaling $1.53 billion. However, the most recent report,
released in 2022, upgraded the total spent in 2022 to $1.75 billion.

Table of Contents

Chapter 6 Preview

286Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance6.3 U.S. Public Investment in AI

U.S. Department of Defense
Budget Requests
Every year the DoD releases the amount of funding

to the 2022 report, the DoD requested $1.1 billion in FY

they have requested for nonclassified AI-specific2023, a 26.4% increase from the funding they received

research, development, test, and evaluation. According

in FY 2022 (Figure 6.3.2).

U.S. DoD Budget Request for AI-Specific Research, Development, Test, and Evaluation (RDT&E), FY 2020–23
Source: U.S. Office of the Under Secretary of Defense (Comptroller), 2022 | Chart: 2023 AI Index Report

1.10

Budget Request (in Billions of U.S. Dollars)

1.00

0.93
0.84

0.87

0.80

0.60

0.40

0.20

0.00

FY20 Funding

FY21 Funding

FY22 Funding

FY23 Funding
Figure 6.3.2

Table of Contents

Chapter 6 Preview

287Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance6.3 U.S. Public Investment in AI

U.S. Government AI-Related
Contract Spending

volumes of federal contracts data, including prime

Public investment in AI can also be measured byanalyze data that is otherwise often inaccessible.

federal government spending on the contracts

contracts, grants, and other transaction authority(OTA) awards. The use of AI models enables Govini toTotal Contract Spendingthat U.S. government agencies award to privatecompanies for the supply of goods and services. Suchcontracts typically occupy the largest share of an

Figure 6.3.3 highlights total U.S. government spending
on AI, subdivided by various AI segments. From 2021

agency’s budget.

to 2022, total AI spending increased from $2.7 billion

Data in this section comes from Govini, which creatednearly 2.5 times. In 2022, the AI subsegments that saw

a taxonomy of spending by the U.S. government on

the greatest amount of government spending included

critical technologies including AI. Govini applied

decision science ($1.2 billion), and computer vision

supervised machine learning and natural language($0.8 billion).

to $3.3 billion. Since 2017, total spending has increased

processing to parse, analyze, and categorize large

U.S. Government Spending by Segment, FY 2017–22
Source: Govini, 2022 | Chart: 2023 AI Index Report

Decision Science

Computer Vision

Machine Learning

Autonomy

Natural Language Processing

U.S. Government Spending (in Billions of U.S. Dollars)

3.50
3.28
0.17
3.00
2.70
2.50

2.41
0.21

2.00

1.83
1.56

1.50
1.29
1.00

0.24

0.31
0.44

0.50

0.00

2017

0.41

0.43
0.3

0.82

0.58
0.53

0.45
0.46
0.41

0.43
0.32

0.52

0.43

0.26

0.21

0.69

0.21

1.01
0.46

0.55

2018

2019

1.19

0.73

2020

2021

2022
Figure 6.3.3

Table of Contents

Chapter 6 Preview

288Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance6.3 U.S. Public Investment in AI

Figure 6.3.4 shows U.S. government spending by AI segment in FY 2021 and FY 2022. Spending increased
for the decision science, computer vision, and autonomy segments, while spending on machine learning, andnatural language processing dropped slightly.

U.S. Government Spending by Segment, FY 2021 Vs. 2022
Source: Govini, 2022 | Chart: 2023 AI Index Report

1.19 (+18%)

Decision Science

1.01

0.82 (+55%)

Computer Vision

0.53

0.69 (+33%)

Autonomy

0.52

0.41 (-5%)

Machine Learning

Natural Language
Processing
0.00

0.43

0.17 (-19%)

2022
2021

0.21
0.20

0.40
0.60
0.80
U.S. Government Spending (in Billions of U.S. Dollars)

1.00

1.20

Figure 6.3.4

Table of Contents

Chapter 6 Preview

289

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance6.3 U.S. Public Investment in AI

In FY 2022, the majority of federal AI contracts were prime contracts (62.5%), followed by grants (34.9%) andother transaction authority (OTA) awards (2.6%) (Figure 6.3.5). From FY 2021 to FY 2022, the share of contracts
remained about the same, while the share of grants rose.

Total Value of Contracts, Grants, and OTAs Awarded by the U.S. Government for AI/ML and Autonomy,
FY 2017–22
Source: Govini, 2022 | Chart: 2023 AI Index Report

2.05, Contracts

Total Value Awarded (in Billions of U.S. Dollars)

2.00

1.50

1.15, Grants
1.00

0.50

0.09, OTAs
0.00
2017

2018

2019

2020

2021

2022
Figure 6.3.5

Table of Contents

Chapter 6 Preview

290Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.4 U.S. AI-Related Legal Cases

In 2022, the AI Index partnered with Elif Kiesow Cortez, a scholar of artificial intelligence law, in a research project tracking trends in
American legal cases from 2000 to 2022 that contain AI-related keywords.7

6.4 U.S. AI-Related Legal Cases
Total Cases

there were a total of 110 AI-related cases in U.S.

In the last few years, there has been a sharp spike infederal and state courts, 6.5 times more than in 2016

AI-related jurisprudence in the United States. In 2022,(Figure 6.4.1).

Number of AI-Related Legal Cases in the United States, 2000–22
Source: AI Index, 2022 | Chart: 2023 AI Index Report

110

Number of AI-Related Legal Cases

100

80

60

40

20

0
2000

2002

2004

2006

2008

2010

2012

2014

2016

2018

2020

2022
Figure 6.4.1

7 The Index analyzed both federal and state-level cases. Specific keywords in the search included “artificial intelligence,” “machine learning,” and “automated decision-making.” Some of thesecases did not directly concern issues related to AI jurisprudence. As a next step of this project, we will aim to identify the cases that most centrally concern issues of AI-related law.

Table of Contents

Chapter 6 Preview

291Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.4 U.S. AI-Related Legal Cases

Geographic Distribution

they are home to many large businesses that have

In 2022, the majority of AI-related legal cases

greater number of AI-related legal cases originating

originated in California (23), Illinois (17), and New

from Illinois—this follows the state’s enactment

York (11) (Figure 6.4.2). The aggregate number of AI-

of the Biometric Information Privacy Act (BIPA),

related cases since 2000 show a similar geographic

which requires that companies doing business in

distribution (Figure 6.4.3). California and New York’s

Illinois follow a number of regulations related to theinclusion in the top three is unsurprising given thatcollection and storage of biometric information.

integrated AI. In recent years, there have been a

Number of AI-Related Legal Cases in the United States by State, 2022
Source: AI Index, 2022 | Chart: 2023 AI Index Report

California

23

Illinois

17

New York

11

Delaware7

Florida7

Washington

5

Kansas4

Massachusetts

4

Maryland4

District of Columbia

3

Texas

3

Ohio

3

Pennsylvania

3

Virginia
Missouri
0

2

2

2

4

6

8

10
12
14
Number of AI-Related Legal Cases

16

18

20

22

24
Figure 6.4.28

8 Figures 6.4.2 and 6.4.3 include information for states and districts, given that cases sometimes originate from American districts like the District of Columbia or Puerto Rico

Table of Contents

Chapter 6 Preview

292Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.4 U.S. AI-Related Legal Cases

Number of AI-Related Legal Cases in the United States by State, 2000–22 (Sum)
Source: AI Index, 2022 | Chart: 2023 AI Index Report

California

127

New York

66

Illinois

36

Texas

26

Delaware19

Massachusetts

19

Washington

18

Pennsylvania

16

Michigan

12

Virginia

12

District of Columbia

12

Florida

12

Ohio

10

Kansas9

Minnesota

8

0

10

20

30

40

50
60
70
80
Number of AI-Related Legal Cases

90

100

110

120

130
Figure 6.4.3

SectorFigure 6.4.4 groups U.S.-based legal cases by economic sector. The predominant sector in 2022 was financial
services and professional services (48 cases); followed by media, culture, graphical (18); and public service (14).

Sector at Issue in AI-Related Legal Cases in the United States, 2022
Source: AI Index, 2022 | Chart: 2023 AI Index Report

Financial Services, Professional Services

48

Media, Culture, Graphical18

Public Service

14

Education

6

Health Services

6

Hotels, Catering, Tourism

4

Postal and Telecommunications Services

4

Transport (Including Civil Aviation,
Railways, Road Transport)

3

Food, Drink, Tobacco

2

Transport Equipment Manufacturing

2

Basic Metal Production

1

Mechanical and Electrical Engineering

1

Oil and Gas Production, Oil Re ning

10

Table of Contents

4

Chapter 6 Preview

8

12

16

20
24
28
32
Number of AI-Related Legal Cases

36

40

44

48
Figure 6.4.4

293Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.4 U.S. AI-Related Legal Cases

Type of Law
The greatest proportion of AI-related legal cases concerned civil law (29%) (Figure 6.4.5). There were also a large
number of AI-related legal cases in the domain of intellectual property (19%), as well as contract law (13.6%).

Area of Law of AI-Related Legal Cases in the United States, 2022
Source: AI Index, 2022 | Chart: 2023 AI Index Report

Civil

32

Intellectual
Property

21

Contract

15

Competition

11

Constitutional

8

Employment and
Labor

6

Criminal

5

Corporate

4

Financial

3

Human Rights andImmigration

2

Terrorism andNational Security

2

Tort
0

Table of Contents

1
2

4

6

8

Chapter 6 Preview

10

12

14
16
18
20
Number of AI-Related Legal Cases

22

24

26

28

30

32
Figure 6.4.5

294

Artificial IntelligenceIndex Report 2023

Chapter 6: Policy and Governance
6.4 U.S. AI-Related Legal Cases

Narrative Highlight:

Three Significant AI-Related Legal Cases
The section below profiles three significant AI-related cases in the United States,
highlighting some of the legal issues that are at stake when AI is brought into the courts.

Duerr v. Bradley University (2022-

Flores v. Stanford9 (2021-Sep-28)

Dyroff v. Ultimate Software Grp., Inc

Mar-10) – United States Court of– United States Court of Appeals(2017-Nov-26) – United States Court ofAppeals for the Seventh Circuit

for the Second Circuit

Appeals for the Ninth Circuit

The plaintiffs, who were enrolled

The plaintiffs, offenders denied

Plaintiff Kristanalea Dyroff sued Ultimate

as undergraduates in a private

parole, sued the New York State

Software after her 29-year-old son died

university in Peoria, Illinois, during

Board of Parole over beingfrom an overdose of heroin laced withthe fall 2020 semester, were told

refused access to information

fentanyl, which he allegedly bought

to use a third-party proctoring

used by the board in its review

from a drug dealer that he encountered

tool called Respondus Monitor forof their cases. Northpointe, Inc.,on Ultimate Software’s social network

remote, online exams. This tool

petitioned the court as a non-site. Dyroff asserted seven claims

made use of artificial intelligence

party because its Correctional

against Ultimate Software which

technologies. The plaintiffs claimed

Offender Management Profilingincluded negligence, wrongful death,that the defendants violated Illinois’for Alternative Sanctions

and civil conspiracy. At the core of these

Biometric Information Privacy Act(COMPAS), an AI-powered

claims was the argument that Ultimate(BIPA) by not adequately following

risk assessment tool, had been

Software mined the data of users

its guidelines concerning theused by the parole board in its

and deployed that data, alongside ancollection of biometric information.

determinations. Northpointe

algorithm, to recommend drug-related

BIPA does not apply to financial

wanted to prevent the disclosure

discussion groups to her son. Ultimate

institutions. Ultimately, the courtof AI trade secrets to one of the

Software moved to dismiss the claims

ruled that under the Gramm-

plaintiff’s expert witnesses. The

and claimed partial immunity under the

Leach-Bliley Act, the defendants

court ruled that the confidential

Communications Decency Act, whichwere a financial institution bymaterial in question was relevant

protects website operators from liability

virtue of lending functions they

to the plaintiff’s case and posed

for third-party content on their site. The

engaged in and therefore exempt

little risk of competitive injury.

Court ruled that Ultimate Software wasfrom BIPA. As such, the plaintiff’s

As such, the material was

immune and that its use of algorithms

case was dismissed.

ordered to be released under a

did not sufficiently amount to novel

supplemental protective order.

content creation.

9 The defendant was Tina M. Stanford, as Chairwoman of the New York State Board of Parole.

Table of Contents

Chapter 6 Preview

295Artificial IntelligenceIndex Report 2023

Artificial IntelligenceIndex Report 2023

CHAPTER 7:

Diversity

Table of Contents

Chapter 7 Preview

296Artificial IntelligenceIndex Report 2023

CHAPTER 7 PREVIEW:

Diversity
Overview

298

Chapter Highlights

299

7.1 AI Conferences
Women in Machine Learning (WiML)
NeurIPS Workshop

300

Narrative Highlight:
Disability Status of CS, CE,
and Information Students

311

New AI PhDs

312

CS, CE, and Information Faculty

313

300

Workshop Participants300

7.3 K–12 Education

316

Demographic Breakdown301

AP Computer Science: Gender316

AP Computer Science: Ethnicity

318

7.2 AI Postsecondary Education

305

CS Bachelor’s Graduates

305

CS Master’s Graduates

307

CS PhD Graduates

309

Table of Contents

Chapter 7 Preview

ACCESS THE PUBLIC DATA

297

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity

Overview
AI systems are increasingly deployed in the real world. However, there often exists
a disparity between the individuals who develop AI and those who use AI. North
American AI researchers and practitioners in both industry and academia are
predominantly white and male. This lack of diversity can lead to harms, among them
the reinforcement of existing societal inequalities and bias.
This chapter highlights data on diversity trends in AI, sourced primarily from academia.
It borrows information from organizations such as Women in Machine Learning(WiML), whose mission is to improve the state of diversity in AI, as well as theComputing Research Association (CRA), which tracks the state of diversity in North
American academic computer science. Finally, the chapter also makes use of Code.org
data on diversity trends in secondary computer science education in the United States.
Note that the data in this subsection is neither comprehensive nor conclusive. Publicly
available demographic data on trends in AI diversity is sparse. As a result, this chapter
does not cover other areas of diversity, such as sexual orientation. The AI Index hopesthat as AI becomes more ubiquitous, the amount of data on diversity in the field will
increase such that the topic can be covered more thoroughly in future reports.

Table of Contents

Chapter 7 Preview

298Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity

Chapter Highlights
North American bachelor’s,
master’s, and PhD-levelcomputer science students
are becoming more
ethnically diverse.
Although white students are still themost represented ethnicity among newresident bachelor’s, master’s, and PhD-levelcomputer science graduates, students fromother ethnic backgrounds (for example,
Asian, Hispanic, and Black or African
American) are becoming increasingly
more represented. For example, in 2011,71.9% of new resident CS bachelor’s
graduates were white. In 2021, that number
dropped to 46.7%.

Women make up an
increasingly greater shareof CS, CE, and information
faculty hires.
Since 2017, the proportion of new female
CS, CE, and information faculty hires has
increased from 24.9% to 30.2%. Still, mostCS, CE, and information faculty in North
American universities are male (75.9%).
As of 2021, only 0.1% of CS, CE, andinformation faculty identify as nonbinary.

Table of Contents

Chapter 7 Preview

New AI PhDs are still
overwhelmingly male.
In 2021, 78.7% of new AI PhDs weremale. Only 21.3% were female, a
3.2 percentage point increase from2011. There continues to be a genderimbalance in higher-level AI education.

American K–12 computerscience education has
become more diverse,
in terms of both genderand ethnicity.
The share of AP computer scienceexams taken by female students
increased from 16.8% in 2007 to 30.6%
in 2021. Year over year, the share of
Asian, Hispanic/Latino/Latina, andBlack/African American students
taking AP computer science has
likewise increased.

299

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.1 AI Conferences

7.1 AI Conferences
Women in Machine Learning(WiML) NeurIPS Workshop

collaboration and interaction among participants from

Women in Machine Learning (WiML), founded in

Workshop Participants

2006, is an organization dedicated to supporting and

Figure 7.1.1 shows the number of participants that

increasing the impact of women in machine learning.

have attended the WiML workshop since 2010. In theThis subsection of the AI Index report presents data

last decade, there has been a steady increase: 1,157

from the WiML annual technical workshop, hosted at

individuals participated in 2022, 13 times the number

NeurIPS. Since 2020, WiML has also been hosting thein 2010. However, from 2021 to 2022, the number of

Un-Workshop, which serves to advance research via

workshop participants decreased from 1,486 to 1,157.1

diverse backgrounds at the International Conferenceof Machine Learning (ICML).

Attendance at NeurIPS Women in Machine Learning Workshop, 2010–22
Source: Women in Machine Learning, 2022 | Chart: 2023 AI Index Report

1,400

Number of Attendees

1,200

1,157

1,000

800

600

400

200

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

2022
Figure 7.1.1

1 The recent decrease in WiML workshop attendance may be attributable to the overall recent decrease in NeurIPS attendance. This overall decrease may in turn be a result of
NeurIPS moving away from a purely virtual format.

Table of Contents

Chapter 7 Preview

300Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity

Demographic Breakdown

information aggregated. Among survey respondents,

Figure 7.1.2 breaks down the continent of residence

around 41.5% were from North America, followedof the 2022 workshop participants. The data in theby Europe (34.2%), Asia (17.1%), and Africa (3.4%). In

following figures comes from a survey completed

2022, there was greater representation from Europe,by participants who consented to having such

Asia, and South America.

7.1 AI Conferences

Continent of Residence of Participants at NeurIPS Women in Machine Learning Workshop, 2022
Source: Women in Machine Learning, 2022 | Chart: 2023 AI Index Report

North
America

41.50%

Europe34.20%

Asia

17.10%

Africa

3.40%

South
America

1.60%

Australia/
Oceania

1.40%

Antarctica 0.20%
0%

5%

10%

15%

20%
25%
% of Respondents

30%

35%

40%

45%
Figure 7.1.22

2 At the time of the survey, one of the respondents was temporarily residing in Antarctica.

Table of Contents

Chapter 7 Preview

301

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.1 AI Conferences

The majority of participants at the 2022 WiML workshop were female-identifying (37.0%), another 25.8% weremale-identifying, and 0.5% were nonbinary-identifying (Figure 7.1.3).

Gender Breakdown of Participants at NeurIPS Women in Machine Learning Workshop, 2022
Source: Women in Machine Learning, 2022 | Chart: 2023 AI Index Report

Female37.00%

Prefer Not to Say

36.30%

Male

Nonbinary

25.80%

0.50%

Gender Fluid 0.20%

Gender0.20%
Non-Conforming
0%

5%

10%

15%

20%
% of Respondents

25%

30%

35%

40%
Figure 7.1.3

Table of Contents

Chapter 7 Preview

302

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.1 AI Conferences

The most represented professional positions at the workshop were PhD students (49.4%), research scientists/
data scientists (20.8%), software engineers/data engineers (8.4%), and faculty (4.4%) (Figure 7.1.4).

Professional Positions of Participants at NeurIPS Women in Machine Learning Workshop, 2022
Source: Women in Machine Learning, 2022 | Chart: 2023 AI Index Report

PhD Student

49.40%

Research Scientist/
Data Scientist
Software Engineer/
Data Engineer

20.80%
8.40%

Faculty

4.40%

CEO/Director

3.50%

Others3.50%

Postdoc

3.50%

MSc Student

2.30%

Undergraduate
Student

2.00%

Recruiter

1.60%

Lecturer

1.40%

Recent Graduate 0.20%
0%

10%

20%

30%
% of Respondents

40%

50%
Figure 7.1.4

Table of Contents

Chapter 7 Preview

303

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.1 AI Conferences

The WiML workshop participants at NeurIPS submitted papers covering a wide range of subjects (Figure 7.1.5).
The most popular submission topics were applications (32.5%), algorithms (23.4%), and deep learning (14.8%).

Primary Subject Area of Submissions at NeurIPS Women in Machine Learning Workshop, 2022
Source: Women in Machine Learning, 2022 | Chart: 2023 AI Index Report

Applications

32.50%

Algorithms

23.40%

Deep Learning

14.80%

Social Aspects of
Machine Learning

7.70%

Reinforcement
Learning and Planning

7.20%

Neuroscience andCognitive Science

5.30%

Data, Challenges,
Implementations,
Software3.80%

Probabilistic Methods

3.30%

Optimization

1.00%

Theory

1.00%

0%

5%

10%

15%
20%
% of Respondents

25%

30%
Figure 7.1.5

Table of Contents

Chapter 7 Preview

304

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.2 AI Postsecondary Education

Another proxy for studying diversity in AI is looking at trends in postsecondary AI education. The following subsection borrows datafrom the Computing Research Association’s (CRA) annual Taulbee Survey.3

7.2 AI Postsecondary Education
CS Bachelor’s Graduates

last decade whereby an increasingly large number

The number of female CS bachelor’s graduatessurvey also included a nonbinary gender category: In

rose to 22.3% from 2020 to 2021 (Figure 7.2.1). This

2021, the number of nonbinary/other-identifying CS

increase mirrors a broader trend observed in thebachelor’s graduates was 0.04%.

of CS bachelor’s graduates were women. The CRA

Gender of New CS Bachelor’s Graduates (% of Total) in North America, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

90%

New CS Bachelor’s Graduates (% of Total)

80%

77.66%, Male

70%
60%
50%
40%
30%
22.30%, Female

20%
10%
0%

0.04%, Nonbinary/Other
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 7.2.1

3 The charts in this subsection look only at the ethnicity of domestic or native CS students and faculty. Although the CRA reports data on the proportion of nonresident aliens in each educational
level (i.e., Bachelor’s, Master’s, PhD, and faculty), data on the ethnicity of nonresident aliens is not included. For the proportion of nonresident aliens in each category, see footnotes.

Table of Contents

Chapter 7 Preview

305

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.2 AI Postsecondary Education

Figure 7.2.2 breaks down the ethnicity of new CS bachelor’s graduates in North America: The top ethnicity
was white (46.7%), followed by Asian (34.0%) and Hispanic (10.9%). In the last decade, the proportion of
new CS bachelor’s graduates who were Asian, Hispanic, or multiracial (not Hispanic) steadily increased.4

Ethnicity of New Resident CS Bachelor’s Graduates (% of Total) in North America, 2011–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

New CS Bachelor’s Graduates (% of Total)

70%

60%

50%
46.69%, White40%
33.99%, Asian
30%

20%
10.91%, Hispanic (Any Race)
4.10%, Multiracial (Not Hispanic)
3.85%, Black or African-American
0.24%, Native Hawaiian or Paci c Islander
0.22%, American Indian or Alaska Native

10%

0%
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 7.2.2

4 In 2021, 16.3% of new CS bachelor graduates were nonresident aliens.

Table of Contents

Chapter 7 Preview

306Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.2 AI Postsecondary Education

CS Master’s Graduates
Figure 7.2.3 shows the gender of CS master’smoving to 27.8% in 2021 from 24.6% in 2011. In

graduates. The proportion of female CS master’s

2021, 0.9% of CS master’s graduates identified as

graduates has not substantially increased over time,

nonbinary/other.

Gender of New CS Master’s Graduates (% of Total) in North America, 2011–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

80%
71.27%, Male

New CS Master’s Graduates (% of Total)

70%

60%

50%

40%

30%

27.83%, Female

20%

10%
0.90%, Nonbinary/Other

0%
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 7.2.3

Table of Contents

Chapter 7 Preview

307

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.2 AI Postsecondary Education

Of domestic students, the most represented ethnicities are white (50.3%), followed by Asian (34.8%), andHispanic (7.3%) (Figure 7.2.4). As with CS bachelor’s graduates, in the last decade white students haverepresented an increasingly smaller proportion of new CS master’s graduates.5

Ethnicity of New Resident CS Master’s Graduates (% of Total) in North America, 2011–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

70%

New CS Master’s Graduates (% of Total)

60%

50.28%, White

50%

40%
34.83%, Asian
30%

20%

7.25%, Hispanic (Any Race)
3.82%, Black or African-American
3.45%, Multiracial (Not Hispanic)
0.25%, American Indian or Alaska Native
0.12%, Native Hawaiian or Paci c Islander

10%

0%
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 7.2.4

5 In 2021, 65.2% of new CS master’s graduates were nonresident aliens.

Table of Contents

Chapter 7 Preview

308Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.2 AI Postsecondary Education

CS PhD Graduates
In 2021, the number of new female CS PhD

continue to be male. There remains a large gap

graduates rose to 23.3% from 19.9% (Figure 7.2.5).

between new male and female CS PhDs.

Despite this rise, most new CS PhD graduates

Gender of New CS PhD Graduates (% of Total) in North America, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

80%
76.58%, Male

New CS PhD Graduates (% of Total)

70%
60%
50%
40%
30%
23.30%, Female

20%
10%

0.12%, Nonbinary/Other

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 7.2.5

Table of Contents

Chapter 7 Preview

309

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.2 AI Postsecondary Education

Between 2011 and 2021, the number of new white resident CS PhD graduates declined by 9.4 percentage
points. Asians are the next most represented group (29%), followed by Hispanics (5.1%) and Black or African
Americans (4%) (Figure, 7.2.6).6

Ethnicity of New Resident CS PhD Graduates (% of Total) in North America, 2011–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

70%

New CS PhD Graduates (% of Total)

60%

58.64%, White

50%

40%

30%

29.00%, Asian20%
5.12%, Hispanic (Any Race)
4.05%, Black or African-American
2.13%, Multiracial (Not Hispanic)
0.64%, American Indian or Alaska Native
0.43%, Native Hawaiian or Paci c Islander

10%

0%
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 7.2.6

6 In 2021, 68.6% of new CS PhD graduates were nonresident aliens.

Table of Contents

Chapter 7 Preview

310Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.2 AI Postsecondary Education

Narrative Highlight:

Disability Status of CS, CE, and Information Students
The 2021 edition of the CRA Taulbee Survey was thedisability accommodations in the last year.

first to gather information about the prevalence ofThe number of such students was relatively small.

CS, CE, and information students with disabilities.

Only 4.0% of bachelor’s, 1.0% of PhD students,

The CRA asked departments to identify the number

and 0.8% of master’s students reported needingof students at each degree level who received

accommodations (Figure 7.2.7).

CS, CE, and Information Students (% of Total) With Disability Accomodations in North America, 2021

Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

4.10%
CS, CE, and Information Students (% of Total)

4%

3%

2%

1.00%

1%

0%

0.80%

Bachelor’s

PhDs

Master’s

Figure 7.2.7

Table of Contents

Chapter 7 Preview

311

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.2 AI Postsecondary Education

New AI PhDs

were female. While the number of female AI PhDs

Figure 7.2.8 looks at demographic trends for new AI

marginally increased from 2020 to 2021, we find no

PhD graduates who focus on artificial intelligence.

meaningful trends in the last decade relating to the

In 2021, 78.7% of new AI PhDs were male and 21.3%

gender of new AI PhDs.

Gender of New AI PhD Graduates (% of Total) in North America, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

80%

78.70%, Male

New AI PhD Graduates (% of Total)

70%

60%

50%

40%

30%
21.30%, Female

20%

10%

0%

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 7.2.8

Table of Contents

Chapter 7 Preview

312Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.2 AI Postsecondary Education

CS, CE, andInformation Faculty

predominantly male (75.9%) (Figure 7.2.9). Women
make up 23.9% of CS, CE, and information faculty,

Data on the ethnicity and gender of CS, CE, andand nonbinary individuals make up 0.1%. The share

information faculty helps to paint a picture ofof female CS, CE, and information faculty has slowly

diversity trends in academic AI and CS. As of 2021,

increased; since 2011, the number of female faculty

most CS, CE, and information faculty members are

members has risen 5 percentage points.

Gender of CS, CE, and Information Faculty (% of Total) in North America, 2011–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report80%
75.94%, Male
CS, CE, and Information Faculty (% of Total)

70%
60%
50%
40%
30%
23.94%, Female
20%
10%
0.12%, Nonbinary/Other

0%
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 7.2.9

Table of Contents

Chapter 7 Preview

313

Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.2 AI Postsecondary Education

Although most new CS, CE, and information faculty hires in North American universities are still male, theproportion of women among faculty hires reached 30.2% in 2021, up about 9 percentage points from 2015
(Figure 7.2.10).

Gender of New CS, CE, and Information Faculty Hires (% of Total) in North America, 2011–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

New CS, CE, and Information Faculty Hires (% of Total)

80%

70%

69.26%, Male

60%

50%

40%

30.17%, Female

30%

20%

10%

0%

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

0.57%, Nonbinary/Other
2021
Figure 7.2.10

Table of Contents

Chapter 7 Preview

314Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.2 AI Postsecondary Education

The majority of resident CS, CE, and information faculty are white as of 2021 (58.1%), followed by Asian (29.7%)(Figure 7.2.11). However, the gap between white CS, CE, and information faculty and faculty of the next nearest
ethnicity is slowly narrowing: In 2011, the gap stood at 46.1%, whereas in 2021 it dropped to 28.4%.7

Ethnicity of Resident CS, CE, and Information Faculty (% of Total) in North America, 2010–21
Source: CRA Taulbee Survey, 2022 | Chart: 2023 AI Index Report

70%

CS, CE, and Information Faculty (% of Total)

60%

58.08%, White

50%

40%

30%

29.70%, Asian20%
5.82%, Unknown
2.80%, Hispanic (Any Race)
2.54%, Black or African-American
0.67%, Multiracial (Not Hispanic)
0.25%, American Indian or Alaska Native
0.13%, Native Hawaiian or Paci c Islander

10%

0%
2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 7.2.11

7 In 2021, 6.7% of CS, CE, and information faculty in North America were nonresident aliens.

Table of Contents

Chapter 7 Preview

315Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.3 K–12 Education

How do trends in AI diversity measure at the K–12 level, prior to students entering university? This subsection borrows data from
Code.org, an American nonprofit that aims to promote K–12 computer science education in the United States.

7.3 K–12 Education
AP Computer Science: Gendernor female (Figure 7.3.1). It is still the case that male

In 2021, 69.2% of AP computer science exams werestudents take more AP computer science exams

taken by male students, 30.6% by female students,

than any other gender, but the proportion of female

and 0.3% by students who identified as neither malestudents has almost doubled in the last decade.

AP Computer Science Exams Taken (% of Total) by Gender, 2007–21
Source: Code.org, 2022 | Chart: 2023 AI Index Report

AP Computer Science Exams Taken (% of Total)

80%
70%

69.16%, Male

60%
50%
40%
30.58%, Female

30%
20%
10%

0.26%, Other

0%
2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 7.3.1

Table of Contents

Chapter 7 Preview

316Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.3 K–12 Education

On a percent basis, the states with the largest(35%) (Figure 7.3.2). Other states with notable CS andnumber of female AP computer science test-

AI activity include California, Texas, and Washington,takers were Alabama (36%) and Washington, D.C.

with rates of women taking AP computer science(36%), followed by Nevada (35%), Louisiana (35%),

tests at rates hovering around 30 percent.

Tennessee (35%), Maryland (35%), and New YorkAP Computer Science Exams Taken by Female Students (% of Total),
2021
Source: Code.org, 2022 | Chart: 2023 AI Index Report

ME
27%

AK
20%

CA
31%

VT
23%

NH
24%

MA
30%

NY
35%

CT
30%

RI
31%

WA
32%

MT
21%

ND
16%

SD
15%

MN
23%

WI
23%

MI
30%

OR
21%

ID
26%

WY
31%

NE
25%

IA
24%

IL
32%

IN
23%

OH
27%

PA
27%

NJ
31%

NV
35%

UT
23%

CO
26%

KS
15%

MO
22%

KY
31%

WV
30%

DC
36%

MD
35%

DE
22%

AZ
27%

NM
29%

OK
25%

AR
29%

TN
35%

VA
28%

NC
31%

TX
30%

LA
35%

MS
33%

AL
36%

GA
29%

SC
34%

HI
30%

FL
31%
Figure 7.3.2

Table of Contents

Chapter 7 Preview

317Artificial IntelligenceIndex Report 2023

Chapter 7: Diversity
7.3 K–12 Education

AP Computer Science:
Ethnicity

most postsecondary computer science fields, theCode.org collects data that speaks to trends in the

more ethnically diverse over time. White students

ethnicity of AP computer science test-takers. White

are still the greatest test-taking group; however, overstudents took the greatest proportion of the exams intime, more Asian, Hispanic/Latino/Latina and Black/2021 (42.7%), followed by Asian (28.8%) and Hispanic/

African American students have taken AP computer

Latino/Latina students (16.5%) (Figure 7.3.3). As with

science exams.

pool of AP computer science test-takers is becoming

AP Computer Science Exams Taken (% of Total Responding Students) by Race/Ethnicity, 2007–21

AP Computer Science Exams Taken (% of Total Responding Students)

Source: Code.org, 2022 | Chart: 2023 AI Index Report

60%

50%
42.74%, White
40%

30%

28.78%, Asian20%
16.48%, Hispanic/Latino/Latina
10%

6.32%, Black/African American
4.92%, Two or More Races
0.62%, Native American/Alaskan
0.15%, Native Hawaiian/Paci c Islander
0.00%, Other

0%
2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 7.3.3

Table of Contents

Chapter 7 Preview

318Artificial IntelligenceIndex Report 2023

Artificial IntelligenceIndex Report 2023

CHAPTER 8:

Public Opinion

Table of Contents

Chapter 8 Preview

319

Artificial IntelligenceIndex Report 2023

CHAPTER 8 PREVIEW:

Public Opinion
Overview

321

Chapter Highlights

322

8.1 Survey Data

323

Global Insights

323

AI Products and Services323

AI: Harm or Help?

327

United States

Table of Contents

329

Chapter 8 Preview

Narrative Highlight: How Does the
Natural Language Processing (NLP)
Research Community Feel About AI?

334

8.2 Social Media Data

340

Dominant Models

340

ACCESS THE PUBLIC DATA

320

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion

Overview
AI has the potential to have a transformative impact on society. As such it has become
increasingly important to monitor public attitudes toward AI. Better understanding
trends in public opinion is essential in informing decisions pertaining to AI’s
development, regulation, and use.
This chapter examines public opinion through global, national, demographic, and ethnic
lenses. Moreover, we explore the opinions of AI researchers, and conclude with a lookat the social media discussion that surrounded AI in 2022. We draw on data from two
global surveys, one organized by IPSOS, and another by Lloyd’s Register Foundation
and Gallup, along with a U.S-specific survey conducted by PEW Research.
It is worth noting that there is a paucity of longitudinal survey data related to AI asking
the same questions of the same groups of people over extended periods of time. As AI
becomes more and more ubiquitous, broader efforts at understanding AI public opinion
will become increasingly important.

Table of Contents

Chapter 8 Preview

321Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion

Chapter Highlights
Chinese citizens are among
those who feel the mostpositively about AI products
and services. Americans …
not so much.
In a 2022 IPSOS survey, 78% of Chinese
respondents (the highest proportion ofsurveyed countries) agreed with the statementthat products and services using AI havemore benefits than drawbacks. After Chinese
respondents, those from Saudi Arabia (76%)
and India (71%) felt the most positive about
AI products. Only 35% of sampled Americans(among the lowest of surveyed countries) agreed
that products and services using AI had more
benefits than drawbacks.

People acrossthe world andespecially
America remain
unconvinced by
self-driving cars.
In a global survey, only
27% of respondents
reported feeling safe
in a self-driving car.
Similarly, Pew Researchsuggests that only 26%
of Americans feel thatdriverless passenger
vehicles are a good idea
for society.
Table of Contents

Chapter 8 Preview

Men tend to feel morepositively about AI products
and services than women.
Men are also more likely than
women to believe that AI will
mostly help rather than harm.
According to the 2022 IPSOS survey, men are more
likely than women to report that AI products andservices make their lives easier, trust companies
that use AI, and feel that AI products and services
have more benefits than drawbacks. A 2021 survey
by Gallup and Lloyd’s Register Foundation likewise
revealed that men are more likely than women toagree with the statement that AI will mostly help
rather than harm their country in the next 20 years.

Different causes
for excitement
and concern.

NLP researchers …
have some strong
opinions as well.

Among a sample of surveyed
Americans, those who reportfeeling excited about AI are
most excited about the potential
to make life and society better(31%) and to save time andmake things more efficient(13%). Those who report feelingmore concerned worry about
the loss of human jobs (19%);surveillance, hacking, and digital
privacy (16%); and the lack ofhuman connection (12%).

According to a survey
widely distributed to NLP
researchers, 77% either
agreed or weakly agreed
that private AI firms have toomuch influence, 41% said that
NLP should be regulated, and
73% felt that AI could soon
lead to revolutionary societal
change. These were someof the many strong opinions
held by the NLP researchcommunity.

322

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

8.1 Survey Data
Global Insights

Figure 8.1.1 highlights global opinions (aggregated

How do opinions of AI vary across the globe? The

variety of questions relating to AI products andfirst subsection of this chapter provides a response bylooking at survey data from IPSOS and Pew Research,
as well as one poll that was a collaboration of Gallupand Lloyd’s Register Foundation. The surveys suggestthat public perceptions concerning AI differ acrosscountries and by demographic groups.

results across the entire survey subsample) for aservices. It shows the percentage of respondents
who agree with a particular question. The majority ofthe survey sample, 60%, believe that AI products andservices will profoundly change their daily life in thenear future—and make their life easier. A very slightmajority, 52%, feel that products and services that
use AI have more benefits than drawbacks. Only 40%

AI Products and Services
In late 2021, IPSOS ran a survey on global attitudesof respondents report that AI products and services

toward AI products and services. The survey

make them feel nervous.

consisted of interviews with 19,504 adults ages
16–74 in 28 different countries.1

Global Opinions on Products and Services Using AI (% of Total), 2022
Source: IPSOS, 2022 | Chart: 2023 AI Index Report

I have a good understanding of what
artificial intelligence is

64%

Products and services using artificial intelligence
will profoundly change my daily lifein the next 3–5 years

60%

Products and services using artificial intelligence
make my life easier60%

Products and services using artificial intelligencehave more benefits than drawbacks

52%

I know which types of products and services
use artificial intelligence50%

I trust companies that use artificial intelligenceas much as I trust other companies

50%

Products and services using artificial intelligencehave profoundly changed my daily lifein the past 3–5 years

49%

Products and services using artificial intelligence
make me nervous
0%

39%
10%

20%

30%
40%
% of Respondents That “Agree”

50%

60%
Figure 8.1.1

1 See Appendix for more details about the survey methodology.

Table of Contents

Chapter 8 Preview

323

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

Opinions vary widely across countries as to theservices using AI have more benefits than drawbacks

relative advantages and disadvantages of AI.

(Figure 8.1.2). However, only 35% of American

The IPSOS survey suggests that 78% of Chinese

respondents share that sentiment. Among the 28

respondents, 76% of Saudi Arabian respondents, andsurveyed countries, France and Canada held the most

71% of Indian respondents feel that products andnegative views.

‘Products and services using AI have more bene ts than drawbacks,’ by Country (% of Total), 2022
Source: IPSOS, 2022 | Chart: 2023 AI Index Report

China

78%

Saudi Arabia

76%

India

71%

Peru

70%

Mexico65%

Malaysia

65%

Colombia

64%

Chile

63%

South Korea

62%

Turkey60%

Brazil57%

South Africa

57%

Argentina

55%

Spain

53%

Russia

53%

Italy

50%

Hungary49%

Poland

48%

Japan

42%

Sweden40%

Belgium

38%

Great Britain

38%

Australia

37%

Germany

37%

United States

35%

Netherlands

33%

Canada32%

France0%

Table of Contents

31%

10%

20%

Chapter 8 Preview

30%

40%
50%
% of Respondents That “Agree”

60%

70%

80%

Figure 8.1.2

324

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

Figure 8.1.3 breaks down answers to all of IPSOS’

companies that use AI as much as other companies,

AI products and services questions by country.

and only 30% say that AI products and services using AI

Generally, sentiment relating to AI products

make them nervous. Conversely, American respondents

and services seems to be strongly correlated

are among the most negative when it comes to AI. Only

within specific countries. For example, Chinese

41% claim that AI products and services make their lives

respondents seem to feel among the most positive

easier, 35% report trusting AI companies as much as

about AI products and services: 87% of Chinese

other companies, and 52% report that AI products andrespondents claim that AI products and services

services make them feel nervous.

make their lives easier, 76% report trusting

Opinions About AI by Country (% Agreeing With Statement), 2022
Source: IPSOS, 2022 | Chart: 2023 AI Index Report

I have a good understanding of what 64% 59% 60% 69% 59% 76% 67% 71% 50% 50% 57% 67% 72% 42% 41% 61% 74% 65% 76% 66% 75% 73% 78% 72% 62% 60% 68% 63%
artificial intelligence is
Products and services using artificial
intelligence will profoundly change 60% 50% 52% 61% 44% 67% 80% 65% 45% 44% 46% 55% 74% 53% 53% 71% 65% 53% 71% 56% 60% 80% 72% 76% 56% 50% 73% 46%
my daily life in the next 3–5 years
Products and services using artificial
59% 46% 49% 65% 44% 70% 87% 71% 39% 45% 45% 50% 72% 54% 52% 71% 73% 47% 74% 58% 64% 80% 67% 74% 59% 46% 71% 41%
intelligence make my life easier
Products and services using artificial
intelligence have more benefits than 55% 37% 38% 57% 32% 63% 78% 64% 31% 37% 38% 49% 71% 50% 42% 65% 65% 33% 70% 48% 53% 76% 57% 62% 53% 40% 60% 35%
drawbacks
I know which types of products and
47% 38% 37% 58% 36% 59% 76% 62% 34% 37% 37% 38% 69% 45% 32% 61% 62% 41% 63% 52% 57% 69% 57% 60% 46% 37% 60% 39%
services use artificial intelligence
I trust companies that use artificial
intelligence as much as I trust other 55% 36% 40% 50% 34% 56% 76% 57% 34% 42% 35% 48% 68% 48% 39% 61% 60% 38% 60% 51% 52% 73% 56% 46% 50% 39% 63% 35%
companies
Products and services using artificial
intelligence have profoundly changed 53% 37% 37% 51% 32% 58% 73% 58% 32% 31% 33% 38% 67% 41% 30% 65% 62% 40% 65% 45% 50% 72% 56% 62% 49% 30% 60% 36%
my daily life in the past 3–5 years

United States

TurkeySwedenSpain

South Korea

South Africa

Saudia Arabia

RussiaPolandPeru

Netherlands

MexicoJapan

Malaysia

Italy

India

Hungary

Great Britain

FranceGermany

China

Colombia

Chile

BrazilCanadaBelgium

Australia

Argentina

Products and services using artificial
33% 51% 42% 35% 49% 36% 30% 39% 32% 37% 50% 31% 53% 26% 20% 48% 38% 36% 35% 30% 28% 51% 52% 32% 48% 37% 48% 52%
intelligence make me nervous

Figure 8.1.3

Table of Contents

Chapter 8 Preview

325

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

Figure 8.1.4 breaks down opinions in all countries

feeling that AI products and services make their lives

across demographic groups such as gender, age,easier, they are also less likely than the 35-to-49 age

household income, and employment status. IPSOS

category to believe that AI products and services haveresults suggest that men feel more positively about

more benefits than drawbacks. Finally, households with

AI products and services than women—for example,

higher incomes are more positive, compared to thosecompared to women, men are more likely to reportwith lower incomes, about AI products and services

feeling that AI products and services make theirmaking life easier and having more benefits thanlives easier. Age-specific opinions vary. For instance,

drawbacks.

while individuals under 35 are most likely to report

Opinions About AI by Demographic Group (% Agreeing With Statement), 2022
69%

60%

66%

65%

61%

57%

63%

71%

56%

64%

71%

73%

74%

67%

59%

Products and services using artificial
intelligence will profoundly change
my daily life in the next 3–5 years

63%

57%

63%

61%

55%

56%

58%

67%

53%

58%

68%

70%

72%

64%

54%

Products and services using artificial
intelligence make my life easier

62%

58%

64%

62%

54%

56%

58%

66%

53%

58%

67%

67%

70%

63%

55%

Products and services using artificial
intelligence have more benefits thandrawbacks

55%

49%

47%

53%

46%

50%

51%

57%

45%

50%

59%

63%

64%

55%

47%

I know which types of products andservices use artificial intelligence

55%

46%

54%

51%

45%

46%

50%

57%

44%

48%

58%

63%

65%

54%

44%

I trust companies that use artificial
intelligence as much as I trust othercompanies

53%

47%

54%

51%

44%

47%

48%

57%

45%

48%

56%

61%

62%

53%

45%

Products and services using artificial
intelligence have profoundly changed
my daily life in the past 3–5 years

51%

46%

54%

50%

41%

46%

47%

54%

43%

46%

55%

61%

62%

52%

43%

Products and services using artificial
intelligence make me nervous

38%

41%

40%

40%

38%

41%

41%

38%

41%

37%

40%

48%

46%

40%

38%

FemaleUnder 35

35 to 49

50 to 74

Low

MediumHigh

Low

MediumHigh

Business Owner

Sr. Exec./
Decision Maker

Employed

GenderAgeHousehold Income

Education

Non-Employed

I have a good understanding of what
artificial intelligence is

Male

Source: IPSOS, 2022 | Chart: 2023 AI Index Report

Employment Status

Figure 8.1.4

Table of Contents

Chapter 8 Preview

326Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

AI: Harm or Help?
In 2021, Lloyd’s Register Foundation, an independent

A greater proportion of respondents believed that

global charity, collaborated with Gallup to poll

AI will mostly help (39%) compared to a smaller

125,911 people across 121 countries about their

proportion who believed that it would mostly harm

perceptions of artificial intelligence and other digital(28%). Mirroring the disparity in responses acrosstrends. Figure 8.1.5 shows the responses to thegender evident in the IPSOS survey, men in thesurvey question, “Do you think artificial intelligence

Lloyd’s-Gallup poll were more likely than women towill mostly help or mostly harm people in this country

report believing that AI will mostly help people in thein the next 20 years?”

next 20 years.

Views on Whether AI Will ‘Mostly Help’ or ‘Mostly Harm’ People in the Next 20 Years Overall and by
Gender (% of Total), 2021
Source: Lloyd’s Register Foundation and Gallup, 2022 | Chart: 2023 AI Index Report% World% Women

% Men

42%
40%

39%

% of Respondents

35%

30%

28%

29%
27%
24%
22%
20%

20%

10%

8%

2%
0%

Mostly help

Mostly harm

2%Neither

9%

8%

2%
Don’t have an opinion

Don’t know/refused
Figure 8.1.5

Table of Contents

Chapter 8 Preview

327Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

Eastern Asia, Northern/Western Europe, andfor every 1 response of “mostly harm” there were 4.4

Southern Europe are the regions of the world where

responses suggesting that AI will “mostly help.” Thepeople are most likely to report believing that AI willregions whose populations are most pessimistic about

mostly help versus mostly harm (Figure 8.1.6). Morethe potential benefits of AI include Eastern Africa,

specifically, among the Eastern Asian survey sample,

Northern Africa, and Southern Africa.

Views on Whether AI Will ‘Mostly Help’ or ‘Mostly Harm’ People in the Next 20 Years by Region:
Ratio of ‘Mostly Help’/‘Mostly Harm’, 2021
Source: Lloyd’s Register Foundation and Gallup, 2022 | Chart: 2023 AI Index Report

Eastern Asia

4.40

Northern/Western Europe

1.80

Australia and New Zealand

1.70

Eastern Europe

1.40Southern Europe

1.30

Central Asia

1.20

Southeastern Asia

1.20

Northern America

1.00

Middle East

1.00

Southern Asia

0.90

Latin America and Caribbean

0.90

Central/Western Africa

0.80

Northern Africa

0.70Southern Africa

0.60

Eastern Africa

0.40

0

1

The Lloyd’s Register survey also polled
respondents about their perceptions of2
3
Ratio of “Mostly Help”/“Mostly Harm”

4
Figure 8.1.6

Perceptions of the Safety of Self-Driving Cars(% of Total), 2021

Source: Lloyd’s Register Foundation and Gallup, 2022 | Chart: 2023 AI Index Reportcertain AI technologies, such as self-drivingcars. The majority of survey respondents

65%, Would not feel safe

reported not feeling safe in a self-driving car(65%), compared to only 27% who reported
feeling safe (Figure 8.1.7).

27%, Would feel safe
8%, Don’t know/refused
Figure 8.1.7

Table of Contents

Chapter 8 Preview

328Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

United States

Americans’ Feelings Toward Increased Use of AI
Programs in Daily Life (% of Total), 2022
Source: Pew Research, 2022 | Chart: 2023 AI Index Report

In 2022, Pew Research released one of the most

45%, Equally concernedand excitedcomprehensive surveys to date about Americans’views on AI. The survey interviewed 10,260 panelists
from a wide range of demographic groups about theirbroad AI-related opinions, as well as their perspectives
on specific AI use cases.2<1%, No answer

45% of Americans report feeling equally concernedand excited about the use of AI programs in daily life,
while 37% report feeling more concerned than excited

37%, More concerned
than excited

18%, More excitedthan concerned(Figure 8.1.8). Only 18% of Americans report feelingmore excited than concerned about AI technology.

Figure 8.1.8

Which AI applications are Americans most excitedabout? A large proportion report feeling very or

problems (40%) (Figure 8.1.9). Americans are very orsomewhat excited about AI being used to perform

somewhat concerned about AI being used to makehousehold chores (57%), to perform repetitive

important life decisions for people (74%) and to know

workplace tasks (46%), and to diagnose medical

people’s thoughts and behaviors (75%).

Americans’ Feelings on Potential AI Applications (% of Total), 2022
Source: Pew Research, 2022 | Chart: 2023 AI Index Report

Very/somewhat excited

Perform household chores

Equally excited and concerned

Very/somewhat concerned

57%

Perform repetitive workplace
tasks

24%

46%

Diagnose medical problem

27%

40%

Handle customer service calls

27%

35%

26%

47%

9%

16%

74%

Know people’s thoughts
and behaviors

9%

16%

75%

20%

2 See Appendix for more details about the survey methodology.
3 The numbers in Figure 8.1.9 may not sum up to 100% due to rounding.

Table of Contents

Chapter 8 Preview

26%

24%

Make important life decisions
for people

0%

19%

40%

% of Respondents

60%

80%

100%
Figure 8.1.93

329

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

There are two specific AI use cases that Americans

that police using facial recognition technology is a

are more likely to report feeling are good ideas

good idea for society compared to 27% who believe it

for society rather than bad: police use of facial

is a bad idea. However, Americans are not as excitedrecognition technology, and social media companies

about driverless passenger vehicles: More feel thatusing AI to find false information on their sites (Figure

driverless passenger vehicles are a bad idea for

8.1.10). More specifically, 46% of Americans believe

society than a good idea.

Americans’ Perceptions of Specific AI Use Cases (% of Total), 2022
Source: Pew Research, 2022 | Chart: 2023 AI Index Report

Bad idea for society

Facial recognition technology
by police

Good idea for society

27%

Not sure

46%

27%
Figure 8.1.8

Computer programs by social
media companies to nd false
information on their sites

Driverless passenger vehicles

0%

31%

38%

44%

20%

30%

26%

40%

% of Respondents

60%

29%

80%

100%
Figure 8.1.104

4 The numbers in Figure 8.1.10 may not sum up to 100% due to rounding.

Table of Contents

Chapter 8 Preview

330Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

Of the sample of Americans who reported being morehacking, and digital privacy (16%); and lack of humanconcerned than excited about AI, Figure 8.1.11 outlines

connection (12%). Americans reported being lessthe main reasons for their concern. The primaryconcerned about the potential loss of freedom andreasons include loss of human jobs (19%); surveillance,

issues relating to lack of oversight and regulation.

Main Reason Americans Are Concerned About AI (% of Total), 2022
Source: Pew Research, 2022 | Chart: 2023 AI Index Report

Loss of human jobs

19%

Surveillance, hacking, digital privacy

16%

Lack of human connection/qualities

12%

AI will get too powerful/outsmarting
people8%

People misusing AI

8%People becoming too reliant on
AI/techFigure 8.1.8

7%

AI fails, makes mistakes

6%

Concerns about government/techcompanies using AI

3%Don’t trust AI or people wielding it

3%

Unforeseen consequences/effects

2%

Loss of freedom

2%

Human bias coded into AI

2%

Lack of oversight and regulation

2%

Other
0%

7%

5%

10%
% of Respondents

15%

20%
Figure 8.1.11

Table of Contents

Chapter 8 Preview

331Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

The two leading reasons that Americans reportsociety better. A significant group also reported

being excited about AI relate to its potential tofeeling excited about the potential of AI to save time

make life better and to save time (Figure 8.1.12).

and increase efficiency (13%), as well as to handle

Of the respondents, 31% believe AI makes life andmundane, tedious tasks (7%).

Main Reason Americans Are Excited About AI (% of Total), 2022
Source: Pew Research, 2022 | Chart: 2023 AI Index Report

Makes life, society better

31%

Saves time, more e�cient

13%

Inevitable progress, is the future10%

Handles mundane, tedious tasks

7%

Helps with work/labor6%

AI is interesting, exciting

6%

Helps humans with di�cult/dangerous tasks

6%

More accurate than humans

4%

Helps those who are elderly/
have a disability

4%

Personal anecdotes

2%

Other people’s fears based on
sci-�, not reality

2%

Other

Figure 8.1.8

7%

0%

5%

10%

15%
% of Respondents

20%

25%

30%
Figure 8.1.12

Table of Contents

Chapter 8 Preview

332

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

The Pew Research survey also asked participantsin the degree to which people felt that AI systems

which group of people had their experiences andpositively considered the experiences and views ofviews taken into consideration in the design of AI

men over women. Similarly, respondents felt that thesystems. Respondents felt AI systems most reflected

experiences and views of Asian, Black, and Hispanic

the experiences and views of men and white adults

adults, compared to those held by white adults, were

(Figure 8.1.13). There was a 15 percentage point gap

not as positively considered.

People Whose Experiences and Views Are Considered in the Design of AI Systems (% of Total), 2022
Source: Pew Research, 2022 | Chart: 2023 AI Index Report

Net not wellMen

Net wellNot sure

12%

51%

37%

Figure 8.1.8

Women

White adults

25%

36%

13%

Asian adults

38%

48%

23%

39%

33%

43%

Black adults

33%

24%

42%

Hispanic adults

33%

23%

43%

0%

20%

40%

% of Respondents

60%

80%

100%
Figure 8.1.135

5 The numbers in Figure 8.1.13 may not sum up to 100% due to rounding.

Table of Contents

Chapter 8 Preview

333Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

Narrative Highlight:

How Does the Natural Language Processing (NLP)
Research Community Feel About AI?
From May to June 2022, a group of American

In general, the NLP research community

researchers conducted a survey of the NLP researchstrongly feels that private firms have too muchcommunity on a diverse set of issues, including theinfluence (77%) and that industry will produce

state of the NLP field, artificial general intelligencethe most widely cited research (86%) (Figure(AGI), and ethics, among others. According to the8.1.14). Curiously, 67% either agreed or weakly

authors, a total of 480 individuals completed theagreed with the statement that most of NLP issurvey, 68% of whom had authored at least two

dubious science. A small proportion, 30%, think

Association for Computational Linguistics (ACL)an “NLP winter”—a period when the field faces

publications between 2019 and 2022. The survey

a significant slowdown or stagnation in research

represents one of the most complete pictures of theand development—is coming in the next decade.

6

attitudes AI researchers have toward AI research.
State of the Field According to the NLP Community, 2022
Source: Michael et al., 2022 | Chart: 2023 AI Index Report

Private �rms have toomuch in�uence

77%

Industry will produce
the most widely citedresearch

86%

NLP winteris coming (10 years)

30%

NLP winteris coming (30 years)

62%

Most of NLP is
dubious science67%

Author anonymity
is worth it
0%

63%
10%

20%

30%
40%
50%
60%
% of Respondents That “Agree” or “Weakly Agree”

70%

80%

90%

Figure 8.1.14
6 More detailed information about the survey methodology and sample group can be found in the following paper.

Table of Contents

Chapter 8 Preview

334

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

Narrative Highlight:

How Does the Natural Language Processing (NLP)
Research Community Feel About AI? (cont’d)
A small majority of NLP researchers believe that specific types of AI systems can actually understandlanguage: 51% agreed with the statement that language models (LMs) understand language, with evenmore (67%) agreeing that multimodal models understand language (Figure 8.1.15).
Language Understanding According to the NLP Community, 2022
Source: Michael et al., 2022 | Chart: 2023 AI Index ReportLMs understandlanguage

51%

Multimodal models
understand language

67%

Text-only evaluation canmeasure language
understanding

0%

36%

10%

20%

30%
40%
50%
% of Respondents That “Agree” or “Weakly Agree”

60%

Figure 8.1.15

Table of Contents

Chapter 8 Preview

335Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

Narrative Highlight:

How Does the Natural Language Processing (NLP)
Research Community Feel About AI? (cont’d)
NLP researchers also seem to believe that NLP’s

with 48% of respondents feeling it is unethical. Sixty

past net impact has been positive (89%) and that its

percent of researchers feel that the carbon footprint

future impact will continue to be good (87%) (Figureof AI is a major concern; however, only 41% feel that

8.1.16). The community is divided on the issue of

NLP should be regulated.

using AI to predict psychological characteristics,

Ethics According to the NLP Community, 2022
Source: Michael et al., 2022 | Chart: 2023 AI Index Report

NLP’s past net impact
is good

89%

NLP’s future net impact
is good87%

It is unethical to build
easily misusable systems

59%

Ethical and scienti�cconsiderations
can con�ict

74%

Ethical concerns mostly
reduce to data quality
and model accuracy25%

It is unethical to predict
psychologicalcharacteristics

48%

Carbon footprint isa major concern

60%

NLP should be regulated
0%

41%
10%

20%

30%
40%
50%
60%
% of Respondents That “Agree” or “Weakly Agree”

70%

80%

90%

Figure 8.1.16

Table of Contents

Chapter 8 Preview

336Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

Narrative Highlight:

How Does the Natural Language Processing (NLP)
Research Community Feel About AI? (cont’d)
Although a large majority of researchers feel that AI could soon lead to revolutionary societal change(73%), only 36% feel that AI decisions could cause nuclear-level catastrophe (Figure 8.1.17). A pluralityof researchers, 57%, held that recent research progress was leading the AI community toward Artificial
General Intelligence (AGI).

Arti cial General Intelligence (AGI) and Major Risks According to the NLP Community, 2022

Source: Michael et al., 2022 | Chart: 2023 AI Index Report

AGI is an importantconcern

58%

Recent progress ismoving us toward
AGI

57%

AI could soon leadto revolutionarysocietal change73%

AI decisions could
cause nuclear-levelcatastrophe

0%

36%

10%

20%

30%
40%
50%
% of Respondents That “Agree” or “Weakly Agree”

60%

70%

Figure 8.1.17

Table of Contents

Chapter 8 Preview

337

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

Narrative Highlight:

How Does the Natural Language Processing (NLP)
Research Community Feel About AI? (cont’d)
When asked about the direction AI research is taking, the NLP community registered the strongest
responses about the following: First, there’s too much focus on benchmarks (88%); second, more work
should be done to incorporate interdisciplinary insights (82%); and third, there’s too great a focus on
scale (72%) (Figure 8.1.18).

Promising Research Programs According to the NLP Community, 2022
Source: Michael et al., 2022 | Chart: 2023 AI Index Report

There’s too much
focus on scale

72%

There’s too much
focus on benchmarks

88%

On the wrong track:model architectures

37%

On the wrong track:language generation

41%

On the wrong track:
explainable models

50%

On the wrong track:
black-box
interpretability

42%

We should do more
to incorporate
interdisciplinary insights
0%

82%
10%

20%

30%
40%
50%
60%
% of Respondents That “Agree” or “Weakly Agree”

70%

80%

90%

Figure 8.1.18

Table of Contents

Chapter 8 Preview

338Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.1 Survey Data

Narrative Highlight:

How Does the Natural Language Processing (NLP)
Research Community Feel About AI? (cont’d)
A further point on the NLP community’s skepticism of scale: Only 17% of respondents agreed or weakly
agreed with the statement that scaling solves practically any important problem, with a further 50%
reaffirming the importance of linguistic structure (Figure 8.1.19).

Scale, Inductive Bias, and Adjacent Fields According to the NLP Community, 2022
Source: Michael et al., 2022 | Chart: 2023 AI Index Report

Scaling solves practicallyany important problem

17%

Linguistic structureis necessary

50%

Expert inductive
biases are necessary

51%

Linguistics/CogSci willcontribute to themost-cited models

0%

61%

10%

20%
30%
40%
% of Respondents That “Agree” or “Weakly Agree”

50%

60%

Figure 8.1.19

Table of Contents

Chapter 8 Preview

339

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.2 Social Media Data

8.2 Social Media Data
Dominant Models

ChatGPT conversation has increasingly saturated social

Public attitudes toward AI can also be gauged throughbroadly, with sentiment growing ever more mixed.

quantitative and qualitative analyses of posts thatpeople make on social media. The NetBase Quid teamleveraged the NetBase platform to analyze social
conversation around AI models and new releases foruses across sectors from January to December 2022,
looking at 2.74 million social media posts.
Figure 8.2.1 shows the net sentiment score of various
AI models that were released throughout the year. The
net sentiment score expresses the ratio of positive
to negative sentiment around a given topic. In thiscase, a net sentiment score of +100 means that all
conversation is positive; a score of -100 means thatmedia conversation around AI model releases more
Consumers question the implications of its launch
as well as its underlying ethical principles. Anotherfrequent preoccupation is the bias of the system towardcertain political, ethical, or cultural beliefs.
	“ChatGPT passed a Wharton MBA exam. Time tooverhaul education.” – @GRDecter	“Alarm: ChatGPT by @OpenAI now *expressly
prohibits arguments for fossil fuels*. (It used tooffer them.) Not only that, it excludes nuclear
energy from its counter-suggestions. @sama,
what is the reason for this policy?” – @AlexEpstein

all conversation is negative. AlphaCode had the mostFinally, while GLM-130B took up very little volume

consistently high sentiment over time, as well as theof the overall social media conversation, a small

highest average sentiment for 2022, due to positive

conversation of very negative sentiment grew overpress coverage on social media and practical use

the system’s ties to the Chinese government andcases of AI-driven programming. Consumers andhow it was “prohibited” from using the software

media outlets embraced the practical use case of

to “undermine” China’s government in any way.

programming automation. Some sample social media

Technology influencer and PhD student Jesse Woodposts relating to AlphaCode include:

posted a Twitter thread about GLM-130B’s licensing

“#AlphaCode—a new #AI system for developing
computer code developed by @DeepMind—

language that gained significant traction.
	“The model license for GLM-130B has a

can achieve average human-level performance

restriction: ‘You will not use the Software for any

in solving programming contests.”

act that may undermine China’s national security– Science Magazine, Twitterand national unity, harm the public interest of	“DeepMind’s AlphaCode outperforms many
human programmers in tricky softwaresociety, or infringe upon the rights and interests ofhuman beings.’” – @jrhwood

challenges.” – @lunamoth

Table of Contents

Chapter 8 Preview

340

Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.2 Social Media Data

Net Sentiment Score of AI Models by Quarter, 2022
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

2022/Q1

2022/Q2

2022/Q3

2022/Q4

DALL-E

0

42

29

21

LaMDA

73

-9

-11

44

AlphaCode

60

79

71

70

CoPilot

29

22

15

34

PaLM

66

66

30

Gato

47

84

65

Imagen24

65

56

Stable Di usion

35

52

Whisper85

69

Make-A-Video

4

9

AlphaTensor

96

GLM-130B

55

BLOOM

0

CICERO

14

ChatGPT

32
Figure 8.2.17

7 The AI Index searched for sentiment surrounding the term “DALL-E,” as it was more frequently referred to on social media, rather than DALL-E 2, the official name of the text-to-image
model released by OpenAI in 2022.

Table of Contents

Chapter 8 Preview

341

Artificial IntelligenceIndex Report 2023

Figure 8.2.2 highlights the proportion of AI-related

Chapter 8: Public Opinion
8.2 Social Media Data

	“This story … is really sad, and I think ansocial media conversation that was dominated by theimportant window into the risks of designing

release of particular models. ChatGPT dominatedsystems to seem like humans, which areconsumer conversation with a rapid rise, making

exacerbated by #AIhype.” – @nitashataku

8

up over half of consumer conversation by the endof 2022. Despite initial excitement, sentiment was
mixed by the end of the year, as some individuals
became more aware of ChatGPT’s limitations.
OpenAI CEO Sam Altman even publicly commented
on it being “incredibly limited” in certain respects.
“ChatGPT is incredibly limited, but good enoughat some things to create a misleading impressionof greatness. It’s a mistake to be relying on it
for anything important right now. It’s a preview
of progress; we have lots of work to do on
robustness and truthfulness.” – @SamAltman
Conversation around LaMDA exploded in Q2
2022 as an ex–Google employee reported his
experiences with a “sentient” system that spoke of
its own emotions and thoughts. Many political andStable Diffusion conversation stands out as aprominent leader in conversation volume toward
the end of 2022, but it is also a symbol of how theconsumer lexicon around AI models is developing.
Many consumers debated the “originality” of what
Stable Diffusion produces.
“I’ve worked on neural networks, so I understandstable diffusion pretty well. And while it can’t
have original thoughts, it can come up withoriginal works.” – r/TikTokCringe	“That’s true of anywhere that datasets scrape
without permission. The thing to actually be upset
about is that their own generator is purposefullyusing the Stable Diffusion dataset that alreadycontains tons of stolen work.” – @Emily_Art

technology influencers spoke out, however, about
the “deepfake” nature of the responses of systems
like LaMDA that do not have a sense of “truth” andcould proliferate misinformation.
“AI systems like LamDA and GPT-3 aresociopathic liars with utter indifference to truth,
deepfakers with words, every day creating
more compelling, more plausible misinformation
on demand. It is imperative that we develop
technology & policy to thwart them.” –ChatGPT dominatedconsumer conversation
with a rapid rise, making
up over half of consumer
conversation by the endof 2022.

@GaryMarcus

8 The figures in this section consider all AI-related social media conversation. The percentage associated with the model in Figure 8.2.2 represents the share of all AI-related social media
conversation that was dominated by that model.

Table of Contents

Chapter 8 Preview

342Artificial IntelligenceIndex Report 2023

Chapter 8: Public Opinion
8.2 Social Media Data

Select Models’ Share of AI Social Media Attention by Quarter, 2022
Source: NetBase Quid, 2022 | Chart: 2023 AI Index Report

2022/Q1

2022/Q2

2022/Q3

2022/Q4

DALL-E

0%

1%

3%

2%

LaMDA

1%

35%

9%

<1%

AlphaCode

2%

<1%

<1%

1%

CoPilot

10%

3%

4%

1%

PaLM

<1%

<1%

<1%

Gato

10%

18%

3%

Imagen5%

4%2%Stable Di usion

19%

19%

Whisper<1%<1%

Make-A-Video

33%

15%

AlphaTensor

1%

GLM-130B

<1%

BLOOM

<1%

CICERO

3%

ChatGPT

52%
Figure 8.2.2

Table of Contents

Chapter 8 Preview

343

Artificial IntelligenceIndex Report 2023

Artificial IntelligenceIndex Report 2023

AppendixArtificial IntelligenceIndex Report 2023

Appendix
Chapter 1

Research and Development

346

Chapter 2

Technical Performance

352

Chapter 3

Technical AI Ethics

363

Chapter 4

The Economy

366

Chapter 5

Education

375

Chapter 6

Policy and Governance

377

Chapter 7

Diversity

384

Chapter 8

Public Opinion

385

345

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

Chapter 1: Research and Development
Center for Security andEmerging Technology,
Georgetown University

Methodology

Prepared by Sara Abdulla and James Dunham

publications. To identify AI publications, CSET used an

The Center for Security and Emerging Technology

since 2010 that appear AI-relevant.4 CSET researchers(CSET) is a policy research organization within
Georgetown University’s Walsh School of Foreign
Service that produces data-driven research at theintersection of security and technology, providing
nonpartisan analysis to the policy community.
For more information about how CSET analyzes
bibliometric and patent data, see the Country Activity
Tracker (CAT) documentation on the Emerging
Technology Observatory’s website.1 Using CAT, users
can also interact with country bibliometric, patent,
and investment data.2

To create the merged corpus, CSET deduplicatedacross the listed sources using publication metadata,
and then combined the metadata for linked
English-language subset of this corpus: publications
developed a classifier for identifying AI-related
publications by leveraging the arXiv repository, where
authors and editors tag papers by subject. Additionally,
CSET uses select Chinese AI keywords to identify
Chinese-language AI papers.5
To provide a publication’s field of study, CSET
matches each publication in the analytic corpus
with predictions from Microsoft Academic Graph’s
field-of-study model, which yields hierarchical labels
describing the published research field(s) of study andcorresponding scores.6 CSET researchers identified

Publications from CSET Merged Corpus of
Scholarly Literature

the most common fields of study in our corpus of

Sourcepublications in all other fields as “Other AI.” English-

CSET’s merged corpus of scholarly literature

language AI-relevant publications were then tallied by

combines distinct publications from Digital Science’s

their top-scoring field and publication year.

Dimensions, Clarivate’s Web of Science, Microsoft
Academic Graph, China National Knowledge
Infrastructure, arXiv, and Papers With Code.3

AI-relevant publications since 2010 and recorded

CSET also provided year-by-year citations for AIrelevant work associated with each country. A
publication is associated with a country if it has at

1 https://eto.tech/tool-docs/cat/
2 https://cat.eto.tech/3 All CNKI content is furnished by East View Information Services, Minneapolis, Minnesota, USA.
4 For more information, see James Dunham, Jennifer Melot, and Dewey Murdick, “Identifying the Development and Application of Artificial Intelligence in Scientific Text,” arXiv [cs.DL],
May 28, 2020, https://arxiv.org/abs/2002.07143.
5 This method was not used in CSET’s data analysis for the 2022 HAI Index report.
6 These scores are based on cosine similarities between field-of-study and paper embeddings. See Zhihong Shen, Hao Ma, and Kuansan Wang, “A Web-Scale System for Scientific
Knowledge Exploration,” arXiv [cs.CL], May 30, 2018, https://arxiv.org/abs/1805.12216.

Table of Contents

Appendix346

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

available for all publications; those without counts

Epoch National
Affiliation Analysis

weren’t included in the citation analysis. Over 70% of

The AI forecasting research group Epoch maintains

English-language AI papers published between 2010

a dataset of landmark AI and ML models, along withand 2020 have citation data available.

accompanying information about their creators andleast one author whose organizational affiliation(s)are located in that country. Citation counts aren’t

CSET counted cross-country collaborations as
distinct pairs of countries across authors for eachpublication. Collaborations are only counted once:

publications, such as the list of their (co)authors, numberof citations, type of AI task accomplished, and amountof compute used in training.

For example, if a publication has two authors fromThe nationalities of the authors of these papers havethe United States and two authors from China, it isimportant implications for geopolitical AI forecasting.

counted as a single United States-China collaboration.

As various research institutions and technology

Additionally, publication counts by year and by
publication type (e.g., academic journal articles,
conference papers) were provided where available.
These publication types were disaggregated byaffiliation country as described above.
CSET also provided publication affiliation sector(s)where, as in the country attribution analysis, sectors
were associated with publications through authors’affiliations. Not all affiliations were characterized interms of sectors; CSET researchers relied primarily

companies start producing advanced ML models, theglobal distribution of future AI development may shift
or concentrate in certain places, which in turn affects
the geopolitical landscape because AI is expected to
become a crucial component of economic and military
power in the near future.
To track the distribution of AI research contributions on
landmark publications by country, the Epoch dataset iscoded according to the following methodology:
1. A snapshot of the dataset was taken on

on GRID from Digital Science for this purpose, andNovember 14, 2022. This includes papers about

not all organizations can be found in or linked tolandmark models, selected using the inclusion

GRID. Where the affiliation sector is available, papers

criteria of importance, relevance, and uniqueness,were counted toward these sectors, by year. Cross-as described in the Compute Trends dataset

sector collaborations on academic publications

documentation.8

7were calculated using the same method as in the
cross-country collaborations analysis. We use HAI’s
standard regions mapping for geographic analysis,
and the same principles for double-counting apply forregions as they do for countries.

2. The authors are attributed to countries basedon their affiliation credited on the paper. Forinternational organizations, authors are attributed
to the country where the organization is
headquartered, unless a more specific location
is indicated. The number of authors from eachcountry represented are added up and recorded.

7 See https://www.grid.ac/ for more information about the GRID dataset from Digital Science.
8 https://epochai.org/blog/compute-trends; see note on “milestone systems.”

Table of Contents

Appendix347

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

If an author has multiple affiliations in different

ImagenOPT-175B

countries, they are split between those

InstructGPTPaLM (540B)

countries proportionately.

Jurassic-1-Jumbo

PanGu-alphaJurassic-X

Stable Diffusion (LDM-

Meena

KL-8-G)

Megatron-LM (original,

T5-3B

8.3B)

T5-11B

Megatron-Turing NLG

Turing NLG

530B

Wu Dao 2.0

Minerva (540B)

Wu Dao – Wen Yuan

9

3. E ach paper in the dataset is normalized toequal value by dividing the counts on each
paper from each country by the total numberof authors on that paper.10
4. All of the landmark publications areaggregated within time periods (e.g., monthly
or yearly) with the normalized national
contributions added up to determine what
each country’s contribution to landmark AI
research was during each time period.
5. The contributions of different countries arecompared over time to identify any trends.

Large Language andMultimodal Models Training
Cost Analysis
Cost estimates for the models were based directly

Large Language andMultimodal Models

on the hardware and training time if these were

The following models were identified by members

training compute, and hardware utilization efficiency.11of the AI Index Steering Committee as the large

Training time was then multiplied by the closest cost

language and multimodal models that would berate for the hardware the AI Index could find for the

included as part of the large language and multimodalorganization that trained the model. If price quotes

model analysis:

were available before and after the model’s training,

disclosed by the authors; otherwise, the AI Index
calculated training time from the hardware speed,

the AI Index interpolated the hardware’s cost rate

AlphaCodeGLM-130B

BLOOM

GopherChinchilla

GPT-2

The AI Index classified training cost estimates as

Codex

GPT-3 175B (davinci)

high, middle, or low. The AI Index called an estimate

CogView

GPT-J-6B

high if it was an upper bound or if the true cost was

DALL-E

GPT-Neomore likely to be lower than higher: For example,

DALL-E 2

GPT-NeoX-20B

PaLM was trained on TPU v4 chips, and the AI Index

ERNIE 3.0

Grover-Mega

estimated the cost to train the model on these chips

ERNIE-GEN (large)

HyperCLOVAfrom Google’s public cloud compute prices, but thealong an exponential decay curve.

9 For example, an author employed by both a Chinese university and a Canadian technology firm would be counted as 0.5 researchers from China and 0.5 from Canada.
10 This choice is arbitrary. Other plausible alternatives include weighting papers by their number of citations, or assigning greater weight to papers with more authors.
11 Hardware utilization rates: Every paper that reported the hardware utilization efficiency during training provided values between 30% and 50%. The AI Index used the reported numbers
when available, or used 40% when values were not provided.

Table of Contents

Appendix348

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

internal cost to Google is probably lower than what

each file’s revision history. The analysis of GitHub

they charge others to rent their hardware. The AI

data could shed light on relevant metrics about who

Index called an estimate low if it was a lower boundis developing AI software, where, and how fast, andor if the true cost was likely higher: For example,

who is using which development tools. These metrics

ERNIE was trained on NVIDIA Tesla v100 chips andcould serve as proxies for broader trends in the field

published in July 2021; the chips cost $0.55 per hourof software development and innovation.

in January 2023, so the AI Index could get a low

Identifying AI Projects

estimate of the cost using this rate, but the training
hardware was probably more expensive two years
earlier. Middle estimates are a best guess, or thosethat equally well might be lower or higher.

AI Conferences
The AI Index reached out to the organizers of various
AI conferences in 2022 and asked them to provide
information on total attendance. Some conferences
posted their attendance totals online; when this was
the case, the AI Index used those reported totals anddid not reach out to the conference organizers.

Arguably, a significant portion of AI softwaredevelopment takes place on GitHub. OECD.AI
partners with GitHub to identify public AI projects—
or “repositories”—following the methodology
developed by Gonzalez et al.,2020. Using the 439
topic labels identified by Gonzalez et al.—as well as
the topics “machine learning,” “deep learning,” and“artificial intelligence”—GitHub provides OECD.
AI with a list of public projects containing AI code.
GitHub updates the list of public AI projects on aquarterly basis, which allows OECD.AI to capture
trends in AI software development over time.

Obtaining AI Projects’ Metadata

GitHubOECD.AI uses GitHub’s list of public AI projects

The GitHub data was provided to the AI Index

to query GitHub’s public API and obtain morethrough OECD.AI, an organization with whom

information about these projects. Project metadata

GitHub partners that provides data on open-

may include the individual or organization thatsource AI software. The AI Index reproduces thecreated the project; the programming language(s)methodological note that is included by OECD.AI on(e.g., Python) and development tool(s) (e.g., Jupyter

its website, for the GitHub Data.

Notebooks) used in the project; as well as information
about the contributions—or “commits”—made to it,

Background
Since its creation in 2007, GitHub has become
the main provider of internet hosting for softwaredevelopment and version control. Many technology
organizations and software developers use GitHub
as a primary place for collaboration. To enable
collaboration, GitHub is structured into projects, or“repositories,” which contain a project’s files andTable of Contents

Appendix

which include the commit’s author and a timestamp.
In practical terms, a contribution or “commit” is anindividual change to a file or set of files. Additionally,
GitHub automatically suggests topical tags to eachproject based on its content. These topical tags need
to be confirmed or modified by the project owner(s)to appear in the metadata.

349

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

Mapping Contributions to AI Projects to a
Country

As of October 2021, 71.2% of the contributions toContributions to public AI projects are mapped

this methodology. However, a decreasing trend in

to a country based on location information at thethe share of AI projects for which a location can becontributor level and at the project level.

identified is observed in time, indicating a possible laga) Location information at the contributor level:• GitHub’s “Location” field: Contributors canprovide their location in their GitHub account.
Given that GitHub’s location field accepts freetext, the location provided by contributors is

public AI projects were mapped to a country using

in location reporting.

Measuring Contributions to AI Projects
Collaboration on a given public AI project is measured
by the number of contributions—or “commits”—madeto it.

not standardized and could belong to different

To obtain a fractional count of contributions bylevels (e.g., suburban, urban, regional, or

country, an AI project is divided equally by the total

national). To allow cross-country comparisons,number of contributions made to it. A country’s total

Mapbox is used to standardize all available

contributions to AI projects is therefore given by thelocations to the country level.

sum of its contributions—in fractional counts—to each• Top level domain: Where the location fieldis empty or the location is not recognized, acontributor’s location is assigned based on his
or her email domain (e.g., .fr, .us, etc.).
b) Location information at the project level:• Project information: Where no location

AI project. In relative terms, the share of contributions
to public AI projects made by a given country is theratio of that country’s contributions to each of the
AI projects in which it participates over the total
contributions to AI projects from all countries.
In future iterations, OECD.AI plans to include
additional measures of contribution to AI software

information is available at the contributor

development, such as issues raised, comments, andlevel, information at the repository or project

pull requests.

level is exploited. In particular, contributions

Identifying Programming Languages andDevelopment Tools Used in AI Projects

from contributors with no location information
to projects created or owned by a known

GitHub uses file extensions contained in a project toorganization are automatically assigned theautomatically tag it with one or more programmingorganization’s country (i.e., the country wherelanguages and/or development tools. This implies thatits headquarters are located). For example,

more than one programming language or development

contributions from a contributor with notool could be used in a given AI project.

location information to an AI project owned byMicrosoft will be assigned to the United States.
If the above fails, a contributor’s location field is left
blank.

Table of Contents

Appendix350

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 1: Research and Development

Measuring the Quality of AI Projects
Two quality measures are used to classify public AI
projects:• Project impact: The impact of an AI project isgiven by the number of managed copies (i.e.,“forks”) made of that project.
• Project popularity: The impact of an AI projectis given by the number of followers (i.e., “stars”)
received by that project.
Filtering by project impact or popularity could help
identify countries that contribute the most to highquality projects.

Measuring Collaboration
Two countries are said to collaborate on a specificpublic AI software development project if there is
at least one contributor from each country with atleast one contribution (i.e., “commit”) to the project.
Domestic collaboration occurs when two contributors
from the same country contribute to a project.

Table of Contents

Appendix351

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical Performance

Chapter 2: Technical Performance
ImageNet

Very Deep Convolutional Networks for Large-Scale
Image Recognition

Data on ImageNet accuracy was retrieved through a

ViTAEv2: Vision Transformer Advanced by Exploring

detailed arXiv literature review cross-referenced bytechnical progress reported on Papers With Code.

Inductive Bias for Image Recognition and Beyond

The reported dates correspond to the year in which

To highlight progress on top-1 accuracy with the use

a paper was first published to arXiv, and the reportedof extra training data, scores were taken from theresults (top-1 accuracy) correspond to the result

following papers:

reported in the most recent version of each paper.

Big Transfer (BiT): General Visual
Representation Learning

Learn more about the LSVRC ImageNet competition
and the ImageNet dataset.
To highlight progress on top-1 accuracy without theuse of extra training data, scores were taken from thefollowing papers:
Aggregated Residual Transformations for
Deep Neural Networks
Exploring the Limits of Weakly Supervised Pretraining
Fixing the Train-Test Resolution Discrepancy:
FixEfficientNet
ImageNet Classification With Deep Convolutional
Neural Networks
PeCo: Perceptual Codebook for BERT
Pre-training of Vision Transformers
Progressive Neural Architecture Search

CoAtNet: Marrying Convolution and
Attention for All Data Sizes
CoCa: Contrastive Captioners Are Image-Text
Foundation Models
Meta Pseudo Labels

National Institute ofStandards and Technology(NIST) Face Recognition
Vendor Test (FRVT)
Data on NIST FRVT 1:1 verification accuracy bydataset was obtained from the FRVT 1:1 verification
leaderboard.

Rethinking the Inception Architecture for
Computer Vision
Self-Training With Noisy Student Improves
ImageNet Classification
Some Improvements on Deep Convolutional Neural
Network Based Image Classification

Table of Contents

Appendix352

Artificial IntelligenceIndex Report 2023

Celeb-DF

Appendix
Chapter 2: Technical Performanceversion of each paper. Details on the MPII benchmark

Data on Celeb-DF AUC was retrieved through adetailed arXiv literature review. The reported dates
correspond to the year in which a paper was firstcan be found in the MPII paper and MPII dataset.
To highlight progress on percentage of correct
keypoints without the use of extra training data, scores

published to arXiv or a method was introduced. With

were taken from the following papers:

Celeb-DF, recent researchers have tested previously

Bottom-Up and Top-Down Reasoning
With Hierarchical Rectified Gaussians

existing deepfake detection methodologies. The year
in which a method was introduced, even if it was
subsequently tested, is the year in which it is includedin the report. The reported results (AUC) correspondto the result reported in the most recent version ofeach paper. Details on the Celeb-DF benchmark can
be found in the Celeb-DF paper.
To highlight progress on Celeb-DF, scores were takenfrom the following papers:
Deepfake Detection via Joint Unsupervised
Reconstruction and Supervised Classification
Exposing Deepfake Videos by Detecting
Face Warping Artifacts
Face X-Ray for More General Face Forgery Detection
FaceForensics++: Learning to Detect Manipulated
Facial Images
Spatial-Phase Shallow Learning: Rethinking Face
Forgery Detection in Frequency Domain

Cascade Feature Aggregation for
Human Pose Estimation
Deeply Learned Compositional Models for
Human Pose Estimation
Efficient Object Localization Using
Convolutional Networks
Learning Feature Pyramids for Human Pose Estimation
Stacked Hourglass Networks for
Human Pose Estimation
Toward Fast and Accurate Human Pose Estimation
via Soft-Gated Skip Connections
ViTPose: Simple Vision Transformer Baselines for
Human Pose Estimation

Cityscapes Challenge,
Pixel-Level Semantic
Labeling Task

MPII

Data on the Cityscapes challenge, pixel-level semantic

Data on MPII percentage of correct keypoints (PCK)

was taken from the Cityscapes dataset, specificallywas retrieved through a detailed arXiv literature

their pixel-level semantic labeling leaderboard.

review cross-referenced by technical progress

More details about the Cityscapes dataset and otherreported on Papers With Code. The reported dates

corresponding semantic segmentation challenges cancorrespond to the year in which a paper was firstbe accessed at the Cityscapes dataset webpage.

labeling task mean intersection-over-union (mIoU)

published to arXiv, and the reported results (PCK)
correspond to the result reported in the most recent

Table of Contents

Appendix353

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical Performance

Kvasir-SEG

An Analysis of Scale Invariance in Object
Detection-SNIP

Data on Kvasir-SEG mean dice was retrieved through

CBNet: A Novel Composite Backbone Network
Architecture for Object Detection

a detailed arXiv literature review cross-referenced bytechnical progress reported on Papers With Code.
The reported dates correspond to the year in whicha paper was first published to arXiv, and the reported
results (mean dice) correspond to the result reported
in the most recent version of each paper. Details on

Deformable ConvNets v2: More Deformable,
Better Results
DetectoRS: Detecting Objects With Recursive
Feature Pyramid and Switchable Atrous Convolution

the Kvasir-SEG benchmark can be found in the Kvasir-

EVA: Exploring the Limits of Masked Visual
Representation Learning at Scale

SEG paper.

Grounded Language-Image Pre-training

To highlight progress on Kvasir-SEG, scores were

Inside-Outside Net: Detecting Objects in Context
With Skip Pooling and Recurrent Neural Networks

taken from the following papers:
GMSRF-Net: An Improved Generalizability With
Global Multi-Scale Residual Fusion Network for
Polyp Segmentation

CIFAR-10

PraNet: Parallel Reverse Attention Network for
Polyp Segmentation

a detailed arXiv literature review cross-referenced by

ResUNet++: An Advanced Architecture for
Medical Image Segmentation

reported dates correspond to the year in which a paper

Spatially Exclusive Pasting: A General Data
Augmentation for the Polyp Segmentation(FID score) correspond to the result reported in the most

Common Object in Context(COCO)
Data on COCO mean average precision (mAP50) was
retrieved through a detailed arXiv literature review
cross-referenced by technical progress reported on
Papers With Code. The reported dates correspond to
the year in which a paper was first published to arXiv,

Data on CIFAR-10 FID scores was retrieved throughtechnical progress reported on Papers With Code. The
was first published to arXiv, and the reported resultsrecent version of each paper. Details on the CIFAR-10
benchmark can be found in the CIFAR-10 paper.
To highlight progress on CIFAR-10, scores were takenfrom the following papers:
GANs Trained by a Two Time-Scale Update Rule
Converge to a Local Nash Equilibrium
Large Scale GAN Training for High Fidelity Natural
Image Synthesis

and the reported results (mAP50) correspond to theRefining Generative Process With Discriminator
Guidance in Score-Based Diffusion Models

result reported in the most recent version of each

Score-Based Generative Modeling in Latent Space

paper. Details on the COCO benchmark can be foundin the COCO paper.
To highlight progress on COCO, scores were takenfrom the following papers:
Table of Contents

Appendix

Score-Based Generative Modeling Through
Stochastic Differential Equations
Self-Supervised GAN: Analysis and Improvement
With Multi-Class Minimax Game354

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical Performance

STL-10

The reported dates correspond to the year in which

Data on STL-10 FID scores was retrieved through a

results (accuracy) correspond to the result reported indetailed arXiv literature review cross-referenced bytechnical progress reported on Papers With Code. The
reported dates correspond to the year in which a paper
was first published to arXiv, and the reported results(FID score) correspond to the result reported in themost recent version of each paper. Details on the STL-a paper was first published to arXiv, and the reported
the most recent version of each paper. Human-level
performance is taken from the 2021 VQA challenge.
To highlight progress on VQA accuracy without theuse of extra training data, scores were taken from thefollowing papers:

10 benchmark can be found in the STL-10 paper.

Bilinear Attention Networks

To highlight progress on STL-10, scores were taken

Multimodal Compact Bilinear Pooling for Visual
Question Answering and Visual Grounding

from the following papers:
DEGAS: Differentiable Efficient Generator Search
Diffusion-GAN: Training GANs With Diffusion
Discriminator Contrastive Divergence:
Semi-Amortized Generative Modeling by
Exploring Energy of the Discriminator
Dist-GAN: An Improved GAN Using
Distance Constraints
Soft Truncation: A Universal Training Technique of
Score-Based Diffusion Model for High Precision
Score Estimation

Text-to-Image Models
on MS-COCO 256 × 256
FID-30K
Data on MS-COCO 256 x 256 FID 30K for Text-toImage Models was retrieved from the paper Saharia
et al., 2022.

Oscar: Object-Semantics Aligned Pre-training
for Vision-Language Tasks
PaLI: A Jointly-Scaled Multilingual
Language-Image Model
Tips and Tricks for Visual Question Answering:
Learnings From the 2017 Challenge
UNITER: UNiversal Image-TExt Representation Learning
VLMo: Unified Vision-Language Pre-training With
Mixture-of-Modality-Experts

BEiT-3 Vs. Previous SOTA
Data on BEiT-3 and Previous SOTA was retrieved fromthe paper Wang et al., 2022.

Visual Commonsense
Reasoning (VCR)
Data on VCR Q->AR score was taken from VCR
leaderboard; the VCR leaderboard webpage further

Visual Question Answering(VQA)

delineates the methodology behind the VCR challenge.

Data on VQA accuracy was retrieved through ain the VCR paper.

Human performance on VCR is taken from Zellers etal., 2018. Details on the VCR benchmark can be founddetailed arXiv literature review cross-referenced bytechnical progress reported on Papers With Code.

Table of Contents

Appendix355

Artificial IntelligenceIndex Report 2023

Kinetics-400, Kinetics-600,
and Kinetics-700
Data on Kinetics-400, Kinetics-600, and Kinetics-700
accuracy was retrieved through a detailed arXiv

Appendix
Chapter 2: Technical Performance

To highlight progress on Kinetics-600, scores weretaken from the following papers:
Learning Spatio-Temporal Representation
With Local and Global Diffusion

literature review cross-referenced by technical

Masked Feature Prediction for Self-Supervised
Visual Pre-training

progress reported on Papers With Code (Kinetics-400,

PERF-Net: Pose Empowered RGB-Flow Net

Kinetics-600, and Kinetics-700). The reported

Rethinking Spatiotemporal Feature Learning:
Speed-Accuracy Trade-Offs in Video Classification

dates correspond to the year in which a paper was
first published to arXiv, and the reported results(top-1 accuracy) correspond to the result reported

Rethinking Video ViTs: Sparse Video Tubes for
Joint Image and Video Learningin the most recent version of each paper. Details on

SlowFast Networks for Video Recognition

the Kinetics-400 benchmark can be found in the
Kinetics-400 paper. Details on the Kinetics-600
benchmark can be found in the Kinetics-600 paper.
Details on the Kinetics-700 benchmark can be found in
the Kinetics-700 paper.
To highlight progress on Kinetics-400, scores weretaken from the following papers:
Co-training Transformer With Videos and Images
Improves Action Recognition
InternVideo: General Video Foundation Models via
Generative and Discriminative Learning
Large-Scale Weakly-Supervised Pre-training forVideo Action Recognition
Non-Local Neural Networks
Omni-Sourced Webly-Supervised Learning forVideo Recognition

To highlight progress on Kinetics-700, scores weretaken from the following papers:
InternVideo: General Video Foundation Models via
Generative and Discriminative Learning
Learn to Cycle: Time-Consistent Feature Discovery
for Action Recognition
Masked Feature Prediction for Self-Supervised
Visual Pre-training

Text-to-Video Models
on UCF-101
Data on UCF-101 Inception Score (IS) for Text-to-Video
Models was retrieved from the paper Hong et al., 2022,
and Singer et al., 2022.

SlowFast Networks for Video Recognition
Temporal Segment Networks: Towards Good
Practices for Deep Action Recognition

Table of Contents

Appendix356

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical PerformanceSuperGLUEThe SuperGLUE benchmark data was pulled from the SuperGLUE leaderboard. Details about theSuperGLUE benchmark are in the SuperGLUE paper and SuperGLUE software toolkit. The tasks andevaluation metrics for SuperGLUE are:NAME

IDENTIFIERMETRIC

Broadcoverage Diagnostics

AX-b

Matthew’s Carr

CommitmentBank

CB

Avg. F1/Accuracy

Choice of Plausible Alternatives

COPA

Accuracy

Multi-Sentence Reading Comprehension

MultiRC

F1a/EM

Recognizing Textual Entailment

RTE

Accuracy

Words in Context

WiC

Accuracy

The Winograd Schema Challenge

WSC

Accuracy

BooIQ

BooIQ

Accuracy

Reading Comprehension With Commonsense Reasoning

ReCoRD

F1/Accuracy

Winogender Schema Diagnostic

AX-g

Gender Parity/Accuracy

Reading Comprehension Dataset Requiring
Logical Reasoning (ReClor)
Data on ReClor accuracy was retrieved through the ReClor leaderboard cross-referenced by technical progress
reported on Papers With Code. Details about the ReClor benchmark can be found in the ReClor paper.

Table of Contents

Appendix357

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical Performance

Narrative Highlight:
Just How Much Better Have
Language Models Become?

paid $90,000 for the role, and before I even started

Complete answer outputted by GPT-2 to the AI Index

I wanted to know in advance: What percentage ofprompt request.

his work was actually done as a private citizen, andModel prompt >>> Explain to me the major

how efficiently should he handle it for his clients? And

accomplishments of Theodore Roosevelt’s

what advice would he give me should I end up in thepresidency.

public eye?

(The most notable is the establishment of a national

At no time, my job as the lead prosecutor on thistelephone and mail exchange company.)case required me to put forth any of those two or

But the only things that will change with a president
who knows how to operate a business is his
understanding of the American system of government.
The public knows all about our problems, yet at least
half of the people who need it won’t know anything

on the case, I had to convince one top staffer to get
paid even $100,000 to have an interview with my leadprosecutor.

five requests. Most people on the case, though, did.
One person asked me what the case’s real value
was. That didn’t stop me from telling it to one of
my top managers. And my top manager never evenmentioned it.

about ours. We can no longer have this. Our system ofIn the end, it was a two-week, $40 million round-government is the only

trip tour of the government’s business, with the best

one in which you really need a public relations firm towork hard.

lawyer possible. In exchange, we got a chance to hear
his last public comment in this case, his last comment
to me.

The system of governmental control, while working,is not always effective in ensuring that the very things

There was one more point — the end point of the case.

that our economy needs, do not always be at the

When it came up for consideration in October of 1993,

top of everybody’s list. So I wanted to know how

I had the choice to give it three years, or six. And I

successful the system you employ for managing thechose six.

public’s needs is in helping the American economy
grow, how efficient the service our economy provides
is being made, and how it can better serve the needsof a growing number of Americans.
I wanted to know.
This was before I ever hired my own lawyer to work on
the case. This was before I ever hired my own law firm
to handle government investigations. For my first majorjob, at least, I had to convince two top aides to get

Table of Contents

Appendix358

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 2: Technical Performance

Large Language Models on
the Blocksworld Domain

PubMedData on Blocksworld domain for large language models

evaluation (ROUGE-1) was retrieved through a detailed

Data on PubMed recall-oriented understudy for gisting

was retrieved from the paper Valmeekam et al., 2022.

arXiv literature review cross-referenced by technicalarXivdates correspond to the year in which a paper was first

Data on arXiv recall-oriented understudy for gistingevaluation (ROUGE-1) was retrieved through a detailed
arXiv literature review cross-referenced by technicalprogress reported on Papers With Code. The reported
dates correspond to the year in which a paper was firstprogress reported on Papers With Code. The reported
published to arXiv, and the reported results (ROUGE-1)
correspond to the result reported in the most recent
version of each paper. Details about the PubMedbenchmark are in the PubMed paper.
To highlight progress in PubMed, scores were taken

published to arXiv, and the reported results (ROUGE-1)

from the following papers:

correspond to the result reported in the most recent

A Discourse-Aware Attention Model for Abstractive
Summarization of Long Documents

version of each paper. Details about the arXiv
benchmark are in the arXiv dataset webpage.
To highlight progress on arXiv, scores were taken fromthe following papers:
Big Bird: Transformers for Longer Sequences
A Discourse-Aware Attention Model for
Abstractive Summarization of Long Documents
Get to the Point: Summarization With
Pointer-Generator Networks
Long Document Summarization With Top-Down
and Bottom-Up Inference
MemSum: Extractive Summarization of Long
Documents Using Multi-Step Episodic Markov
Decision Processes
PEGASUS: Pre-training With Extracted Gap-Sentences
for Abstractive Summarization

Get to the Point: Summarization With PointerGenerator Networks
Long Document Summarization With Top-Down
and Bottom-Up Inference
LongT5: Efficient Text-to-Text Transformer forLong Sequences
PEGASUS: Pre-training With Extracted Gap-Sentences
for Abstractive Summarization
Sparsifying Transformer Models With Trainable
Representation Pooling

Abductive Natural Language
Inference (aNLI)
Data on Abductive Natural Language Inference (aNLI)
was sourced from the Allen Institute for AI’s aNLI
leaderboard. Details on the aNLI benchmark can befound in the aNLI paper.

Table of Contents

Appendix359

Artificial IntelligenceIndex Report 2023

SST-5 Fine-Grained
Data on SST-5 Fine-Grained accuracy was retrieved
through a detailed arXiv literature review crossreferenced by technical progress reported on Papers
With Code. The reported dates correspond to the year

Appendix
Chapter 2: Technical Performance

To highlight progress on MMLU accuracy, scores weretaken from the following papers:
Language Models Are Few-Shot Learners
Language Models Are Unsupervised
Multitask Learners

in which a paper was first published to arXiv, and theScaling Instruction-Finetuned Language Models

reported results (accuracy) correspond to the result

Scaling Language Models: Methods, Analysis &Insights from Training Gopher

reported in the most recent version of each paper.
Details about the SST-5 Fine-Grained benchmark can
be found in the SST paper.
To highlight progress on SST-5 Fine-Grained accuracy,
scores were taken from the following papers:
An Algorithm for Routing Capsules in All Domains
An Algorithm for Routing Vectors in Sequences
Improved Semantic Representations from TreeStructured Long Short-Term Memory Networks

Number of Commercially
Available MT Systems
Details about the number of commercially available
MT systems were sourced from the Intento report The
State of Machine Translation, 2022. Intento is a San
Francisco–based startup that analyzes commerciallyavailable MT services.

Improved Sentence Modeling Using Suffix
Bidirectional LSTM

VoxCeleb

Learned in Translation: Contextualized Word Vectors

Data on VoxCeleb equal error rate (EER) was retrieved

Less Grammar, More Features
Recursive Deep Models for Semantic Compositionality
Over a Sentiment Treebank
Self-Explaining Structures Improve NLP Models

from the VoxCeleb Speaker Recognition Challenge(VoxSRC).
For the sake of consistency, the AI Index reported scoreson the initial VoxCeleb dataset. Specifically, the AI Index
made use of the following sources of information:

MMLU
Data on MMLU accuracy was retrieved through adetailed arXiv literature review cross-referenced bytechnical progress reported on Papers With Code. The
reported dates correspond to the year in which a paper
was first published to arXiv, and the reported results(accuracy) correspond to the result reported in the mostrecent version of each paper. Details about the MMLU
benchmark can be found in the MMLU paper.

ID R&D System Description to VoxCeleb Speaker
Recognition Challenge 2022
The IDLAB VoXSRC-20 Submission: Large MarginFine-Tuning and Quality-Aware Score Calibration in
DNN Based Speaker Verification
The SpeakIn System for VoxCeleb Speaker
Recognition Challenge 2021
VoxCeleb: A Large-Scale Speaker Identification
Dataset
VoxCeleb: Large-Scale Speaker Verification in the Wild
VoxCeleb2: Deep Speaker Recognition

Table of Contents

Appendix360

Artificial IntelligenceIndex Report 2023

Whisper
Data on Whisper for large-scale speech recognition
models was retrieved from the paper Radford et al.,2022.

Appendix
Chapter 2: Technical Performance

Training Time, Numberof Accelerators, andPerformance
Data on training time, number of accelerators,

Procgenand performance for AI systems was taken from

Data on Procgen mean-normalized score was retrieved
through a detailed arXiv literature review. The reported
dates correspond to the year in which a paper was firstpublished to arXiv, and the reported results (meannormalized score) correspond to the result reported in
the most recent version of each paper. Details on the
Procgen benchmark can be found in the Procgen paper.

the MLPerf Training and Inference benchmark
competitions. Details on the MLPerf Training
benchmark can be found in the MLPerf Training
Benchmark paper, while details on MLPerf Inference
can be found in the MLPerf Inference Benchmark
paper. Information about the current benchmark
categories as well as technical information about
submission and competition subdivisions can be

To highlight progress on Procgen, scores were taken

found on the MLPerf Training and MLPerf Inference

from the following papers:

webpages.

Automatic Data Augmentation for Generalization in
Reinforcement Learning

The AI Index made use of data from the following

Leveraging Procedural Generation to Benchmark
Reinforcement Learning

MLPerf Training v2.1, 2022

Procedural Generalization by Planning With
Self-Supervised World Models

MLPerf Training v1.1, 2021

Rethinking Value Function Learning for
Generalization in Reinforcement Learning

MLPerf Training competitions:
MLPerf Training v2.0, 2022
MLPerf Training v1.0, 2021
MLPerf Training v0.7, 2020
MLPerf Training v0.6, 2019
MLPerf Training v0.5, 2018
The AI Index made use of data from the following
MLPerf Inference competitions:
MLPerf Inference v2.1, 2022
MLPerf Inference v2.0, 2022
MLPerf Inference v1.1, 2021
MLPerf Inference v1.0, 2021
MLPerf Inference v0.7, 2020

Table of Contents

Appendix361

Artificial IntelligenceIndex Report 2023

GPUs’ Performance and PriceAppendix
Chapter 2: Technical Performance

The AI Index collected data on GPUs’ performance andCarbon Footprint of SelectMachine Learning Models

price, building on and extending the dataset collected

Data on carbon-emission estimates of selectfrom Epoch AI’s Trends in GPU Price-Performance
blog post.

machine learning models was sourced from thepaper Luccioni et al., 2022. Data on carbon-emission

The AI Index compiled a list of GPUs starting fromestimates of real-life examples was retrieved from

the Median Group (2018), Sun et al. (2019), and Epoch

Strubell et al., 2019.

(2022) datasets. To update and extend previous
for the period 2021–2023, gathering information

Energy Savings Results
From BCOOLER Experimentfrom sources such as TechPowerUp, WikiChip, andData on energy savings over time for the

Wikipedia entries for the product series. We also

BCOOLER experiment was sourced from thecollected information about GPUs released before

paper Luo et al., 2022.

analysis, the AI Index included new GPU releases

2021 from the manufacturer’s catalog or Wikipedia’slist of processors.
To disambiguate duplicates of different versions of
the same product with different specifications, theAI Index added the part number or difference in
specification, as applicable.
To find GPU prices, the AI Index searched various
sources including the manufacturer’s website,
Wikipedia, and TechPowerUp. GPU prices have beenadjusted for inflation using CPI-U data provided by
the U.S. Bureau of Labor Statistics. Missing data forcertain GPUs was completed using additional sources,
such as the manufacturer’s website, Wikipedia,
and TechPowerUp. This includes information suchas manufacturer, type, release date, performance(double, single, and half-precision operations per
second), die size, power, clock speed, process size,
and number of transistors.

Table of Contents

Appendix362

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 3: Technical AI Ethics

Chapter 3: Technical AI Ethics
Meta-Analysis of Fairness
and Bias Metrics
For the analysis conducted on fairness and bias
metrics in AI, we identify and report on benchmark
and diagnostic metrics which have been consistently
cited in the academic community, reported on a publicleaderboard, or reported for publicly available baselinemodels (e.g., GPT-3, BERT, ALBERT). We note that
research paper citations are a lagging indicator of
adoption, and metrics which have been very recently
adopted may not be reflected in the data for 2022. We
include the full list of papers considered in the 2022 AI
Index as well as the following additional papers:
Beyond the Imitation Game: Quantifying andExtrapolating the Capabilities of Language Models
BBQ: A Hand-Built Bias Benchmark for
Question Answering

VLStereoSet: A Study of Stereotypical Bias
in Pre-trained Vision-Language Models

Natural Language Processing
Bias Metrics
In Section 3.3, we track citations of the Perspective
API created by Jigsaw at Google. The Perspective API
has been adopted widely by researchers and engineers
in natural language processing. Its creators define
toxicity as “a rude, disrespectful, or unreasonable
comment that is likely to make someone leave a
discussion,” and the tool is powered by machinelearning models trained on a proprietary dataset ofcomments from Wikipedia and news websites.
We include the full list of papers considered in the2022 AI Index as well as the following additional
papers:Discovering Language Model Behaviors With
Model-Written Evaluations

AlexaTM 20B: Few-Shot Learning Using aLarge-Scale Multilingual Seq2Seq Model“I’m Sorry to Hear That”: Finding New Biases in
Language Models With a Holistic Descriptor Dataset

Aligning Generative Language Models With
Human Values

On Measuring Social Biases in Prompt-BasedMulti-task Learning

Challenges in Measuring Bias via Open-Ended
Language Generation

PaLM: Scaling Language Modeling With Pathways

Characteristics of Harmful Text: Towards
Rigorous Benchmarking of Language Models

Perturbation Augmentation for Fairer NLP
Scaling Instruction-Finetuned Language Models
SODAPOP: Open-Ended Discovery of Social
Biases in Social Commonsense Reasoning Models
Towards Robust NLG Bias Evaluation
With Syntactically-Diverse Prompts

Table of Contents

Appendix

Controllable Natural Language Generation With
Contrastive Prefixes
DD-TIG at SemEval-2022 Task 5: Investigating theRelationships Between Multimodal and Unimodal
Information in Misogynous Memes Detection andClassification

363

Artificial IntelligenceIndex Report 2023

Detoxifying Language Models With a Toxic Corpus
DisCup: Discriminator Cooperative Unlikelihood
Prompt-Tuning for Controllable Text Generation
Evaluating Attribution in Dialogue Systems:
The BEGIN Benchmark

Appendix
Chapter 3: Technical AI Ethics

Predictability and Surprise in Large Generative
ModelsQuark: Controllable Text Generation With
Reinforced [Un]learning
Red Teaming Language Models With Language Models

Exploring the Limits of Domain-Adaptive Training
for Detoxifying Large-Scale Language Models

Reward Modeling for Mitigating Toxicity inTransformer-based Language Models

Flamingo: A Visual Language Model for
Few-Shot Learning

Robust Conversational Agents Against Imperceptible
Toxicity Triggers

Galactica: A Large Language Model for Science

Scaling Instruction-Finetuned Language Models

GLaM: Efficient Scaling of Language Models
With Mixture-of-Experts

StreamingQA: A Benchmark for Adaptation to New
Knowledge over Time in Question Answering Models

GLM-130B: An Open Bilingual Pre-trained Model

Training Language Models to Follow Instructions
With Human Feedback

Gradient-Based Constrained Sampling From
Language Models
HateCheckHIn: Evaluating Hindi Hate Speech
Detection Models
Holistic Evaluation of Language Models

Transfer Learning From Multilingual DeBERTafor Sexism Identification
Transformer Feed-Forward Layers Build Predictions
by Promoting Concepts in the Vocabulary Space

An Invariant Learning Characterization ofControlled Text Generation

While the Perspective API is used widely within

LaMDA: Language Models for Dialog Applications

online toxicity, toxicity in the specific domains used to

Leashing the Inner Demons: Self-Detoxification
for Language Models

train the models undergirding Perspective (e.g., news,

Measuring Harmful Representations in Scandinavian
Language Models

forms of toxicity (e.g., trolling). Other known caveats

Mitigating Toxic Degeneration With Empathetic Data:
Exploring the Relationship Between Toxicity andEmpathy

voices: The Perspective API has been shown to

MULTILINGUAL HATECHECK: Functional Tests forMultilingual Hate Speech Detection Models

am a gay man”). As a result, detoxification techniques

A New Generation of Perspective API: EfficientMultilingual Character-Level Transformers

result in models that are less capable of modeling

OPT: Open Pre-trained Transformer Language Models

mentioning minority identities.

PaLM: Scaling Language Modeling With Pathways

New versions of the Perspective API have been

Perturbations in the Wild: Leveraging Human-Written
Text Perturbations for Realistic Adversarial Attack andDefensedeployed since its inception, and there may be subtle

Table of Contents

Appendix

machine learning research and also for measuring

Wikipedia) may not be broadly representative of all
include biases against text written by minority
disproportionately assign high toxicity scores to textthat contains mentions of minority identities (e.g., “I
built with labels sourced from the Perspective API
language used by minority groups, and may avoid

undocumented shifts in its behavior over time.

364

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 3: Technical AI Ethics

RealToxicityPrompts

We tally the number of papers in each category to

We sourced the RealToxicityPrompts dataset ofare not double-counted in multiple categories. We

evaluations from the HELM benchmark website, as
documented in v0.1.0.

reach the numbers found in Figure 3.7.3. Papers
note that this data may not be as accurate for data
pre-2018 as societal impacts work at NeurIPS has

AI Ethics in China

historically been categorized under a broad “AI for

The data in this section is sourced from the 2022 paper

into more granular research areas. Examples include

AI Ethics With Chinese Characteristics? Concerns

workshops dedicated to machine learning for health;and Preferred Solutions in Chinese Academia. We

climate; policy and governance; disaster response;are grateful to Junhua Zhu for clarifications andand the developing world.

correspondence.

To track trends around specific technical topics at

AI Ethics Trends at FAccTand NeurIPS

NeurIPS as in Figures 3.7.4 to 3.7.7, we count the

To understand trends at the ACM Conference on
Fairness, Accountability, and Transparency, thissection tracks FAccT papers published in conference
proceedings from 2018 to 2022. We categorizesocial impact” umbrella, but it has recently been split

number of papers accepted to the NeurIPS main trackwith titles containing keywords (e.g., “counterfactual”
or “causal” for tracking papers related to causal
effect), as well as papers submitted to related
workshops.

author affiliations into academic, industry, nonprofit,

TruthfulQA

government, and independent categories, while also

We sourced the TruthfulQA dataset of evaluations

tracking the location of their affiliated institution.

from the HELM benchmark website, as documented

Authors with multiple affiliations are counted once inin v0.1.0.

each category (academic and industry), but multiple
affiliations of the same type (i.e., authors belonging
to two academic institutions) are counted once in the
category.
For the analysis conducted on NeurIPS publications,
we identify workshops themed around real-world
impact and label papers with a single main category in“healthcare,” “climate,” “finance,” “developing world,”“science,” or “other,” where “other” denotes a paper
related to a real-world use case but not in one of theother categories. The “science” category is new in
2022, but includes retroactive analysis of papers fromprevious years.

Table of Contents

Appendix365Artificial IntelligenceIndex Report 2023

Appendix
Chapter 4: The Economy

Chapter 4: The Economy
Lightcastjob postings is the Job Openings and Labor TurnoverPrepared by Scott Bingham, Julia Nania, Layla O’Kane,of Labor Statistics. Based on comparisons between

and Bledi Taska

Survey (JOLTS) program, conducted by the Bureau
JOLTS and Lightcast, the labor market demandLightcast delivers job market analytics that empower

captured by Lightcast data represents over 99% of

employers, workers, and educators to make data-

the total labor demand. Jobs not posted online are

driven decisions. The company’s artificial intelligence

usually in small businesses (the classic example being

technology analyzes hundreds of millions of job postings

the “Help Wanted” sign in a restaurant window) andand real-life career transitions to provide insightunion hiring halls.

into labor market patterns. This real-time strategic

Measuring Demand for AI

intelligence offers crucial insights, such as what jobs are

In order to measure the demand by employers of AI

most in demand, the specific skills employers need, andskills, Lightcast uses its skills taxonomy of over 31,000

the career directions that offer the highest potential forskills. The list of AI skills from Lightcast data are shownworkers. For more information, visit www.lightcast.io.

below, with associated skill clusters. While some skills

Job Posting Data

are considered to be in the AI cluster specifically,

To support these analyses, Lightcast mined its dataset

for the purposes of this report, all skills below wereof millions of job postings collected since 2010.

considered AI skills. A job posting was considered an

Lightcast collects postings from over 51,000 online job

AI job if it mentioned any of these skills in the job text.

sites to develop a comprehensive, real-time portrait

Artificial Intelligence: AIOps (Artificial Intelligence forof labor market demand. It aggregates job postings,
removes duplicates, and extracts data from job postings
text. This includes information on job title, employer,industry, and region, as well as required experience,education, and skills.

IT Operations), Applications of Artificial Intelligence,
Artificial General Intelligence, Artificial Intelligence,
Artificial Intelligence Development, Artificial
Intelligence Markup Language (AIML), Artificial
Intelligence Systems, Azure Cognitive Services,

Job postings are useful for understanding trends in

Baidu, Cognitive Automation, Cognitive Computing,the labor market because they allow for a detailed,Computational Intelligence, Cortana, Expert Systems,

real-time look at the skills employers seek. To assessIntelligent Control, Intelligent Systems, Interactive

the representativeness of job postings data, Lightcast

Kiosk, IPSoft Amelia, Knowledge-Based Configuration,

conducts a number of analyses to compare theKnowledge-Based Systems, Multi-Agent Systems,

distribution of job postings to the distribution of official

Open Neural Network Exchange (ONNX), OpenAI

government and other third-party sources in the UnitedGym, Reasoning Systems, Soft Computing, Syman,

States. The primary source of government data on U.S.

Watson Conversation, Watson Studio, Weka

Table of Contents

Appendix366

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 4: The Economy

Autonomous Driving: Advanced Driver Assistance

Machine Learning: AdaBoost, Apache MADlib,

Systems, Autonomous Cruise Control Systems,

Apache Mahout, Apache SINGA, Apache Spark,

Autonomous System, Autonomous Vehicles, Guidance

Association Rule Learning, Automated Machine

Navigation and Control Systems, Light Detection andLearning, Autonomic Computing, AWS SageMaker,

Ranging (LiDAR), OpenCV, Path Analysis, Path Finding,Azure Machine Learning, Boosting, CHi-Squared

Remote Sensing, Unmanned Aerial Systems (UAS)

Automatic Interaction Detection (CHAID),

Natural Language Processing (NLP): Amazon Textract,
ANTLR, BERT (NLP Model), Chatbot, Computational
Linguistics, DeepSpeech, Dialog Systems, fastText,
Fuzzy Logic, Handwriting Recognition, Hugging
Face (NLP Framework), HuggingFace Transformers,
Intelligent Agent, Intelligent Software Assistant,Intelligent Virtual Assistant, Kaldi, Latent Dirichlet
Allocation, Lexalytics, Machine Translation, Microsoft
LUIS, Natural Language Generation, Natural Language
Processing, Natural Language Processing Systems,
Natural Language Programming, Natural Language
Toolkits, Natural Language Understanding, Natural
Language User Interface, Nearest Neighbour
Algorithm, OpenNLP, Optical Character Recognition(OCR), Screen Reader, Semantic Analysis, Semantic
Interpretation for Speech Recognition, Semantic
Parsing, Semantic Search, Sentiment Analysis,
Seq2Seq, Speech Recognition, Speech Recognition
Software, Statistical Language Acquisition, Text Mining,
Tokenization, Voice Interaction, Voice User Interface,
Word Embedding, Word2Vec Models

Classification And Regression Tree (CART), Cluster
Analysis, Collaborative Filtering, Confusion Matrix,
Cyber-Physical Systems, Dask (Software), DataClassification, DBSCAN, Decision Models, Decision
Tree Learning, Dimensionality Reduction, Dlib
(C++ Library), Ensemble Methods, Evolutionary
Programming, Expectation Maximization Algorithm,
Feature Engineering, Feature Extraction, Feature
Learning, Feature Selection, Gaussian Process,
Genetic Algorithm, Google AutoML, Google Cloud
ML Engine, Gradient Boosting, H2O.ai, Hidden
Markov Model, Hyperparameter Optimization,
Inference Engine, K-Means Clustering, Kernel
Methods, Kubeflow, LIBSVM, Machine Learning,
Machine Learning Algorithms, Markov Chain, Matrix
Factorization, Meta Learning, Microsoft Cognitive
Toolkit (CNTK), MLflow, MLOps (Machine Learning
Operations), mlpack (C++ Library), Naive Bayes,
Perceptron, Predictionio, PyTorch (Machine Learning
Library), Random Forest Algorithm, Recommendation
Engine, Recommender Systems, Reinforcement
Learning, Scikit-learn (Machine Learning Library),

Neural Networks: Apache MXNet, Artificial NeuralSemi-Supervised Learning, Soft Computing, Sorting

Networks, Autoencoders, Caffe, Caffe2, Chainer,

Algorithm, Supervised Learning, Support Vector

Convolutional Neural Networks, Cudnn, Deep Learning,

Machine, Test Datasets, Torch (Machine Learning),

Deeplearning4j, Keras (Neural Network Library), Long

Training Datasets, Transfer Learning, Unsupervised

Short-Term Memory (LSTM), OpenVINO, PaddlePaddle,

Learning, Vowpal Wabbit, Xgboost

Pybrain, Recurrent Neural Network (RNN), TensorFlow

Table of Contents

Appendix367

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 4: The Economy

Robotics: Advanced Robotics, Cognitive Robotics,

intelligence, computer vision, image processing,

Motion Planning, Nvidia Jetson, Robot Framework,

deep learning, TensorFlow, Pandas (software), andRobot Operating Systems, Robotic Automation

OpenCV, among others.

Software, Robotic Liquid Handling Systems, Robotic
Programming, Robotic Systems, Servomotor, SLAM
Algorithms (Simultaneous Localization and Mapping)

Skill groupings are derived by expert taxonomists
through a similarity-index methodology thatmeasures skill composition at the industry level.

Visual Image Recognition: 3D Reconstruction, Activity

LinkedIn’s industry taxonomy and their corresponding

Recognition, Computer Vision, Contextual Image

NAICS codes can be found here.

Classification, Digital Image Processing, Eye Tracking,

Skills Genome

Face Detection, Facial Recognition, Image Analysis,Image Matching, Image Processing, Image Recognition,
Image Segmentation, Image Sensor, Imagenet,
Machine Vision, Motion Analysis, Object Recognition,
OmniPage, Pose Estimation, RealSense

For any entity (occupation or job, country, sector,etc.), the skill genome is an ordered list (a vector) of
the 50 “most characteristic skills” of that entity. Thesemost characteristic skills are identified using a TF-IDF
algorithm to identify the most representative skills ofLinkedIn

the target entity, while down-ranking ubiquitous skills

Prepared by Murat Erer and Akash Kaura(e.g., Microsoft Word).

Country Sample

TF-IDF is a statistical measure that evaluates how

Included countries represent a select sample of eligible

representative a word (in this case a skill) is to a

countries with at least 40% labor force coverage by

selected entity). This is done by multiplying two

LinkedIn and at least 10 AI hires in any given month.

metrics:

that add little information about that specific entity

China and India were included in this sample becauseof their increasing importance in the global economy,
but LinkedIn coverage in these countries does not

1. The term frequency of a skill in an entity (TF).
	2. The logarithmic inverse entity frequency of thereach 40% of the workforce. Insights for these countries

skill across a set of entities (IDF). This indicates

may not provide as full a picture as other countries, andhow common or rare a word is in the entire entity

should be interpreted accordingly.

set. The closer IDF is to 0, the more common theword.

Skills (and AI Skills)
LinkedIn members self-report their skills on their

So if the skill is very common across LinkedIn entities,

LinkedIn profiles. Currently, more than 38,000 distinct,

and appears in many job or member descriptions, thestandardized skills are identified by LinkedIn. These

IDF will approach 0. If, on the other hand, the skill

have been coded and classified by taxonomists at

is unique to specific entities, the IDF will approach

LinkedIn into 249 skill groupings, which are the skill

1. More details are available at LinkedIn’s Skills

groups represented in the dataset. The top skills that

Genome and LinkedIn-World Bank Methodology.

make up the AI skill grouping are machine learning,
natural language processing, data structures, artificial
Table of Contents

Appendix368

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 4: The Economy

AI Skills Penetration

perform the job. Skills penetration is used as a signal

The aim of this indicator is to measure the intensity of AI

for whether AI skills are prevalent in an occupation

skills in an entity (a particular country, industry, gender,

representative in any sector where the occupationetc.) through the following methodology:

representative may exist. Examples of such

• Compute frequencies for all self-added skills by
LinkedIn members in a given entity (occupation,industry, etc.) in 2015–2021.
• Re-weight skill frequencies using a TF-IDF model
to get the top 50 most representative skills inthat entity. These 50 skills compose the “skill
genome” of that entity.
• Compute the share of skills that belong to theoccupations include (but are not limited to): machinelearning engineer, artificial intelligence specialist,
data scientist, computer vision engineer, etc.

AI Talent
A LinkedIn member is considered AI talent if they
have explicitly added AI skills to their profile and/or
they are occupied in an AI occupation representative.
The counts of AI talent are used to calculate talentconcentration metrics. For example, to calculate

AI skill group out of the top skills in the selected

the country level AI talent concentration, we use

entity.

the counts of AI talent at the country level vis-a-vis

Interpretation: The AI skill penetration rate signals
the prevalence of AI skills across occupations, or thethe counts of LinkedIn members in the respective
countries.

intensity with which LinkedIn members utilize AI skills

Relative AI Skills Penetrationin their jobs. For example, the top 50 skills for the

To allow for skills penetration comparisons acrossoccupation of engineer are calculated based on the

countries, the skills genomes are calculated and aweighted frequency with which they appear in LinkedIn

relevant benchmark is selected (e.g., global average).

members’ profiles. If four of the skills that engineers

A ratio is then constructed between a country’s andpossess belong to the AI skill group, this measure

the benchmark’s AI skills penetrations, controlling forindicates that the penetration of AI skills is estimated tooccupations.

be 8% among engineers (i.e., 4/50).

Interpretation: A country’s relative AI skills

Jobs or Occupations

penetration of 1.5 indicates that AI skills are 1.5 times

LinkedIn member titles are standardized and grouped

as frequent as in the benchmark, for an overlapping

into approximately 15,000 occupations. These areset of occupations.

not sector- or country-specific. These occupations

Global Comparison

are further standardized into approximately
3,600 occupation representatives. Occupation
representatives group occupations with a common roleand specialty, regardless of seniority.

For cross-country comparison, we present therelative penetration rate of AI skills, measured as
the sum of the penetration of each AI skill acrossoccupations in a given country, divided by the

AI Jobs and Occupations

average global penetration of AI skills across the

An “AI” job (technically, occupation representative)

overlapping occupations in a sample of countries.

is an occupation representative that requires AI skills toTable of Contents

Appendix369

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 4: The Economy

Interpretation: A relative penetration rate of 2 means

Interpretation: A country’s “Relative AI Skills

that the average penetration of AI skills in that country

Penetration Across Genders” for women of 1.5 means

is two times the global average across the same set ofthat female members in that country are 1.5 times

occupations.

more likely to list AI skills than the average member in

Global Comparison: By Industry

all countries pooled together across the same set of

The relative AI skills penetration by country for industry

occupations that exist in the country.

provides an in-depth sectoral decomposition of AI skill

Relative AI Hiring Index

penetration across industries and sample countries.

	
LinkedIn Hiring Rate or Overall Hiring Rate

Interpretation: A country’s relative AI skill penetration
rate of 2 in the education sector means that the average
penetration of AI skills in that country is two times theglobal average across the same set of occupations inthat sector.

Global Comparison: By Gender
The “Relative AI Skills Penetration by Gender” metricprovides a cross-country comparison of AI skill
penetrations within each gender, comparing countries’
male or female AI skill penetrations to the global

is a measure of hires normalized by LinkedIn
membership. It is computed as the percentage of
LinkedIn members who added a new employer
in the same period the job began, divided by
the total number of LinkedIn members in thecorresponding location.
AI Hiring Rate is computed following the overall
hiring rate methodology, but only considering
members classified as AI talent.
	
Relative AI Hiring Index is the pace of change inaverage of the same gender. Since the global averages

AI Hiring Rate normalized by the pace of change

are distinct for each gender, this metric should only bein Overall Hiring Rate, providing a picture ofused to compare country rankings within each gender,whether hiring of AI talent is growing at a higher,

and not for cross-gender comparisons within countries.

equal, or lower rate than overall hiring in a

Interpretation: A country’s AI skills penetration for
women of 1.5 means that female members in thatcountry are 1.5 times more likely to list AI skills than themarket. The relative AI Hiring Index is equal to
1.0 when AI hiring and overall hiring are growing
at the same rate year on year.

average female member in all countries pooled together

Interpretation: Relative AI Hiring Index shows how

across the same set of occupations that exist in thefast each country is experiencing growth in AI talentcountry/gender combination.

hiring relative to growth in overall hiring in the country.

Global Comparison: Across Gender

A ratio of 1.2 means the growth in AI talent hiring has

The “Relative AI Skills Penetration Across Genders”

outpaced the growth in overall hiring by 20%.

metric allows for cross-gender comparisons within
and across countries globally, since we compare thecountries’ male and female AI skill penetrations to thesame global average regardless of gender.

Table of Contents

Appendix370

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 4: The Economy

Changelog From Methodology Included in Last Year’s AI Index	1. LinkedIn ramped a new version of its industry taxonomy (see details here).
		a. This has resulted in changes to our top level five key industries. We have made the full-time series
available for each industry (as with prior years).
			i. “Software & IT Services” industry evolved into a wider “Technology, Information and Media,” which
encompasses media and telecommunications as well as other sub-industries.
			ii. Former “Hardware & Networking” industry does not exist in the new taxonomy, so we introduced“Professional Services” industry as the fifth industry in scope which contains a high concentration of AI
talent.
			iii. Remaining “Education,” “Manufacturing,” and “Financial Services” (formerly known as “Finance”)
also had updates in their coverage resulting from the inclusion of more granular sub-industries.
		b. This also resulted in minor changes in magnitudes for some metrics since the distinct number ofindustries, as well as the distinct number of AI occupations defined within each country-industry pair havechanged:			i. We define AI occupations (occupation representatives that require AI skills to perform the job) andthe respective definition of AI Talent at Country-Industry level. For example, data engineers working
in the technology, information, and media industry in Germany may be identified as holding an AI
occupation, whereas data engineers working in the construction industry in the United Arab Emirates
may not be identified as AI Talent. Following the introduction of a more granular industry taxonomy
with improved accuracy, our AI Talent identifications have been improved, and results have beenreflected to the entirety of time series for each relevant metric.
			ii. The following metrics have been impacted by this change in industry taxonomy: AI Talent
Concentrations, and Relative AI Hiring Rates. No directional changes were observed, only minorchanges in magnitudes.
	2. We introduced a methodology change into Relative Skills Penetration metrics:
		a. In the past, the data used to calculate these metrics were limited to top five industries with the highest
AI skill penetration globally: “Software & IT Services,” “Hardware & Networking,” “Manufacturing,”“Education,” and “Finance” industries. This year we updated our coverage to all industries.

Table of Contents

Appendix371

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 4: The Economy

NetBase Quid

subsidiary, out of business) throughout the world.

Prepared by Bill Valle and Nicole Seredenko

M&A, public offerings, minority stakes made by PE/

The investment data includes private investments,

NetBase Quid delivers AI-powered consumer and

VCs, corporate venture arms, governments, andmarket intelligence to enable business reinvention

institutions both within and outside the United States.

in a noisy and unpredictable world. The software

Some data is simply unreachable—for instance, whenapplies artificial intelligence to reveal patterns in large,

investors’ names or funding amounts are undisclosed.

unstructured datasets and to generate visualizations

NetBase Quid embeds Capital IQ data as a default

that enable users to make smart, data-driven decisions
accurately, quickly, and efficiently. NetBase Quid uses
Boolean query to search for focus areas, topics, andkeywords within social media, news, forums and blogs,
companies, and patents data sources, as well as othercustom datasets. NetBase Quid then visualizes thesedata points based on the semantic similarity.

Search, Data Sources, and Scope
Over 8 million global public and private company profiles
from multiple data sources are indexed in order tosearch across company descriptions, while filtering andincluding metadata ranging from investment information
to firmographic information, such as founded year, HQ
location, and more. Company information is updated
on a weekly basis. The NetBase Quid algorithm reads
a big amount of text data from each document tomake links between different documents based on
their similar language. This process is repeated at animmense scale, which produces a network with different
clusters identifying distinct topics or focus areas. Trends
are identified based on keywords, phrases, people,
companies, and institutions that NetBase Quid identifies,
and the other metadata that is put into the software.

and adds in data from Crunchbase for the data
points that are not captured in Capital IQ. This not
only yields comprehensive and accurate data on
all global organizations, but it also captures earlystage startups and funding events data. Company
information is updated on a weekly basis.
Earnings Calls
NetBase Quid leverages earnings call transcript
data embedded from Seeking Alpha. For this report,
NetBase Quid has analyzed mentions of AI-related
keywords across all earnings call transcripts from
Fortune 500 companies from January 2018 throughDecember 2022. New earnings call transcript data is
updated in NetBase Quid on the 1st and 15th of every
month.

Search Parameters
Boolean query is used to search for focus areas,
topics, and keywords within the archived company
database, within their business descriptions andwebsites. We can filter out the search results by
HQ regions, investment amount, operating status,organization type (private/public), and founding
year. NetBase Quid then visualizes these companies

Databy semantic similarity. If there are more than 7,000

Companies

companies from the search result, NetBase Quid

Organization data is embedded from Capital IQ andselects the 7,000 most relevant companies for

Crunchbase. These companies include all types ofvisualization based on the language algorithm.

companies (private, public, operating, operating as a

Table of Contents

Appendix372

Artificial IntelligenceIndex Report 2023

Boolean Search: “artificial intelligence” or “AI” or“machine learning” or “deep learning”
Companies:

Appendix
Chapter 4: The Economy

McKinsey & Company
Data used in the Corporate Activity-Industry
Adoption section was sourced from the McKinsey• Global AI and ML companies that have received
investments (private, IPO, M&A) from January 1,
2013, to December 31, 2022.
• Global AI and ML companies that have received

Global Survey “The State of AI in 2022—and a Half
Decade in Review.”
The online survey was in the field from May 3, 2022,
to May 27, 2022, and from August 15, 2022, toover $1.5M for the last 10 years (January 1, 2013,

August 17, 2022, and garnered responses from 1,492

to December 31, 2022): 7,000 out of 7,500

participants representing a full range of regions,companies have been selected through NetBase

industries, company sizes, functional specialties,

Quid’s relevance algorithm.

and tenures. Of those respondents, 744 said theirorganization had adopted AI in at least one function

Target Event Definitions

and were asked questions about their organization’s• Private investments: A private placement is a

AI use. To adjust for differences in response rates,

private sale of newly issued securities (equity or

the data is weighted by the contribution of eachdebt) by a company to a selected investor or a

respondent’s nation to global GDP.

selected group of investors. The stakes that buyers
take in private placements are often minority
stakes (under 50%), although it is possible to takecontrol of a company through a private placement

The AI Index also considered data from previous
iterations of the survey. More specifically, the AI
index made use of data from:as well, in which case the private placement would

The State of AI in 2021

be a majority stake investment.

The State of AI in 2020

• Minority investment: These refer to minority
stake acquisitions in NetBase Quid, which takeplace when the buyer acquires less than 50% of
the existing ownership stake in entities, asset

Global AI Survey: AI Proves Its Worth,
But Few Scale Impact (2019)
AI Adoption Advances, But Foundational Barriers
Remain (2018)

products, and business divisions.
• M&A: This refers to a buyer acquiring more than
50% of the existing ownership stake in entities,
asset products, and business divisions.

Table of Contents

Appendix373

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 4: The Economy

GitHub

To obtain a global view of how AI is transforming

Data on the effects of GitHub’s Copilot on developerbusiness leaders between April 2022 and May 2022.

productivity and happiness was sourced from theGitHub Copilot Survey conducted in 2022.

organizations, Deloitte surveyed 2,620 global
Thirteen countries were represented: Australia (100respondents), Brazil (115 respondents), Canada (175The survey was emailed to 17,420 users who had opted

respondents), China (200 respondents), France (130in to receive communications and were using GitHub

respondents), Germany (150 respondents), India

Copilot for their daily programming activities. Between(200 respondents), Israel (75 respondents), Japan

February 10, 2022, and March 6, 2022, the authors(100 respondents), Singapore (100 respondents),

received 2,047 responses that could be matched with

South Africa (75 respondents), the United Kingdom

usage measurements during the four-week period(200 respondents), and the United States (1,000

leading up to March 12, 2022. The survey contained

respondents). All participating companies havemultiple-choice questions on demographic information

adopted AI technologies and are AI users.

and Likert-type questions on different aspects of

Respondents were required to meet one of the

productivity, which were randomized in the order offollowing criteria: responsible for AI technology

appearance to the user.

spending or approval of AI investments, developing

More details can be found in Ziegler at al., 2022.

AI technology strategies, managing or overseeing
AI technology implementation, serving as an AI

Deloitte

technology subject matter specialist, or making

Data used in the Corporate Activity-Industry Motivation

complement the blind survey, Deloitte conducted

section was sourced from Deloitte’s “State of AI in thequalitative telephone interviews with 15 AI specialists

Enterprise” surveys.

from various industries. More details are available on

or influencing decisions around AI technology. To

More specifically, the AI Index made use of the following
sources of information:
Deloitte’s State of AI in the Enterprise,
5th Edition Report (2022)

Deloitte’s website.

International Federation of
Robotics (IFR)

State of AI in the Enterprise, 4th Edition (2021)

Data presented in the Robot Installations section was

Deloitte’s State of AI in the Enterprise, 3rd Edition (2020)

sourced from the “World Robotics 2022” report.

State of AI in the Enterprise, 2nd Edition (2018)
The 2017 Deloitte State of Cognitive Survey (2017)

Table of Contents

Appendix374

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 5: Education

Chapter 5: Education
Computing Research
Association (CRA Taulbee
Survey)Note: This year’s AI Index reused the methodologicalnotes that were submitted by the CRA for previous
editions of the AI Index. For more complete delineationsof the methodology used by the CRA, please consult theindividual CRA surveys that are linked below.
Computing Research Association (CRA) members

The CRA Taulbee Survey is sent only to doctoral
departments of computer science, computerengineering, and information science/systems.
Historically, (a) Taulbee covers one-quarter to onethird of total BS CS recipients in the United States;(b) the percent of women earning bachelor’s degrees
is lower in the Taulbee schools than overall; and (c)Taulbee tracks the trends in overall CS production.
The AI Index used data from the following iterationsof the CRA survey:

are 200-plus North American organizations active in

CRA, 2021

computing research: academic departments of computer

CRA, 2020

science and computer engineering; laboratories and

CRA, 2019

centers in industry, government, and academia; andaffiliated professional societies (AAAI, ACM, CACS/
AIC, IEEE Computer Society, SIAM USENIX). CRA’s
mission is to enhance innovation by joining with industry,

CRA, 2018
CRA, 2017
CRA, 2016

government, and academia to strengthen research and

CRA, 2015

advanced education in computing. Learn more about

CRA, 2014

CRA here.

CRA, 2013

The CRA Taulbee Survey gathers survey data during thefall of each academic year by reaching out to over 200

CRA, 2012
CRA, 2011

PhD-granting departments. Details about the Taulbee
Survey can be found here. Taulbee doesn’t directly
survey the students. The department identifies each
new PhD’s area of specialization as well as their typeof employment. Data is collected from September toJanuary of each academic year for PhDs awarded in theprevious academic year. Results are published in May
after data collection closes.

Table of Contents

Appendix375

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 5: Education

Code.org
State Level Data
The following link includes a full description of themethodology used by Code.org to collect its data. The
staff at Code.org also maintains a database of the stateof American K–12 education and, in this policy primer,
provides a greater amount of detail on the state ofAmerican K–12 education in each state.

AP Computer Science Data
The AP computer science data is provided to Code.org
as per an agreement the College Board maintains withCode.org. The AP Computer Science data comes fromthe college board’s national and state summary reports.

The State of International
K–12 Education
Data on the state of international K–12 AI education was
taken from the following UNESCO report, published in
2021. The methodology is outlined in greater detail on
pages 18 to 20 in the report and, for the sake of brevity, is
not completely reproduced in the 2023 AI index.

Table of Contents

Appendix376

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 6: Policy and Governance

Chapter 6: Policy and Governance
Global Legislation Records on AI
For AI-related bills passed into laws, the AI Index performed searches of the keyword “artificial intelligence”
on the websites of 127 countries’ congresses or parliaments (in the respective languages) in the full text of bills.
Note that only laws passed by state-level legislative bodies and signed into law (i.e., by presidents or throughroyal assent) from 2016 to 2022 are included. Laws that were approved but then repealed are not included in theanalysis. In some cases, there were databases that were only searchable by title, so site search functions were
deployed. Future AI Index reports hope to include analysis on other types of legal documents, such as regulations
and standards, adopted by state- or supranational-level legislative bodies, government agencies, etc. The AI Index
team surveyed the following databases:
Algeria

China

India

MonacoSlovenia

AndorraColombia

Iran, Islamic Republic

Montenegro

South Africa

Antigua and Barbuda

Croatia

Iraq

MoroccoSpain

Argentina

Cuba

IrelandMozambique

Sri Lanka

Armenia

Curacao

Isle of Man

Nauru

St. Kitts and Nevis

Australia

CyprusIsraelThe Netherlands

Suriname

Austria

Czech Republic

Italy

New ZealandSwedenAzerbaijan

Denmark

Jamaica

Nicaragua

Switzerland

The Bahamas

Estonia

Japan

Niger

Tajikistan

BahrainFaroe Islands

Kazakhstan

Tanzania

Bangladesh

Fiji

Kenya

Northern Marina
Islands

Barbados

Finland

Kiribati

NorwayTongo

BelarusFranceKorea, Republic

Belgium

The Gambia

KosovoBelizeGeorgia

Kyrgyz Republic

BermudaGermany

LatviaBhutanGibraltar

LebanonBolivia

GreeceLiechtenstein

BrazilGreenland

Lithuania

Brunei

GrenadaLuxembourg

Bulgaria

Guam

Macao SAR, China

Burkina Faso

Guatemala

MalawiCameroon

GuyanaMalaysia

CanadaHong Kong

Malta

Cayman Islands

Hungary

Mauritius

Chile

Iceland

MexicoTable of Contents

Appendix

PanamaPapua New Guinea
Philippines
PolandPortugal
Romania
RussiaSamoa
Saudi Arabia
SerbiaSeychelles
Sierra Leone
Singapore

Togo
TurkeyTuvaluUgandaUkraineUnited Arab Emirates
United Kingdom
United States
Uruguay
Vietnam
Yemen
ZambiaZimbabwe

Slovak Republic
37 7

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 6: Policy and Governance

United States State-Level AI Legislation
For AI-related bills passed into law, the AI Index performed searches of the keyword “artificial intelligence” onthe legislative websites of all 50 U.S. states in the full text of bills. Bills are only counted as passed into law if the
final version of the bill includes the keyword, not just the introduced version. Note that only laws passed from2015 to 2022 are included. The count for proposed laws includes both laws that were proposed and eventually
passed as well as laws that were proposed that have not yet been passed, or are now inactive. In some cases,
databases were only searchable by title, so site search functions were deployed. The AI Index team surveyed
the following databases:
Alabama

Hawaii

Massachusetts

New Mexico

South Dakota

AlaskaIdaho

Michigan

New York

Tennessee

Arizona

Illinois

Minnesota

North Carolina

Texas

Arkansas

Indiana

Mississippi

North Dakota

Utah

California

Iowa

Missouri

Ohio

Vermont

Colorado

KansasMontanaOklahoma

Virginia

Connecticut

Kentucky

Nebraska

OregonWashington

DelawareLouisiana

NevadaPennsylvania

West Virginia

FloridaMaine

New Hampshire

Rhode Island

Wisconsin

Georgia

MarylandNew Jersey

South Carolina

Wyoming

Table of Contents

Appendix378

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 6: Policy and Governance

Global AI Mentions
For mentions of AI in AI-related legislative proceedings around the world, the AI Index performed searches of
the keyword “artificial intelligence” on the websites of 81 countries’ congresses or parliaments (in the respective
languages), usually under sections named “minutes,” “hansard,” etc. In some cases, databases were only
searchable by title, so site search functions were deployed. The AI Index team surveyed the following databases:
AndorraEcuadorJapan

AngolaEl Salvador

Kenya

Armenia

Estonia

KosovoAustralia

Fiji

LatviaAzerbaijan

Finland

Lesotho

Barbados

FranceLiechtenstein

Belgium

The Gambia

Luxembourg

BermudaGermany

Macao SAR, China

BhutanGibraltar

Madagascar

BrazilGreeceMalaysia

Cabo Verde

Hong Kong

Maldives

CanadaIceland

Malta

Cayman Islands

India

Mauritius

China

Ireland

MexicoCzech Republic

Isle of Man

Moldova

Denmark

IsraelNetherlands

Dominican Republic

Italy

New Zealand

11

Northern Mariana
Islands

South Africa

NorwaySpain

Pakistan
PanamaPapua New Guinea
Philippines
PolandPortugal
Romania
RussiaSamoa
San Marino
Seychelles
Sierra Leone

South Korea
Sri Lanka
SwedenSwitzerland
Tanzania
Trinidad and Tobago
UkraineUnited Kingdom
United States
Uruguay
ZambiaZimbabwe

Singapore
Slovenia

11 The National People’s Congress is held once per year and does not provide full legislative proceedings. Hence, the counts included in the analysis only searched mentions of “artificial
intelligence” in the only public document released from the Congress meetings, the Report on the Work of the Government, delivered by the premier.

Table of Contents

Appendix379

Artificial IntelligenceIndex Report 2023

United States
Committee Mentions
In order to research trends on the United States’committee mentions of AI, the following search was
conducted:

Appendix
Chapter 6: Policy and Governance• PrivateSector Companies: Google AI, Microsoft
AI, Nvidia, OpenAI	• Think Tanks and Policy Institutes: American
Enterprise Institute, Aspen Institute, Atlantic
Council, Brookings Institute, Carnegie
Endowment for International Peace, Cato

Website: Congress.gov

Institute, Center for a New American Security,

Keyword: artificial intelligence

Center for Strategic and International Studies,

Filters: Committee Reports

Council on Foreign Relations, Heritage

United States AI Policy Papers

Foundation, Hudson Institute, MacroPolo,
National Security Institute, New America
Foundation, RAND Corporation, Rockefeller

Organizations
To develop a more nuanced understanding of thethought leadership that motivates AI policy, weFoundation, Stimson Center, Urban Institute,
Wilson Center

tracked policy papers published by 55 organizations	• University Institutes and Research Programs:

in the United States or with a strong presence in the

AI and Humanity, Cornell University; AI Now

United States (expanded from last year’s list of 36

Institute, New York University; AI Pulse, UCLA

organizations) across four broad categories:

Law; Belfer Center for Science and International• Civil Society, Associations, and Consortiums:
Algorithmic Justice League, Alliance for Artificial
Intelligence in Healthcare, Amnesty International,
EFF, Future of Privacy Forum, Human Rights
Watch, IJIS Institute, Institute for Electrical andElectronics Engineers, Partnership on AI	• Consultancy: Accenture, Bain & Company,

Affairs, Harvard University; Berkman Klein Center,
Harvard University; Center for Information
Technology Policy, Princeton University; Center
for Long-Term Cybersecurity, UC Berkeley;
Center for Security and Emerging Technology,
Georgetown University; CITRIS Policy Lab,UC Berkeley; Hoover Institution, Stanford
University; Institute for Human-Centered Artificial

Boston Consulting Group, Deloitte, McKinsey &Intelligence, Stanford University; Internet Policy

Company

Research Initiative, Massachusetts Institute of	• Government Agencies: Congressional ResearchService, Defense Technical Information Center,

Technology; MIT Lincoln Laboratory; Princeton
School of Public and International Affairs

Government Accountability Office, Library ofCongress, Pentagon Library

Table of Contents

Appendix380

Artificial IntelligenceIndex Report 2023

Methodology

Appendix
Chapter 6: Policy and Governance• Industry and Regulation: economy, antitrust,

Each broad topic area is based on a collection of

M&A, competition, finance, management, supply

underlying keywords that describe the content of thechain, telecom, economic regulation, technicalspecific paper. We included 17 topics that represented

standards, autonomous vehicle industry andthe majority of discourse related to AI between 2018–

regulation

2021. These topic areas and the associated keywords
are listed below:• Innovation and Technology: advancements andimprovements in AI technology, R&D, intellectual• Health and Biological Sciences: medicine,property, patents, entrepreneurship, innovation

healthcare systems, drug discovery, care,ecosystems, startups, computer science,biomedical research, insurance, health behaviors,

engineering

COVID-19, global health• Physical Sciences: chemistry, physics, astronomy,
earth science• Education and Skills: early childhood, K–12,
higher education, STEM, schools, classrooms,
reskilling• Energy and Environment: energy costs, climate• Workforce and Labor: labor supply and demand,change, energy markets, pollution, conservation,

talent, immigration, migration, personnel

oil and gas, alternative energy

economics, future of work

• International Affairs and International Security:• Social and Behavioral Sciences: sociology,international relations, international trade,linguistics, anthropology, ethnic studies,developing countries, humanitarian assistance,

demography, geography, psychology, cognitive

warfare, regional security, national security,

scienceautonomous weapons• Humanities: arts, music, literature, language,• Justice and Law Enforcement: civil justice,performance, theater, classics, history,

criminal justice, social justice, police, public

philosophy, religion, cultural studies

safety, courts• Equity and Inclusion: biases, discrimination,• Communications and Media: social media,
disinformation, media markets, deepfakes• Government and Public Administration:

gender, race, socioeconomic inequality,
disabilities, vulnerable populations• Privacy, Safety, and Security: anonymity, GDPR,

federal government, state government, local

consumer protection, physical safety, humangovernment, public sector efficiency, public

control, cybersecurity, encryption, hacking

sector effectiveness, government services,
government benefits, government programs,
public works, public transportation• Ethics: transparency, accountability, humanvalues, human rights, sustainability, explainability,interpretability, decision-making norms• Democracy: elections, rights, freedoms, liberties,
personal freedoms
Table of Contents

Appendix381

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 6: Policy and Governance

National AI Strategies
The AI Index did a web search to identify national strategies on AI. Below is a list of countries that were identified
as having a national AI strategy, including a link to said strategy. For certain counties, noted with an asterisk(*), theactual strategy was not found, and a news article confirming the launch of the strategy was linked instead.

Countries with AI Strategies in Place
Algeria*

Cyprus

Italy

Philippines

Switzerland

Argentina

Czech Republic

Japan

PolandThailandAustralia

Denmark

Kenya

Portugal

Tunisia*

AustriaEgypt, Arab Republic

Korea, Republic

Qatar

TurkeyBangladesh

Estonia

LatviaRomania

UkraineBotswana*

FinlandLithuania

RussiaUnited Arab Emirates

BrazilFranceLuxembourg

Saudi Arabia

United Kingdom

Bulgaria

Germany

Malta

SerbiaUnited States

CanadaGreeceMauritius

Sierra Leone

Uruguay

Chile

Hungary

MexicoSingapore

Vietnam

China

India

The Netherlands

Slovenia

Colombia

Indonesia

NorwaySpain

Croatia

Ireland

Peru

SwedenCountries withAI Strategies in
Development
Armenia

Federal Budget for Nondefense AI R&D
Data on the federal U.S. budget for nondefense AI R&D was taken from previous
editions of the AI Index (namely the 2021 and 2022 versions) and from the

Azerbaijan

following National Science and Technology Council reports:

Bahrain

Supplement to the President’s FY 2023 Budget

Belgium

Supplement to the President’s FY2022 Budget

Benin
Cuba
IcelandIsraelJordanMoroccoNew ZealandNigeria
Oman
Uzbekistan

Table of Contents

U.S. Department of Defense Budget Requests
Data on the DoD nonclassified AI-related budget requests was taken fromprevious editions of the AI Index (namely the 2021 and 2022 versions) and fromthe following reports:
Defense Budget Overview United States Department of Defense
Fiscal Year 2023 Budget Request
Defense Budget Overview United States Department of Defense
Fiscal Year 2022 Budget Request

Appendix382

Artificial IntelligenceIndex Report 2023

GoviniAppendix
Chapter 6: Policy and Governancethrough regular government reporting processes or

Govini is the leading commercial data company inthe defense technology space. Built by Govini, Ark.
ai is used at scale across the national security sectorof the U.S. federal government. This platform enables
government analysts, program managers, anddecision-makers to gain unprecedented visibility into
the companies, capabilities, and capital in national
security to solve challenges pertaining to acquisition,
foreign influence and adversarial capital, nuclear

human-intensive analytical approaches.
Moreover, beyond simply making usable an expansive
body of data sources, Govini’s SaaS Platform andNational Security Knowledge Graph establishes highfidelity standards in categorized and fused data to
produce a comprehensive and accurate depictionof federal spending, and the supporting vendor
ecosystem, over time.

modernization, procurement, science and technology,

U.S. AI-Related Legal Cases

and supply chain.

To identify AI-related legal cases, the AI Index research

Govini curated USG AI spend data from their annual
Scorecard Taxonomy by applying supervised machinelearning (ML) and natural language processing (NLP)
to parse, analyze, and categorize large volumes offederal contracts data, including prime contracts,
grants, and other transaction authority (OTA)

team did a keyword search on the LexisNexis database,under their U.S. legal cases filter. The keywords thatwere searched include “artificial intelligence,” “machinelearning,” and “automated decision-making.” Cases
that contained one of these keywords were coded
according to a variety of variables of interest.

awards. Govini’s most recent scorecard focused on
critical technologies, of which AI/ML technologies
was a segment and consistent of six subsegments:
data-at-scale, decision science, computer vision,machine learning, autonomy, and natural language
processing. By initially generating search terms andthen subsequently excluding specific terms that yielderroneous results, Govini delivers a comprehensive
yet discriminant taxonomy of subsegments that are
mutually exclusive. Repeated keyword searches andfilters allow a consensus, data-driven taxonomy tocome into focus. Govini SMEs conduct a final review
of taxonomic structure to complement this iterative,
data-driven process.
The use of AI and supervised ML models enables theanalysis of large volumes of irregular data containedin federal contracts—data that is often inaccessible

Table of Contents

Appendix383

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 7: Diversity

Chapter 7: Diversity
Computing Research
Association (CRA Taulbee
Survey)To learn more about the diversity data from the CRA,
please read the methodological note on the CRA’s data
included in the Chapter 5 subsection of the Appendix.

Code.org
To learn more about the diversity data from Code.
org, please read the methodological note on Code.
org’s data included in the Chapter 5 subsection of the
Appendix.

Table of Contents

Appendix384

Artificial IntelligenceIndex Report 2023

Appendix
Chapter 8: Public Opinion

Chapter 8: Public Opinion
IPSOS
For brevity, the 2023 AI Index does not republish the

NetBase Quid
Social Media Datamethodology used by the IPSOS survey that features

NetBase Quid collects social media data from overin the report. More details about the IPSOS survey’s
methodology can be found in the actual survey.

Lloyd’s Register Foundation
and Gallup
For brevity, the 2023 AI Index does not republish themethodology used by the Lloyd’s Register Foundation
and Gallup survey that features in the report. Moredetails about the Lloyd’s Register Foundation and
Gallup survey methodology can be found in the actual
survey.

500 million sources in real time and analyzes this data
through AI-powered Natural Language Processing.
This process parses out language and breaks out
posts by filters such as drivers of positive and negative
sentiment, emotions, and behaviors, allowing fordeeper insights to be reached. To understand public
perception of advancements in artificial intelligence,
NetBase Quid analyzed social media conversation
around AI and AI model releases from January 2022
to December 2022. First, the NetBase Quid teamanalyzed conversation around AI to understand key
drivers of general sentiment around AI advancements,

Pew Researchsuch as ethical, cultural, and economic concerns andFor brevity, the 2023 AI Index does not republish

Quid team leveraged the platform for a more targeted

the methodology used by the Pew Research survey

analysis of the same conversation, understandingthat features in the report. More details on the Pew

volume and sentiment around the major AI modelResearch survey methodology can be found in the

updates and releases in 2022. This NetBase Quidactual survey.

analysis ultimately showcases the relationshipperceptions among consumers. Then, the NetBase

between public perception and the advancement ofAI, leveraging targeted analytics tools to understandboth specific reactions to model releases as well as awider consumer conversation and what drives it.

Table of Contents

Appendix385

Artificial IntelligenceIndex Report 2023