Artificial IntelligenceIndex Report 2022

Artificial IntelligenceIndex Report 2022

INTRODUCTION TO THE
AI INDEX REPORT 2022
Welcome to the fifth edition of the AI Index Report! The latest edition includes data from a broad set of academic,
private, and nonprofit organizations as well as more self-collected data and original analysis than any previous
editions, including an expanded technical performance chapter, a new survey of robotics researchers around theworld, data on global AI legislation records in 25 countries, and a new chapter with an in-depth analysis of technical
AI ethics metrics.
The AI Index Report tracks, collates, distills, and visualizes data related to artificial intelligence. Its mission is toprovide unbiased, rigorously vetted, and globally sourced data for policymakers, researchers, executives, journalists,
and the general public to develop a more thorough and nuanced understanding of the complex field of AI. The reportaims to be the world’s most credible and authoritative source for data and insights about AI.

F R O M T H E C O - D I R E CTO R SThis year’s report shows that AI systems are starting to be deployed widely into the economy, but at the same timethey are being deployed, the ethical issues associated with AI are becoming magnified. Some of this is natural—after
all, we tend to care more about the ethical aspects of a given technology when it is being rolled out into the world.
But some of it is bound up in the peculiar traits of contemporary AI—larger and more complex and capable AI systems
can generally do better on a broad range of tasks while also displaying a greater potential for ethical concerns.
This is bound up with the broad globalization and industrialization of AI—a larger range of countries are developing,
deploying, and regulating AI systems than ever before, and the combined outcome of these activities is the creationof a broader set of AI systems available for people to use, and reductions in their prices. Some parts of AI are not very
globalized, though, and our ethics analysis reveals that many AI ethics publications tend to concentrate on Englishlanguage systems and datasets, despite AI being deployed globally.
If anything, we expect the above trends to continue: 103% more money was invested in the private investment of AI
and AI-related startups in 2021 than in 2020 ($96.5 billion versus $46 billion).

Jack Clark and Ray Perrault2Artificial IntelligenceIndex Report 2022

TOP TAKEAWAYS
Private investment in AI soared while investment concentration intensified:•T he private investment in AI in 2021 totaled around $93.5 billion—more than double the total private
investment in 2020, while the number of newly funded AI companies continues to drop, from 1051 companies in2019 and 762 companies in 2020 to 746 companies in 2021. In 2020, there were 4 funding rounds worth $500
million or more; in 2021, there were 15.

U.S. and China dominated cross-country collaborations on AI:• Despite rising geopolitical tensions, the United States and China had the greatest number of cross-country
collaborations in AI publications from 2010 to 2021, increasing five times since 2010. The collaboration between
the two countries produced 2.7 times more publications than between the United Kingdom and China—the second
highest on the list.

Language models are more capable than ever, but also more biased:• L arge language models are setting new records on technical benchmarks, but new data shows that larger models arealso more capable of reflecting biases from their training data. A 280 billion parameter model developed in 2021
shows a 29% increase in elicited toxicity over a 117 million parameter model considered the state of the art asof 2018. The systems are growing significantly more capable over time, though as they increase in capabilities, sodoes the potential severity of their biases.

The rise of AI ethics everywhere:• Research on fairness and transparency in AI has exploded since 2014, with a fivefold increase in related publicationsat ethics-related conferences. Algorithmic fairness and bias has shifted from being primarily an academic pursuit tobecoming firmly embedded as a mainstream research topic with wide-ranging implications. Researchers with industry
affiliations contributed 71% more publications year over year at ethics-focused conferences in recent years.

AI becomes more affordable and higher performing:•S ince 2018, the cost to train an image classification system has decreased by 63.6%, while training times
have improved by 94.4%. The trend of lower training cost but faster training time appears across other MLPerf task
categories such as recommendation, object detection and language processing, and favors the more widespreadcommercial adoption of AI technologies.

Data, data, data:•T op results across technical benchmarks have increasingly relied on the use of extra training data to set new state-ofthe-art results. As of 2021, 9 state-of-the-art AI systems out of the 10 benchmarks in this report are trained withextra data. This trend implicitly favors private sector actors with access to vast datasets.

More global legislation on AI than ever:• An AI Index analysis of legislative records on AI in 25 countries shows that the number of bills containing “artificial
intelligence” that were passed into law grew from just 1 in 2016 to 18 in 2021. Spain, the United Kingdom, and theUnited States passed the highest number of AI-related bills in 2021 with each adopting three.

Robotic arms are becoming cheaper:• An AI Index survey shows that the median price of robotic arms has decreased by 46.2% in the past five years—
from $42,000 per arm in 2017 to $22,600 in 2021. Robotics research has become more accessible and affordable.

3

Artificial IntelligenceIndex Report 2022

Steering Committee
Co-Directors
Jack Clark
Anthropic, OECD

Raymond Perrault
SRI International

Members
Erik Brynjolfsson
Stanford University
John Etchemendy
Stanford University
Terah Lyons

James Manyika
Google, Universityof Oxford
Juan Carlos Niebles
Stanford University,
Salesforce

Michael Sellitto
Stanford University
Yoav Shoham(Founding Director)
Stanford University,
AI21 Labs

Staff and Researchers
Research Manager and Editor in Chief

Research AssociateDaniel Zhang
Stanford University

Nestor Maslej
Stanford University

Affiliated Researchers
Andre Barbe
The World Bank

Helen Ngo
Cohere

Latisha Harry
Independent Consultant

Ellie Sakhaee
Microsoft

Graduate Researcher
Benjamin Bronkema-Bekker
Stanford University

4Artificial IntelligenceIndex Report 2022

How to Cite This Report
Daniel Zhang, Nestor Maslej, Erik Brynjolfsson, John Etchemendy, Terah Lyons, James
Manyika, Helen Ngo, Juan Carlos Niebles, Michael Sellitto, Ellie Sakhaee, Yoav Shoham,Jack Clark, and Raymond Perrault, “The AI Index 2022 Annual Report,” AI Index Steering
Committee, Stanford Institute for Human-Centered AI, Stanford University, March 2022.
The AI Index 2022 Annual Report by Stanford University is licensed under
Attribution-NoDerivatives 4.0 International. To view a copy of this license,
visit http://creativecommons.org/licenses/by-nd/4.0/.

Public Data and Tools
The AI Index 2022 Report is supplemented by raw data and an interactive tool. We invite eachreader to use the data and the tool in a way most relevant to their work and interests.
•R aw data and charts: The public data and high-resolution images of all the charts in thereport are available on Google Drive.
•G lobal AI Vibrancy Tool: We redesigned the Global AI Vibrancy Tool this year with abetter visualization to compare up to 29 countries across 23 indicators.

AI Index and Stanford HAI
The AI Index is an independent initiative at the
Stanford Institute for Human-Centered Artificial Intelligence (HAI).

The AI Index was conceived within the One Hundred Year Study on AI (AI100).

We welcome feedback and new ideas for next year.
Contact us at AI-Index-Report@stanford.edu.

5Artificial IntelligenceIndex Report 2022

Supporting Partners

Analytics and Research Partners

6Artificial IntelligenceIndex Report 2022

Contributors
We want to acknowledge the following individuals by chapter and section for their contributionsof data, analysis, advice, and expert commentary included in the AI Index 2022 Report:

Research and Development
Sara Abdulla, Catherine Aiken, Jack Clark, James Dunham, Nezihe Merve Gürel, Nestor Maslej, Ray Perrault,
Sarah Tan, Daniel Zhang

Technical Performance
Jack Clark, David Kanter, Nestor Maslej, Deepak Narayanan, Juan Carlos Niebles, Konstantin Savenkov,
Yoav Shoham, Daniel Zhang

Technical AI Ethics
Jack Clark, Nestor Maslej, Helen Ngo, Ray Perrault, Ellie Sakhaee, Daniel Zhang

The Economy and Education
Betsy Bizot, Erik Brynjolfsson, Jack Clark, John Etchemendy, Murat Erer, Akash Kaura, Julie Kim, Nestor Maslej,
James Manyika, Brenden McKinney, Julia Nitschke, Ray Perrault, Brittany Presten, Tejas Sirohi, Bledi Taska,
Rucha Vankudre, Daniel Zhang

AI Policy and GovernanceAmanda Allen, Benjamin Bronkema-Bekker, Jack Clark, Latisha Harry, Taehwa Hong, Cameron Leuthy, Terah
Lyons, Nestor Maslej, Ray Perrault, Michael Sellitto, Teruki Takiguchi, Daniel Zhang

Conference Attendance
Terri Auricchio (ICML), Christian Bessiere (IJCAI), Meghyn Bienvenu (KR), Andrea Brown (ICLR),
Alexandra Chouldechova (FAccT), Nicole Finn (ICCV, CVPR), Enrico Gerding (AAMAS), Carol Hamilton (AAAI),
Seth Lazar (FAccT), Max Qing Hu Meng (ICRA), Jonas Martin Peters (UAI), Libor Preucil (IROS),
Marc’Aurelio Ranzato (NeurIPS), Priscilla Rasmussen (EMNLP, ACL), Hankz Hankui Zhuo (ICAPS)

Global AI Vibrancy Tool
Andre Barbe, Latisha Harry, Daniel Zhang

Robotics Survey
Pieter Abbeel, David Abbink, Farshid Alambeigi, Farshad Arvin, Nikolay Atanasov, Ruzena Bajcsy, Philip Beesley,
Tapomayukh Bhattacharjee, Jeannette Bohg, David J. Cappelleri, Qifeng Chen, I-Ming Chen, Jack Cheng,
Cynthia Chestek, Kyujin Cho, Dimitris Chrysostomou, Steve Collins, David Correa, Brandon DeHart,
Katie Driggs-Campbell, Nima Fazeli, Animesh Garg, Maged Ghoneima, Tobias Haschke, Kris Hauser, David Held,Yue Hu, Josie Hughes, Soo Jeon, Dimitrios Kanoulas, Jonathan Kelly, Oliver Kroemer, Changliu Liu, Ole Madsen,
Anirudha Majumdar, Genaro J. Martinez, Saburo Matunaga, Satoshi Miura, Norrima Mokhtar, Elena De Momi,
Chrystopher Nehaniv, Christopher Nielsen, Ryuma Niiyama, Allison Okamura, Necmiye Ozay, Jamie Paik,
Frank Park, Karthik Ramani, Carolyn Ren, Jan Rosell, Jee-Hwan Ryu, Tim Salcudean, Oliver Schneider,
Angela Schoellig, Reid Simmons, Alvaro Soto, Peter Stone, Michael Tolley, Tsu-Chin Tsao, Michiel van de Panne,
Andy Weightman, Alexander Wong, Helge Wurdemann, Rong Xiong, Chao Xu, Geng Yang, Junzhi Yu,
Wenzhen Yuan, Fu Zhang, Yuke Zhu
7Artificial IntelligenceIndex Report 2022

We thank the following organizations and individuals who provided data forinclusion in the AI Index 2022 Report:

Organizations
Bloomberg Government
Amanda Allen, Cameron Leuthy

Intento
Grigory Sapunov, Konstantin Savenkov

Center for Security and Emerging
Technology, Georgetown University
Sara Abdulla, Catherine Aiken,
James Dunham

LinkedIn
Murat Erer, Akash Kaura

Computing Research Association
Betsy Bizot
Emsi Burning Glass
Julia Nitschke, Bledi Taska,
Rucha Vankudre

McKinsey Global InstituteBrenden McKinney, Brittany Presten
MLCommonsDavid Kanter
NetBase Quid
Julie Kim, Tejas Sirohi
Women in Machine Learning
Nezihe Merve Gürel, Sarah Tan

We also would like to thank Jeanina Casusi, Nancy King, Shana Lynch, Jonathan Mindes,
Stacy Peña, Michi Turner, and Justin Sherman for their help in preparing this report, andJoe Hinman, Travis Taylor, and the team at Digital Avenues for their efforts in designing anddeveloping the AI Index and HAI websites.

8Artificial IntelligenceIndex Report 2022

Table of Contents
REPORT HIGHLIGHTS		
10
CHAPTER 1

Research and Development		 13CHAPTER 2

Technical Performance		 47CHAPTER 3

Technical AI Ethics

100

CHAPTER 4

The Economy and Education

139

CHAPTER 5

AI Policy and Governance

172

APPENDIX		196

ACCESS THE PUBLIC DATA

9

Artificial IntelligenceIndex Report 2022

REPORT HIGHLIGHTSC H A P T E R 1 : R E S E A R C H A N D D E V E LO P M E N T• Despite rising geopolitical tensions, the United States and China had the greatest number of cross-country
collaborations in AI publications from 2010 to 2021, increasing five times since 2010. The collaboration between
the two countries produced 2.7 times more publications than between the United Kingdom and China—thesecond highest on the list.
• In 2021, China continued to lead the world in the number of AI journal, conference, and repository
publications—63.2% higher than the United States with all three publication types combined. In the meantime,
the United States held a dominant lead among major AI powers in the number of AI conference and repository
citations.
• From 2010 to 2021, the collaboration between educational and nonprofit organizations produced thehighest number of AI publications, followed by the collaboration between private companies and educational
institutions and between educational and government institutions.
• The number of AI patents filed in 2021 is more than 30 times higher than in 2015, showing a compound
annual growth rate of 76.9%.

C H A P T E R 2: T E C H N I CA L P E R F O R M A N C E• Data, data, data: Top results across technical benchmarks have increasingly relied on the use of extra training
data to set new state-of-the-art results. As of 2021, 9 state-of-the-art AI systems out of the 10 benchmarksin this report are trained with extra data. This trend implicitly favors private sector actors with access to vast
datasets.
• Rising interest in particular computer vision subtasks: In 2021, the research community saw a greater levelof interest in more specific computer vision subtasks, such as medical image segmentation and masked-faceidentification. For example, only 3 research papers tested systems against the Kvasir-SEG medical imaging
benchmark prior to 2020. In 2021, 25 research papers did. Such an increase suggests that AI research ismoving toward research that can have more direct, real-world applications.
• AI has not mastered complex language tasks, yet: AI already exceeds human performance levels on basic
reading comprehension benchmarks like SuperGLUE and SQuAD by 1%–5%. Although AI systems are still
unable to achieve human performance on more complex linguistic tasks such as abductive natural
language inference (aNLI), the difference is narrowing. Humans performed 9 percentage points better on
aNLI in 2019. As of 2021, that gap has shrunk to 1.

10Artificial IntelligenceIndex Report 2022

• Turn toward more general reinforcement learning: For the last decade, AI systems have been able to masternarrow reinforcement learning tasks in which they are asked to maximize performance in a specific skill, suchas chess. The top chess software engine now exceeds Magnus Carlsen’s top ELO score by 24%. However, inthe last two years AI systems have also improved by 129% on more general reinforcement learning tasks(Procgen) in which they must operate in novel environments. This trend speaks to the future development ofAI systems that can learn to think more broadly.
• AI becomes more affordable and higher performing: Since 2018, the cost to train an image classification
system has decreased by 63.6%, while training times have improved by 94.4%. The trend of lower training
cost but faster training time appears across other MLPerf task categories such as recommendation, object
detection and language processing, and favors the more widespread commercial adoption of AI technologies.
• Robotic arms are becoming cheaper: An AI Index survey shows that the median price of robotic arms has
decreased by 46.2% in the past five years—from $42,000 per arm in 2017 to $22,600 in 2021. Robotics
research has become more accessible and affordable.

C H A P T E R 3: T E C H N I CA L A I E T H I C S• Language models are more capable than ever, but also more biased: Large language models are setting newrecords on technical benchmarks, but new data shows that larger models are also more capable of reflecting
biases from their training data. A 280 billion parameter model developed in 2021 shows a 29% increase inelicited toxicity over a 117 million parameter model considered the state of the art as of 2018. The systems
are growing significantly more capable over time, though as they increase in capabilities, so does the potential
severity of their biases.
• The rise of AI ethics everywhere: Research on fairness and transparency in AI has exploded since 2014, with afivefold increase in related publications at ethics-related conferences. Algorithmic fairness and bias has shiftedfrom being primarily an academic pursuit to becoming firmly embedded as a mainstream research topic withwide-ranging implications. Researchers with industry affiliations contributed 71% more publications year
over year at ethics-focused conferences in recent years.
• Multimodal models learn multimodal biases: Rapid progress has been made on training multimodal languagevision models which exhibit new levels of capability on joint language-vision tasks. These models have set newrecords on tasks like image classification and the creation of images from text descriptions, but they also reflectsocietal stereotypes and biases in their outputs—experiments on CLIP showed that images of Black people
were misclassified as nonhuman at over twice the rate of any other race. While there has been significant
work to develop metrics for measuring bias within both computer vision and natural language processing, thishighlights the need for metrics that provide insight into biases in models with multiple modalities.

11Artificial IntelligenceIndex Report 2022

C H A P T E R 4 : T H E E C O N O M Y A N D E D U CAT I O N• New Zealand, Hong Kong, Ireland, Luxembourg, and Sweden are the countries or regions with the highest growth
in AI hiring from 2016 to 2021.
• In 2021, California, Texas, New York, and Virginia were states with the highest number of AI job postings in the
United States, with California having over 2.35 times the number of postings as Texas, the second greatest.
Washington, D.C., had the greatest rate of AI job postings compared to its overall number of job postings.
• The private investment in AI in 2021 totaled around $93.5 billion—more than double the total private
investment in 2020, while the number of newly funded AI companies continues to drop, from 1051 companies in2019 and 762 companies in 2020 to 746 companies in 2021. In 2020, there were 4 funding rounds worth $500
million or more; in 2021, there were 15.
• “Data management, processing, and cloud” received the greatest amount of private AI investment in 2021—
2.6 times the investment in 2020, followed by “medical and healthcare” and “fintech.”• In 2021, the United States led the world in both total private investment in AI and the number of newly funded AI
companies, three and two times higher, respectively, than China, the next country on the ranking.
• Efforts to address ethical concerns associated with using AI in industry remain limited, according to a McKinsey
survey. While 29% and 41% of respondents recognize “equity and fairness” and “explainability” as risks
while adopting AI, only 19% and 27% are taking steps to mitigate those risks.
• In 2020, 1 in every 5 CS students who graduated with PhD degrees specialized in artificial intelligence/machine learning, the most popular specialty in the past decade. From 2010 to 2020, the majority of AI PhDs in
the United States headed to industry while a small fraction took government jobs.

C H A P T E R 5: A I P O L I CY A N D G OV E R N A N C E• An AI Index analysis of legislative records on AI in 25 countries shows that the number of bills containing“artificial intelligence” that were passed into law grew from just 1 in 2016 to 18 in 2021. Spain, the United
Kingdom, and the United States passed the highest number of AI-related bills in 2021, with each adopting three.
• The federal legislative record in the United States shows a sharp increase in the total number of proposed bills
that relate to AI from 2015 to 2021, while the number of bills passed remains low, with only 2% ultimately
becoming law.
• State legislators in the United States passed 1 out of every 50 proposed bills that contain AI provisions in 2021,
while the number of such bills proposed grew from 2 in 2012 to 131 in 2021.
• In the United States, the current congressional session (the 117th) is on track to record the greatest number ofAI-related mentions since 2001, with 295 mentions by the end of 2021, half way through the session,compared to 506 in the previous (116th) session.

12Artificial IntelligenceIndex Report 2022

CHAPTER 1:

Research &Development

Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT

CHAPTER 1:

Chapter Preview
Overview

15

AI Repositories

32

Chapter Highlights

16

Overview

32

By Region

33

1.1 PUBLICATIONS

17

By Geographic Area

34

Overview

17

Citations35

Total Number of AI Publications

17

By Type of Publication

18

Overview

36

By Field of Study19

By Region and Application Status

37

By Sector

20

Cross-Country Collaboration

22

By Geographic Area andApplication Status

39

Cross-Sector Collaboration

23

AI Patents

36

24

1.2 CONFERENCES

41

Overview

24

Conference Attendance

41

By Region

25

By Geographic Area

26

Women in Machine Learning (WiML)
NeurIPS Workshop

43

Citations

27

AI Journal Publications

AI Conference Publications

28

Workshop Participants

43

Demographics Breakdown

44

Overview

28

By Region29

1.3 AI OPEN-SOURCE SOFTWARELIBRARIES

45

By Geographic Area

30

GitHub Stars

45

Citations

31

ACCESS THE PUBLIC DATA

Table of Contents

14Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT

Overview
Research and development is an integral force that drives the rapid progressof artificial intelligence (AI). Every year, a wide range of academic, industry,
government, and civil society experts and organizations contribute to AI
R&D via a slew of papers, journal articles, and other AI-related publications,conferences on AI or on particular subtopics like image recognition or
natural language processing, international collaboration across borders, andthe development of open-source software libraries. These R&D efforts arediverse in focus and geographically dispersed.
Another key feature of AI R&D, making it somewhat distinct from other areasof STEM research, is its openness. Each year, thousands and thousands ofAI publications are released in the open source, whether at conferencesor on file-sharing websites. Researchers will openly share their findings at
conferences; government agencies will fund AI research that ends up in the
open source; and developers use open software libraries, freely available tothe public, to produce state-of-the-art AI applications. This openness also
contributes to the globally interdependent and interconnected nature ofmodern AI R&D.
This first chapter draws on multiple datasets to analyze key trends in the
AI research and development space in 2021. It first looks at AI publications,
including conference papers, journal articles, patents, and repositories.
It then analyzes AI conference attendance. And finally, it examines
AI open-source software libraries used in the R&D process.

Table of Contents

Chapter 1 Preview

15Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT

CHAPTER HIGHLIGHTS• Despite rising geopolitical tensions, the United States and China had the greatest number ofcross-country collaborations in AI publications from 2010 to 2021, increasing five times since2010. The collaboration between the two countries produced 2.7 times more publications
than between the United Kingdom and China—the second highest on the list.
• In 2021, China continued to lead the world in the number of AI journal, conference, andrepository publications—63.2% higher than the United States with all three publication types
combined. In the meantime, the United States held a dominant lead among major AI powers
in the number of AI conference and repository citations.
• From 2010 to 2021, the collaboration between educational and nonprofit organizations
produced the highest number of AI publications, followed by the collaboration between
private companies and educational institutions and between educational and government
institutions.
• The number of AI patents filed in 2021 is more than 30 times higher than in 2015, showing acompound annual growth rate of 76.9%.

Table of Contents

Chapter 1 Preview

16Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

This section draws on data from the Center for Security and Emerging Technology (CSET) at Georgetown University. CSET maintains
a merged corpus of scholarly literature that includes Digital Science’s Dimensions, Clarivate’s Web of Science, Microsoft Academic
Graph, China National Knowledge Infrastructure, arXiv, and Papers with Code. In that corpus, CSET applied a classifier to identify
English-language publications related to the development or application of AI and ML since 2010.1

1.1 PUBLICATIONS2OV E R V I E W

Total Number of AI Publications

The figures below capture the total number of Englishlanguage AI publications globally from 2010 to 2021—by
type, affiliation, cross-country collaboration, and crossindustry collaboration. The section also breaks down
publication and citation data by region for AI journal
articles, conference papers, repositories, and patents.

Figure 1.1.1 shows the number of AI publications in
the world. From 2010 to 2021, the total number of AI
publications doubled, growing from 162,444 in 2010 to334,497 in 2021.

NUMBER of AI PUBLICATIONS in the WORLD, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

Number of AI Publications (in thousands)

350

334.50

300

250

200

150

100

50
0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.1

1 See the Appendix for more information on CSET’s methodology. Due to the change in data provider and classification method, the publication trend/data may be different from past reports. For moreon the challenge of defining AI and correctly capturing relevant bibliometric data, see the AI Index team’s discussion in the paper “Measurement in AI Policy: Opportunities and Challenges.”
2 The number of AI publications in 2021 in this section may be lower than the actual count due to the lag in the collection of publication metadata by aforementioned databases.

Table of Contents

Chapter 1 Preview

17Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

By Type of Publication

Books, book chapters, theses, and unknown document
types comprised the remaining 10.1% of publications.
While journal and repository publications have grown
2.5 and 30 times, respectively, in the past 12 years, thenumber of conference papers has declined since 2018.

Figure 1.1.2 shows the variation overtime in the types ofAI publications released globally. In 2021, 51.5% of all AI
documents published were journal articles, 21.5% wereconference papers, and 17.0% were from repositories.

NUMBER of AI PUBLICATIONS by TYPE, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

172.11, Journal

180

Number of AI Publications (in thousands)

160
140
120
100
80
71.92, Conference60

56.73, Repository40
20.60, Book Chapter
10.84, Unknown
1.16, Book
1.14, Thesis

20
0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.2

Table of Contents

Chapter 1 Preview

18Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

By Field of Study
Figure 1.1.3 shows that publications in pattern recognition
and machine learning have more than doubled since 2015.

Other areas strongly influenced by deep learning, suchas computer vision, data mining and natural language
processing, have shown smaller increases.

NUMBER of AI PUBLICATIONS by FIELD of STUDY (EXCLUDING OTHER AI), 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

51.69, Pattern Recognition

Number of AI Publications (in thousands)

50

39.93, Machine Learning40

30
24.80, Computer Vision20

18.30, Algorithm
15.27, Data Mining
13.43, Natural Language Processing
9.70, Human–Computer Interaction
8.45, Control Theory

10

5.78, Linguistics

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.3

Table of Contents

Chapter 1 Preview

19Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

By Sector

United Kingdom (Figure 1.1.4d).3 The education sector
dominates in each of the regions. The level of company
participation is highest in the United States, then in theEuropean Union. China is the only area in which the shareof education has been rising.

This section shows the number of AI publications affiliatedwith industry, education, government, and nonprofit in
the world (Figure 1.1.4a), China (Figure 1.1.4b), the UnitedStates (Figure 1.1.4c), and the European Union plus the
AI PUBLICATIONS (% of TOTAL) by SECTOR, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report60%

59.58%, Education

AI Publications (% of Total)

50%

40%

30%

20.77%, Unknown

20%

11.27%, Nonpro t

10%

5.21%, Company
3.17%, Government

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.4a

AI PUBLICATIONS in UNITED STATES (% of TOTAL) by SECTOR, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

60%

57.63%, Education

AI Publications (% of Total)

50%

40%

30%

20%

17.63%, Unknown
12.36%, Nonpro t
9.76%, Company

10%2.62%, Government

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021

3 The categorization is adapted based on the Global Research Identifier Database (GRID). See definitions of each category here. Healthcare, including hospitals and facilities,
are included under nonprofit. Publications affiliated with state-sponsored universities are included in the education sector.

Table of Contents

Chapter 1 Preview

Figure 1.1.4b

20

Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

AI PUBLICATIONS in CHINA (% of TOTAL) by SECTOR, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report60.24%, Education

60%

AI Publications (% of Total)

50%

40%

30%
23.74%, Unknown

20%

10%

8.47%, Nonpro t
3.93%, Company
3.62%, Government

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.4c

AI PUBLICATIONS in EUROPEAN UNION and UNITED KINGDOM (% of TOTAL) by SECTOR, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

60%
54.81%, Education

AI Publications (% of Total)

50%

40%

30%

20%

19.76%, Unknown
16.14%, Nonpro t

10%
5.68%, Company
3.61%, Government

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.4d

Table of Contents

Chapter 1 Preview

21Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

Cross-Country Collaboration

By far, the greatest
number of collaborationsin the past 12 years
took place between the
United States and China,
increasing five times
since 2010.

Cross-border collaborations between academics,
researchers, industry experts, and others are a key
component of modern STEM development that accelerate
the dissemination of new ideas and the growth of researchteams. Figures 1.1.5a and 1.1.5b depict the top crosscountry AI collaborations from 2010 to 2021. CSET counted
cross-country collaborations as distinct pairs of countries
across authors for each publication (e.g., four U.S. andfour Chinese-affiliated authors on a single publication are
counted as one U.S.-China collaboration; two publications
between the same authors counts as two collaborations).
By far, the greatest number of collaborations in the past
12 years took place between the United States and China,
increasing five times since 2010. The next largest set ofcollaborations is between the United Kingdom and both theUnited States and China, which have increased more than
three times since 2010. In 2021, the number of collaborations
between the United States and China was 2.7 times greaterthan between the United Kingdom and China.

UNITED STATES and CHINA COLLABORATIONS in AI PUBLICATIONS, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

Number of AI Publications (in thousands)

10

9.66

8

6

4

2

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.5a

Table of Contents

Chapter 1 Preview

22Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

CROSS-COUNTRY COLLABORATIONS in AI PUBLICATIONS (EXCLUDING U.S. and CHINA), 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

3.56, United Kingdom and China

Number of AI Publications (in thousands)

3.34, United States and United Kingdom

32.63, China and Australia
2.44, United States and Germany
2.37, United States and Canada
2.16, United States and Australia

2

1.42, United States and France

1

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.5b

Cross-Sector Collaboration
The increase in AI research outside of universities has broughtcollaborations between universities and other industries.
Figure 1.1.6 shows that in 2021, educational institutions
and nonprofits had the greatest number of collaborations(29,839), followed by companies and educational institutions(11,576), and governments and educational institutions(8,087). There were 2.5 times as many collaborations
between educational institutions and nonprofits in 2021 as
between educational institutions and companies.

CROSS-SECTOR COLLABORATIONS in AI PUBLICATIONS, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

29.84, Education and Nonpro t

Number of AI Publications (in thousands)

30

25

20

15
11.58, Company and Education

10
8.09, Education and Government

52.48, Government and Nonpro t
1.83, Company and Nonpro t
0.66, Company and Government

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.6

Table of Contents

Chapter 1 Preview

23Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

A I J O U R N A L P U B L I CAT I O N S

since 2015 (Figure 1.1.7). As a percentage of all journal
publications, as captured in Figure 1.1.8, AI journal
publications in 2021 were about 2.5% of all publications,compared to 1.5% in 2010.

Overview
After growing only slightly from 2010 to 2015, thenumber of AI journal publications grew almost 2.5 times
NUMBER of AI JOURNAL PUBLICATIONS, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

Number of AI Journal Publications (in thousands)

172.11

150

100

50

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.7

AI JOURNAL PUBLICATIONS (% of TOTAL JOURNAL PUBLICATIONS), 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

2.53%

AI Journal Publications (% of Total Journal Publications)

2.50%

2.00%

1.50%

1.00%

0.50%

0.00%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.8

Table of Contents

Chapter 1 Preview

24Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

In addition, South Asiaand the Middle East andNorth Africa saw the mostsignificant growth astheir number of AI journal
publications grew around12 and 7 times, respectively,in the last 12 years.

By Region4
Figure 1.1.9 shows the share of AI journal publications
by region between 2010 and 2021. In 2021, East Asia
and Pacific leads with 42.9%, followed by Europe andCentral Asia (22.7%) and North America (15.6%). In
addition, South Asia and the Middle East and North
Africa saw the most significant growth as their numberof AI journal publications grew around 12 and 7 times,
respectively, in the last 12 years.

AI JOURNAL PUBLICATIONS (% of WORLD TOTAL) by REGION, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

AI Journal Publications (% of World Total)

50%
42.87%, East Asia and Paci c

40%

30%

22.70%, Europe and Central Asia

20%
15.63%, North America

10%7.99%, South Asia
6.18%, Middle East and North Africa
3.57%, Latin America and the Caribbean
1.06%, Sub-Saharan Africa

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.9

4 Regions in this chapter are classified according to the World Bank analytical grouping.

Table of Contents

Chapter 1 Preview

25Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

By Geographic Area5

powers. China has remained the leader throughout, with
31.0% in 2021, followed by the European Union plus theUnited Kingdom at 19.1% and the United States at 13.7%

Figure 1.1.10 breaks down the share of AI journal
publications over the past 12 years by three major AI

AI JOURNAL PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

AI Journal Publications (% of World Total)

40%

31.04%, China

30%

19.05%, European Union and United Kingdom

20%

13.67%, United States

10%

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.10

5 Geographic areas in this chapter combine the number of publications between the European Union and the United Kingdom to reflect the historically strong association between them with regards toresearch collaboration.

Table of Contents

Chapter 1 Preview

26Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

Citations

The three geographic areas
combined accounted formore than 66% of the total
citations in the world.

On the number of citations of AI journal
publications, China’s share has gradually increased
while those of the European Union plus the United
Kingdom and the United States have decreased.
The three geographic areas combined accounted formore than 66% of the total citations in the world.

AI JOURNAL CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

30%

AI Journal Citations (% of World Total)

27.84%, China

25%

21.13%, European Union and United Kingdom

20%

17.45%, United States

15%

10%

5%

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.11

Table of Contents

Chapter 1 Preview

27Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

A I C O N F E R E N C E P U B L I CAT I O N S(Figure 1.1.12). Despite the decline in the total numbers,
however, the share of AI conference publications among
total conference publications in the world has increasedby more than five percentage points since 2010 (Figure
1.1.13).

Overview
The number of AI conference publications peaked in
2019, and fell about 19.4% below the peak in 2021
NUMBER of AI CONFERENCE PUBLICATIONS, 2010–21

Number of AI Conference Publications (in thousands)

Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

80
71.92

60

40

20

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.12

AI CONFERENCE PUBLICATIONS (% of TOTAL CONFERENCE PUBLICATIONS), 2010–21
AI Conference Publications (% of Total Conference Publications)

Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

17.83%

16%

12%

8%

4%

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.13

Table of Contents

Chapter 1 Preview

28Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

By Region

share represented by East Asia and Pacific continues torise since taking the lead in 2014, accounting for 40.4%
in 2021, followed by Europe and Central Asia (23.0%) andNorth America (19.0%). The percentage of AI conference
publications in South Asia saw a noticeable rise in thepast 12 years, from 4.0% in 2010 to 10.4% in 2021.

Figure 1.1.14 shows the number of AI conference
publications by region. Similar to the trend in AI journal
publication, East Asia and Pacific, Europe and Central
Asia, and North America account for the world’s highest
numbers of AI conference publications. Specifically, theAI CONFERENCE PUBLICATIONS (% of WORLD TOTAL) by REGION, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

40.39%, East Asia and Paci c

AI Conference Publications (% of World Total)

40%

30%

22.93%, Europe and Central Asia

20%

18.93%, North America

10%

10.44%, South Asia

0%3.82%, Middle East and North Africa
2.83%, Latin America and the Caribbean
0.66%, Sub-Saharan Africa

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.14

Table of Contents

Chapter 1 Preview

29Artificial IntelligenceIndex Report 2022

By Geographic Area
In 2021, China produced the greatest share of the world’s
AI conference publications at 27.6%, opening an evengreater lead than in 2020, while the European Union plus
the United Kingdom followed at 19.0% and the UnitedStates came in third at 16.9% (Figure 1.1.15).

AI CONFERENCE PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

AI Conference Publications (% of World Total)

27.64%, China

25%

20%

18.95%, European Union and United Kingdom
16.90%, United States

15%

10%

5%

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.15

Table of Contents

Chapter 1 Preview

30Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

Citations
Despite China producing the most AI conference
publications in 2021, Figure 1.1.16 shows that the UnitedStates led among the major powers with respect tothe number of AI conference citations, with 29.5% in
2021, followed by the European Union plus the United
Kingdom (23.3%) and China (15.3%).

AI CONFERENCE CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

AI Conference Citations (% of World Total)

35%

30%

29.52%, United States25%
23.32%, European Union and United Kingdom

20%

15%

15.32%, China

10%

5%
0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.16

Table of Contents

Chapter 1 Preview

31Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

A I R E P O S I TO R I E S

work outside traditional avenues for publications. Those
repositories allow researchers to share their findings before
submitting them to journals and conferences, which greatly
accelerates the cycle of information discovery. The numberof AI repository publications grew almost 30 times in thepast 12 years (Figure 1.1.17), now account for 15.3% of all
repository publications (Figure 1.1.18).

Overview
Publishing pre-peer-reviewed papers on repositories ofelectronic preprints (such as arXiv and SSRN) has become
a popular way among AI researchers to disseminate their
NUMBER of AI REPOSITORY PUBLICATIONS, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

56.73

Number of AI Repository Publications (in thousands)

50

20

10

5

2

1
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.17

AI REPOSITORY PUBLICATIONS (% of TOTAL REPOSITORY PUBLICATIONS), 2010–21
AI Repository Publications (% of Total Repository Publications)

Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

15.30%

15%

10%

5%

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.18

Table of Contents

Chapter 1 Preview

32Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

By Region

repository publications in the world since 2014 while thatof Europe and Central Asia has declined. Since 2013, theshare of East Asia and Pacific has grown significantly.

The analysis by region in Figure 1.1.19 shows that North
America has maintained a steady lead in the share of AI

AI REPOSITORY PUBLICATIONS (% of WORLD TOTAL) by REGION, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

AI Repository Publications (% of World Total)

40%
35.89%, North America

30%
27.25%, Europe and Central Asia
26.38%, East Asia and Paci c

20%

10%
4.23%, South Asia
4.13%, Middle East and North Africa
1.67%, Latin America and the Caribbean
0.44%, Sub-Saharan Africa

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.19

Table of Contents

Chapter 1 Preview

33Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

By Geographic Area
While the United States has held the lead in the percentage
of AI repository publications in the world since 2011, China
is catching up while the European Union plus the United
Kingdom’s share continues to drop (Figure 1.1.20). In 2021,

the United States accounted for 32.5% of the world’s AI
repository publications—a higher percentage comparedto journal and conference publications, followed by theEuropean Union plus the United Kingdom (23.9%) andChina (16.6%).

AI REPOSITORY PUBLICATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2010–21

AI Repository Publications (% of World Total)

Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

32.52%, United States30%

23.92%, European Union and United Kingdom

20%
16.60%, China

10%

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.20

Table of Contents

Chapter 1 Preview

34Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

Citations
On the citations of AI repository publications, Figure 1.1.21
shows that the United States tops the list with 38.6%

of overall citations in 2021, establishing a dominantlead over the European Union plus the United Kingdom(20.1%) and China (16.4%).

AI REPOSITORY CITATIONS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

AI Repository Citations (% of World Total)

40%

38.60%, United States30%

20.13%, European Union and United Kingdom

20%

16.44%, China

10%

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.21

Table of Contents

Chapter 1 Preview

35Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

A I PAT E N T S

The number of patents
filed in 2021 is morethan 30 times higher
than in 2015, showing
a compound annual
growth rate of 76.9%.

This section draws on data from CSET and 1790 Analytics
on patents relevant to AI development and applications—indicated by Cooperative Patent Classification (CPC)/
International Patent Classification (IPC) codes andkeywords. Patents were grouped by country and year andthen counted at the “patent family” level, before CSET
extracted year values from the most recent publication
date within a family.

Overview
Figure 1.1.22 captures the number of AI patents filed from2010 to 2021. The number of patents filed in 2021 is morethan 30 times higher than in 2015, showing a compound
annual growth rate of 76.9%.
NUMBER of AI PATENT FILINGS, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report141.24

Number of AI Patents (in thousands)

140

120

100

80

60

40

20
0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.22

Table of Contents

Chapter 1 Preview

36Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

By Region and Application Status
Figure 1.1.23a breaks down AI patent filings by region.
The share of East Asia and Pacific took off in 2014 andled the rest of the world in 2021 with 62.1% of all patentapplications, followed by North America and Europe andCentral Asia. In terms of granted patents in those regions,North America leads with 57.0%, followed by East Asia
and Pacific (31.0%), and Europe and Central Asia (11.3%)(Figure 1.1.23b). The other regions combine to make up
roughly 1% of world patents (Figure 1.1.23c).

AI PATENT FILINGS (% of WORLD TOTAL) by REGION, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report70%

AI Patent Filings (% of World Total)

60%

62.14%, East Asia and Paci c

50%

40%

30%

20%

17.07%, North America

10%4.16%, Europe and Central Asia

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.23a

Table of Contents

Chapter 1 Preview

37Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

GRANTED AI PATENTS (% of WORLD TOTAL) by REGION, 2010–21 (Part 1)
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

80%

Granted AI Patents (% of World Total)

70%

60%
56.96%, North America

50%

40%
31.09%, East Asia and Paci c

30%

20%
11.27%, Europe and Central Asia

10%
0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.23b

GRANTED AI PATENTS (% of WORLD TOTAL) by REGION, 2010–21 (Part 2)
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

Granted AI Patents (% of World Total)

1.50%

1.25%

1.00%

0.75%

0.50%
0.35%, Latin America and the Caribbean

0.25%

0.25%, South Asia
0.08%, Middle East and North Africa

0.00%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.23c

Table of Contents

Chapter 1 Preview

38Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

By Geographic Area and Application Status
Trends revealed by the regional analysis can also be
observed in AI patent data broken down by geographic
area (Figure 1.1.24a and Figure 1.1.24b). China is now
filing over half of the world’s AI patents and being granted
about 6%, about the same as the European Union plus theUnited Kingdom. The United States, which files almost
all the patents in North America, does so at one-third therate of China. Figure 1.1.24c shows that compared to theincreasing numbers of AI patents applied and granted,
China has far greater numbers of patent applications(87,343 in 2021) than those granted (1,407 in 2021).

AI PATENT FILINGS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

AI Patent Filings (% of World Total)

60%

50%

51.69%, China

40%

30%

20%
16.92%, United States

10%
3.89%, European Union and United Kingdom

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.24a

GRANTED AI PATENTS (% of WORLD TOTAL) by GEOGRAPHIC AREA, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

Granted AI Patents (% of World Total)

70%

60%

50%

40%

39.59%, United States

30%

20%

10%

7.56%, European Union and United Kingdom
5.90%, China

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.1.24b

Table of Contents

Chapter 1 Preview

39Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.1 Publications

AI PATENTS by APPLICATION STATUS by GEOGRAPHIC AREA, 2010–21
Source: Center for Security and Emerging Technology, 2021 | Chart: 2022 AI Index Report

China
90
Application
Grant

Number of AI Patent Filings (in thousands)

80

European Union and United Kingdom

United States

87.34

70
60
50
40
30
19.61

20

9.45

10

4.88
1.81

1.41

0
2011

2016

2021

2011

2016

2021

2011

2016

2021
Figure 1.1.24c

Table of Contents

Chapter 1 Preview

40Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.2 Conferences

AI conferences are key venues for researchers to publish and communicate their work, as well as to connect with peers andcollaborators. Conference attendance is an indication of broader industrial and academic interest in a scientific field. In the past
20 years, AI conferences have grown not only in size but also in number and prestige. This section presents data on the trends inattendance at major AI conferences, covering more conferences (16) than previous Index reports.

1.2 CONFERENCESC O N F E R E N C E AT T E N DA N C E

Figure 1.2.1 shows that attendance at top AI conferencesin 2021 was relatively consistent with 2020, with morethan 88,000 participants worldwide. Figure 1.2.2 andFigure 1.2.3 show the attendance data for individual
conferences, with 16 major AI conferences split into
two categories: large AI conferences with over 2,500
attendees and small AI conferences with fewer than
2,500 attendees.6

Similar to 2020, most AI conferences were offeredvirtually in 2021. Only the International Conference on
Robotics and Automation (ICRA) and the Conferenceon Empirical Methods in Natural Language Processing(EMNLP) were held using a hybrid format. Conference
organizers reported measuring the exact attendance
numbers at a virtual conference is difficult, as virtual
conferences allow for higher attendance of researchers
from all around the world.

NUMBER of ATTENDEES at SELECT AI CONFERENCES, 2010–21
Source: Conference Data, 2021 | Chart: 2022 AI Index Report

90

88.76

Number of Attendees (in thousands)

80
70
60
50
40
30
20
10
0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.2.1

6 The International Conference on Machine Learning (ICML) used the number of session visitors as a proxy for the number of conference attendees, which explains the high attendance count in 2021.
The International Conference on Intelligent Robots and Systems (IROS) extended the virtual conference to allow users to watch events for up to three months, which explains the high attendance countin 2020. For the AAMAS conference, the attendance in 2020 is based on the number of users on site reported by the platform that recorded the talks and managed the online conference, while the 2021
number is for total registrants.

Table of Contents

Chapter 1 Preview

41Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.2 Conferences

ATTENDANCE at LARGE AI CONFERENCES, 2010–21
Source: Conference Data, 2021 | Chart: 2022 AI Index Report30

29.54, ICML

Number of Attendees (in thousands)

25

20
17.09, NeurIPS

15

10
8.24, CVPR
6.31, ICLR
5.01, ICCV
4.09, AAAI
3.76, EMNLP
2.59, IROS

5

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.2.2

ATTENDANCE at SMALL AI CONFERENCES, 2010–21

Number of Attendees (in thousands)

Source: Conference Data, 2021 | Chart: 2022 AI Index Report3

2.10, UAI

2

1.90, IJCAI
1.35, FaccT
1.08, AAMAS

1

1.00, ICRA
0.67, KR0.51, ICAPS

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.2.3

Table of Contents

Chapter 1 Preview

42Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.2 Conferences

WO M E N I N M AC H I N E L E A R N I N G( W I M L ) N E U R I P S WO R KS H O PWorkshop Participants
The number of participants attending the WiML workshop
has steadily increased since it was first introduced in2006. For the 2021 edition, Figure 1.2.4 shows an estimateof 1,486 attendees over all workshop sessions, counted
as the number of unique individuals who accessed thevirtual workshop platform at neurips.cc. The 2021 WiML
Workshop at NeurIPS happened as multiple sessions
over three days, which was a change in format from 2020.
As in 2020, the workshop was held virtually due to thepandemic.

Founded in 2006, Women in Machine Learning is anorganization dedicated to supporting and increasing
the impact of women in machine learning. This section
presents data from its annual technical workshop
colocated with NeurIPS. Starting in 2020, WiML has also
been hosting the Un-Workshop, which aims to advanceresearch via collaboration and interaction among
participants from diverse backgrounds at ICML.

ATTENDANCE at NEURIPS WOMEN in MACHINE LEARNING WORKSHOP, 2010–21
Source: Women in Machine Learning, 2021 | Chart: 2022 AI Index Report

1,486

1400

Number of Attendees

1200

1000

800

600

400

200
0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.2.4

Table of Contents

Chapter 1 Preview

43Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.2 Conferences

Demographics Breakdown

by Europe (19.9%), Asia (16.2%), and Africa (7.3%) (Figure
1.2.5). Figure 1.2.6 shows that Ph.D. students madeup almost half of the survey participants, while theshare of university faculty is around 1.2%. Researcher
scientists/engineers, data scientists/engineers, andsoftware engineers were among the most commonly held
professional positions.

This section shows the continent of residence andprofessional position breakdowns of the 2021 workshop
participants based on a survey filled by participants
who consented to have such information aggregated.
Among the survey respondents, more than half of thesurvey respondents were from North America, followed

CONTINENT of RESIDENCE of PARTICIPANTS at NEURIPS WOMEN in MACHINE LEARNING WORKSHOP, 2021
Source: Women in Machine Learning, 2021 | Chart: 2022 AI Index Report

North
America

53.40%

19.90%

Europe16.20%

Asia

7.30%

AfricaSouth
America

1.00%

Australia

1.00%

0%

10%

20%

30%

40%

50%

% of Respondents

Figure 1.2.5

PROFESSIONAL POSITIONS of PARTICIPANTS at NEURIPS WOMEN in MACHINE LEARNING WORKSHOP, 2021
Source: Women in Machine Learning, 2021 | Chart: 2022 AI Index Report

49.40%PhD Student
16.50%MSc Student

14.20%

Research Scientist / Engineer
10.20%

Undergraduate Student
7.40%

Data Scientist / Engineer

5.70%

Postdoctoral Researcher
2.30%

Software Engineer
High School Student

0.60%

Professor (Post-Tenure)

0.60%

Professor (Pre-Tenure)

0.60%

0%

10%

20%

30%
% of Respondents

Table of Contents

Chapter 1 Preview

40%

50%
Figure 1.2.6

44

Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.3 AI Open-Source Software Libraries

A software library is a collection of computer code that is used to create applications and products. Popular AI-specific software
libraries—such as TensorFlow and PyTorch—help developers create their AI solutions quickly and efficiently. This section analyzes the
popularity of software libraries through GitHub data.

1.3 AI OPEN-SOURCESOFTWARE LIBRARIESG I T H U B S TA R S
Figures 1.3.1 and 1.3.2 reflect the number of users of
GitHub open-source AI software libraries from 2015 to
2021. TensorFlow remained by far the most popular in
2021, with around 161,000 cumulative GitHub stars—aslight increase over 2020. TensorFlow was about threetimes as popular in 2021 as the next-most-starred

GitHub open-source AI software library, OpenCV, which
was followed by Keras, PyTorch, and Scikit-learn.
Figure 1.3.2 shows library popularity for libraries withfewer than 40,000 GitHub stars—led by FaceSwap witharound 40,000 stars, followed by 100-Days-Of-ML-Code,
AiLearning, and BVLC/caffe.

NUMBER of GITHUB STARS by AI LIBRARY (OVER 40K STARS), 2014–21
Source: GitHub, 2021 | Chart: 2022 AI Index Report

160.7, TensorFlow

Number of Cumulative GitHub Stars (in thousands)

160
140
120
100
80

58.6, OpenCV
53.2, Keras
52.7, PyTorch
48.0, Scikit-learn
46.7, DeepLearning-500-questions
41.5, TenserFlow-Examples

60
40
20
0
2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.3.1

Table of Contents

Chapter 1 Preview

45Artificial IntelligenceIndex Report 2022

CHAPTER 1: RESEARCH & DEVELOPMENT
1.3 AI Open-Source Software Libraries

NUMBER of GITHUB STARS by AI LIBRARY (UNDER 40K STARS), 2014–21
Source: GitHub, 2021 | Chart: 2022 AI Index Report

39.88, faceswap

Number of Cumulative GitHub Stars (in thousands)

40

33.58, 100-Days-Of-ML-Code32.27, AiLearning
32.14, BVLC/ca e
32.02, Real-Time-Voice-Cloning
32.00, deeplearningbook-chinese
31.28, Deep Learning Papers Reading Roadmap
30.26, DeepFaceLab

35
30
25
20
15
10
5
0
2014

2015

2016

2017

2018

2019

2020

2021
Figure 1.3.2

Table of Contents

Chapter 1 Preview

46Artificial IntelligenceIndex Report 2022

CHAPTER 2:

Technical
Performance

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCECHAPTER 2:

Chapter Preview
Overview

50

Chapter Highlights

51

2.1 COMPUTER VISION—IMAGE

52

Image Classification

52

ImageNet

52

ImageNet: Top-1 Accuracy

52

ImageNet: Top-5 Accuracy

52

Image Generation

54

	STL-10: Fréchet Inception
Distance (FID) Score

54

CIFAR-10: Fréchet Inception
Distance (FID) Score

55

Medical Image Segmentation

61

CVC-ClinicDB and Kvasir-SEG

61

Face Detection and Recognition

62

National Institute of Standards
and Technology (NIST) Face
Recognition Vendor Test (FRVT)

62

Face Detection: Effects of Mask-Wearing

63

Face Recognition Vendor Test (FRVT):
Face-Mask Effects

63

Highlight: Masked Labeled Faces
in the Wild (MLFW)

64

Visual Reasoning

65

Visual Question Answering(VQA) Challenge

65

Deepfake Detection

56

FaceForensics++

56

2.2 COMPUTER VISION—VIDEO

67

Celeb-DF

57

Activity Recognition

67

Human Pose Estimation

57

Kinetics-400, Kinetics-600, Kinetics-700 67

Leeds Sports Poses: Percentage
of Correct Keypoints (PCK)58

ActivityNet: Temporal Action
Localization Task

Human3.6M: Average Mean
Per Joint Position Error (MPJPE)

59

Semantic Segmentation
Cityscapes

Object Detection
60
60

69
70

Common Object in Context (COCO)71

You Only Look Once (YOLO)

72

Visual Commonsense Reasoning (VCR)

73

ACCESS THE PUBLIC DATA

Table of Contents

48

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCECHAPTER 2: CHAPTER PREVIEW (CONT’D)
2.3 LANGUAGE

74

2.4 SPEECH

86

English Language Understanding74

Speech Recognition

86

SuperGLUE

74

Transcribe Speech: LibriSpeech(Test-Clean and Other Datasets)

86

VoxCeleb87

Stanford Question Answering
Dataset (SQuAD)

75

Reading Comprehension Dataset
Requiring Logical Reasoning (ReClor)

76

2.5 RECOMMENDATION

88

78

88

Text Summarization
arXiv

78

Commercial Recommendation:
MovieLens 20M

PubMed79

Click-Through Rate Prediction: Criteo

89

2.6 REINFORCEMENT LEARNING

90

Reinforcement Learning Environments

90

Natural Language Inference
Stanford Natural Language
Inference (SNLI)
Abductive Natural Language
Inference (aNLI)
Sentiment Analysis
SemEval 2014 Task 4 Sub Task 2
Machine Translation (MT)8080

Arcade Learning Environment: Atari-57 90
81

Procgen

91

82

Human Games: Chess

93

82
83

WMT 2014, English-German
and English-French

84

Number of Commercially
Available MT Systems

85

ACCESS THE PUBLIC DATA

Table of Contents

2.7 HARDWARE

94

MLPerf: Training Time94

MLPerf: Number of Accelerators

96

IMAGENET: Training Cost97

2.8 ROBOTICS

98

Price Trends in Robotic Arms

98

AI Skills Employed by Robotics Professors 99

49

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE

Overview
This year, the technical performance chapter includes more analysis
than ever before of the technical progress in various subfields ofartificial intelligence, including trends in computer vision, language,
speech, recommendation, reinforcement learning, hardware, androbotics. It uses a number of quantitative measurements, from
common AI benchmarks and prize challenges to a field-wide survey,
to highlight the development of top-performing AI systems.

Table of Contents

Chapter 2 Preview

50Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCECHAPTER HIGHLIGHTS• Data, data, data: Top results across technical benchmarks have increasingly relied on the use of
extra training data to set new state-of-the-art results. As of 2021, 9 state-of-the-art AI systems
out of the 10 benchmarks in this report are trained with extra data. This trend implicitly favors
private sector actors with access to vast datasets.
• Rising interest in particular computer vision subtasks: In 2021, the research community saw
a greater level of interest in more specific computer vision subtasks, such as medical image
segmentation and masked-face identification. For example, only 3 research papers tested
systems against the Kvasir-SEG medical imaging benchmark prior to 2020. In 2021, 25
research papers did. Such an increase suggests that AI research is moving toward research that
can have more direct, real-world applications.
• AI has not mastered complex language tasks, yet: AI already exceeds human performance levels
on basic reading comprehension benchmarks like SuperGLUE and SQuAD by 1%–5%. Although
AI systems are still unable to achieve human performance on more complex linguistic tasks
such as abductive natural language inference (aNLI), the difference is narrowing. Humans
performed 9 percentage points better on aNLI in 2019. As of 2021, that gap has shrunk to 1.
• Turn toward more general reinforcement learning: For the last decade, AI systems havebeen able to master narrow reinforcement learning tasks in which they are asked to maximize
performance in a specific skill, such as chess. The top chess software engine now exceeds
Magnus Carlsen’s top ELO score by 24%. However, in the last two years AI systems have also
improved by 129% on more general reinforcement learning tasks (Procgen) in which theymust operate in novel environments. This trend speaks to the future development of AI systems
that can learn to think more broadly.
• AI becomes more affordable and higher performing: Since 2018, the cost to train an image
classification system has decreased by 63.6%, while training times have improved by 94.4%.
The trend of lower training cost but faster training time appears across other MLPerf task
categories such as recommendation, object detection and language processing, and favors themore widespread commercial adoption of AI technologies.
• Robotic arms are becoming cheaper: An AI Index survey shows that the median price ofrobotic arms has decreased by 46.2% in the past five years—from $42,000 per arm in 2017
to $22,600 in 2021. Robotics research has become more accessible and affordable.

Table of Contents

Chapter 2 Preview

51Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

Computer vision is the subfield of AI that teaches machines to understand images and videos. There is a wide range of computer
vision tasks, such as image classification, object recognition, semantic segmentation, and face detection. As of 2021, computers canoutperform humans on a plethora of computer vision tasks. Computer vision technologies have a variety of important real-world
applications, such as autonomous driving, crowd surveillance, sports analytics, and video-game creation.

2.1 COMPUTER VISION—IMAGEI M AG E C L A S S I F I CAT I O N​​
Imageclassification refers to the ability of machines tocategorize what they see in images (Figure 2.1.1). In apractical sense, image recognition systems can help cars
identify objects in their surroundings, doctors detecttumors, and factory managers spot production defects.
The past decade has seen tremendous advances in thetechnical capacity of image recognition systems, especiallyas researchers have embraced more machine learningtechniques. Moreover, progress in algorithmic, hardware,and data technologies has meant that image recognition
has become more affordable, widely applicable, andaccessible than ever before.

A DEMONSTRATION OF IMAGE CLASSIFICATION
Source: Krizhevsky, 2020

ImageNet
ImageNet is a database that includes over 14 million images
across 20,000 categories publicly available to researchers
working on image classification problems. Created in 2009,
ImageNet is now one of the most common ways scientists
benchmark improvement on image classification.

ImageNet: Top-1 Accuracy
Benchmarking on ImageNet is measured through accuracy
metrics, which quantify how frequently AI systems assign
the right labels to the given images. Top-1 accuracy
measures the rate at which the top prediction made bya classification model for a given image matches theimage’s actual target label. In recent years, it has become
increasingly common to improve the performance ofsystems on ImageNet by pretraining them with additional
data from other image datasets.
As of late 2021, the top image classification system makes
on average 1 error for every 10 classification attempts
on Top-1 accuracy compared to an average of 4 errors
for every 10 attempts in late 2012 (Figure 2.1.2). In 2021,

Table of Contents

Chapter 2 Preview

Figure 2.1.1

the top pretrained system was CoAtNets, produced byresearchers on the Google Brain Team.

ImageNet: Top-5 Accuracy
Top-5 accuracy considers whether any of the model’s 5highest probability answers align with the image label. As
highlighted in Figure 2.1.3, AI systems presently achieve
near perfect Top-5 estimation. Currently, the state-of-theart performance on Top-5 accuracy with pretraining is
99.0%, achieved in November 2021 by Microsoft Cloud andMicrosoft AI’s Florence-CoSwim-H model.
Improvements in Top-5 accuracy on ImageNet seem to beplateauing, which is perhaps unsurprising. If your system
is classifying correctly 98 or 99 out of 100 times, there is
only so much higher you can go.

52

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

IMAGENET CHALLENGE: TOP-1 ACCURACY
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

90.88%, With Extra Training Data

90%

Top-1 Accuracy (%)

87.80%, Without Extra Training Data

80%

70%

60%
2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.2

IMAGENET CHALLENGE: TOP-5 ACCURACY
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

100%

99.02%, With Extra Training Data

97.90%, Without Extra Training Data

Top-5 Accuracy (%)

95%

94.90 %, Human Baseline

90%

85%

80%
2012

Table of Contents

2013

2014

Chapter 2 Preview

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.3

53

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–ImageI M AG E G E N E R AT I O NImage generation is the task of generating images that are
indistinguishable from real ones. Image generation can be
widely useful in generative domains where visual content
has to be created, for example entertainment (companies
like NVIDIA have already used image generators to create
virtual worlds for gaming), fashion (designers can let AI

systems generate different design patterns), and healthcare(image generators can synthetically create novel drug
compounds). Figure 2.1.4 illustrates progress made in
image generation by presenting several human faces thatwere synthetically generated by AI systems in the last year.

GAN PROGRESS ON FACE GENERATION
Source: Goodfellow et al., 2014; Radford et al., 2016; Liu & Tuzel, 2016;
Karras et al., 2018; Karras et al., 2019; Goodfellow, 2019; Karras et al.,2020; AI Index, 2021; Vahdat et al., 2021

2021

STL-10: Fréchet Inception Distance (FID) Score
The Fréchet Inception Distance score tracks the similarity
between an artificially generated set of images and thereal images from which it was generated. A low scoremeans that the generated images are more similar to
the real ones, and a score of zero indicates that the fakeimages are identical to the real ones.

Figure 2.1.4

Figure 2.1.5 documents the gains generative models havemade in FID on the STL-10 dataset, one of the most widely
cited datasets in computer vision. The state-of-the-art
model on STL-10 developed by researchers at the Korea
Advanced Institute of Science and Technology as well as the
University of Seoul posted a FID score of 7.7, significantly
better than the state-of-the-art result from 2020.

STL-10: FRÉCHET INCEPTION DISTANCE (FID) SCORE

Fréchet Inception Distance (FID) Score

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

30

20

10
7.71

0
2018

2019

2020

2021
Figure 2.1.5

Table of Contents

Chapter 2 Preview

54Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–ImageCIFAR-10: Fréchet Inception Distance (FID) Score
Progress on image generation can also be benchmarked
on CIFAR-10, a dataset of 60,000 color images across 10
different object classes. The state-of-the-art results on
CIFAR-10 posted in 2021 were achieved by researchers
from NVIDIA.

The FID scores achieved by the top image generation
models are much lower on CIFAR-10 than STL-10. Thisdifference is likely attributable to the fact that CIFAR-10
contains images of much lower resolution (32 x 32 pixels)
than those on STL-10 (96 x 96 pixels).

CIFAR-10: FRÉCHET INCEPTION DISTANCE (FID) SCORE
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

Fréchet Inception Distance (FID) Score

30

20

10

2.10

0
2017

2018

2019

2020

2021
Figure 2.1.6

Table of Contents

Chapter 2 Preview

55Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

D E E P FA K E D E T E CT I O N

dataset that contains approximately 1,000 original video
sequences sourced from YouTube videos. Progress on
FaceForensics++ is measured in terms of accuracy: thepercentage of altered images an algorithm can correctly
identify.

Many AI systems can now generate fake images that are
indistinguishable from real ones. A related technology
involves superimposing one person’s face onto another,creating a so-called “deepfake.” Deepfakes are used
for purposes ranging from advertising to generating
misogynistic pornography and disinformation (in 2018,
for example, a deepfake video of Barack Obama uttering
profanities about Donald Trump was circulated online over2 million times). In the last few years, AI researchers havesought to keep up with improving deepfake technologies by
crafting stronger deepfake detection algorithms.

Although FaceForensics++ was introduced in 2019,researchers have tested previously existing deepfake
detection methods on the dataset in order to trackprogress over time in deepfake detection (Figure 2.1.7).
In the last decade, AI systems have become better andbetter at detecting deepfakes. In 2012, the top-performing
systems could correctly identify 69.9% of deepfakes
across all four FaceForenics++ datasets. In 2021, that
number increased to 97.7%.1

FaceForensics++
FaceForensics++ is a deepfake detection benchmarking

FACEFORENSICS++: ACCURACY
Source: arXiv, 2021 | Chart: 2022 AI Index Report

99.98%, Face2Face
99.47%, DeepFake
98.27%, FaceSwap

100%

93.25%, NeuralTextures

Accuracy (%)

90%

80%

70%

60%
2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.7

1 These numbers were taken by averaging performance across all four FaceForensics++ datasets.

Table of Contents

Chapter 2 Preview

56Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

Celeb-DF
The Celeb-DF deepfake detection dataset is composed of
590 original celebrity YouTube videos manipulated into
5,639 deepfakes. Celeb-DF was introduced in 2019. In
2021, the top score on Celeb-DF was 76.9 and came fromresearchers at the University of Science and Technology ofChina and Alibaba group (Figure 2.1.8).

Top detection models perform significantly worse (by 20
percentage points) on Celeb-DF than FaceForensics++,suggesting that Celeb-DF is a more challenging dataset totest out techniques on. As deepfake technologies continue
to improve in the upcoming years, it will be important
to continue monitoring progress on Celeb-DF and othersimilarly challenging deepfake detection datasets.

CELEB-DF: AREA UNDER CURVE SCORE (AUC)
Source: arXiv, 2021 | Chart: 2022 AI Index Report80

Area Under Curve Score (AUC)

76.88

75

70

65

60
2018

2019

2020

2021
Figure 2.1.8

H U M A N P O S E E S T I M AT I O N
Human pose estimation is the task of estimating different
positions of human body joints (arms, head, torso, etc.)from a single image (Figure 2.1.9), and then combining theseestimates to correctly label the pose the human is taking.

Human pose estimation can be used to facilitate activity
recognition for purposes such as sports analytics, crowd
surveillance, CGI development, virtual environment design,and transportation (for example, identifying the bodylanguage signs of an airport runway controller).

A DEMONSTRATION OF
HUMAN POSE ESTIMATION
Source: Cao et al., 2019

Figure 2.1.9

Table of Contents

Chapter 2 Preview

57Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

Leeds Sports Poses: Percentage of Correct
Keypoints (PCK)In 2021, the top-performing human pose estimation
model correctly identified 99.5% of keypoints on Leeds
Sports Poses (Figure 2.1.10). Given that maximum
performance on Leeds Sports is 100.0%, more challenging
benchmarks for human pose estimation will have to be
developed, as we are close to saturating the benchmark.

The Leeds Sports Poses dataset contains 2,000 images
collected from Flickr of athletes playing a sport. Each
image includes information on 14 different body joint
locations. Performance on the Leeds Sports Poses
benchmark is assessed by the percentage of correctly
estimated keypoints.

LEEDS SPORTS POSES: PERCENTAGE of CORRECT KEYPOINTS (PCK)
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

99.50%

Percentage of Correct Keypoints (PCK)

100%

90%

80%

70%
2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.10

Table of Contents

Chapter 2 Preview

58Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

Human3.6M: Average Mean Per Joint Position
Error (MPJPE)
3D human pose estimation is a more challenging type ofpose estimation, where AI systems are asked to estimate
poses in a three- rather than two-dimensional space. The
Human3.6M dataset tracks progress in 3D human pose
estimation. Human3.6M is a collection of over 3.6 million
images of 17 different types of human poses (talking on
the phone, discussing, and smoking, etc.). Performanceon Human3.6M is measured in average mean per joint
position error in millimeters, which is the average difference
between an AI model’s position estimations and the actual
position annotation.
In 2014, the top-performing model was making an average
per joint error of 16 centimeters, half the size of a standard
school ruler. In 2021, this number fell to 1.9 centimeters,less than the size of an average paper clip.

HUMAN3.6M: AVERAGE MEAN PER JOINT POSITION ERROR (MPJPE)
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

150

Average MPJPE (mm)

120

90

60

22.70, Without Extra Training Data30

18.70, With Extra Training Data

0
2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.11

Table of Contents

Chapter 2 Preview

59Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–ImageS E M A N T I C S E G M E N TAT I O N

A DEMONSTRATION OF SEMANTIC SEGMENTATION
Source: Visual Object Classes Challenge, 2012

Semantic segmentation is the task of assigningindividual image pixels a category (such as person,
bicycle, or background) (Figure 2.1.12). A plethoraof real world domains require pixel-level image
segmentation such as autonomous driving (identifying
which parts of the image a car sees are pedestrians andwhich parts are roads), image analysis (distinguishing
the foreground and background in photos), andmedical diagnosis (segmenting tumors in lungs).

Figure 2.1.12

Cityscapes
The Cityscapes dataset contains images of urban street
environments from 50 cities, taken during the daytime indifferent seasons, and it allows for evaluation on a wide
range of semantic segmentation tasks (instance-level,
panoptic, and 3D vehicle).
The task to which most researchers submit is pixel-level
semantic labeling, a challenge in which AI systems must
semantically label an image on a per-pixel level. Challengers
are evaluated on the intersection-over-union (IoU)

metric, with a higher IoU score corresponding to better
segmentation accuracy. In practical terms, a higher scoremeans that a greater proportion of the image segments
predicted by the model overlap with the image’s actual
segments.
The top-performing AI systems on Cityscapes in 2021
reported scores that are 14.6 percentage points higher thanthose in 2015. As with other computer vision tasks, the top
performing models on Cityscapes have been pretrained on
additional training data in the last few years.

CITYSCAPES CHALLENGE, PIXEL-LEVEL SEMANTIC LABELING TASK: MEAN INTERSECTION-OVER-UNION (IOU)
Source: Cityscapes Challenge, 2021 | Chart: 2022 AI Index Report90%

Mean Intersection-Over-Union (mIoU)

86.20%, With Extra Training Data

84.30%, Without Extra Training Data

80%

70%

60%
2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.1.13

Table of Contents

Chapter 2 Preview

60Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–ImageM E D I CA L I M AG E S E G M E N TAT I O N

A DEMONSTRATION OF KIDNEY SEGMENTATION

Medical image segmentation refers to the ability of AI
systems to segment objects of interest, such as organs,lesions, or tumors, in medical images (Figure 2.1.14).
Technical progress in this task is vital to streamlining
medical diagnoses. Advances in medical image
segmentation mean doctors can spend less time on
diagnosis and more time treating patients.

Source: Kidney and Kidney Tumor Segmentation, 2021

CVC-ClinicDB and Kvasir-SEG
CVC-ClinicDB is a dataset that includes over 600 highresolution images from 31 colonoscopies. Kvasir-SEG is a
public dataset of 1,000 high-resolution gastrointestinal polypimages that were manually segmented by doctors and crossverified by professional gastroenterologists. Both datasets
are used to track progress in medical image segmentation.
Performance is measured in mean DICE, which represents theaverage overlap between the polyp segments identified by an
AI system and the actual polyp segments.
AI systems are now capable of correctly segmenting
colonoscopy polyps at a rate of 94.2% on CVC-ClinicDB,
representing an 11.9 percentage point improvement
since 2015, and a 1.8 percentage improvement since

Figure 2.1.14

2020 (Figure 2.1.15). Similar progress has been made on
Kvasir-SEG, where presently the top-performing AI model
can accurately segment gastrointestinal polyps at a rateof 92.2%. The top results on both the CVC-ClinicDB andKvasir-SEG benchmarks were achieved by the MSRF-Net
model, one of the first convolutional neural networks
designed specifically for medical image segmentation.

CVC-CLINICDB: MEAN DICEKVASIR-SEG: MEAN DICE

Source: Papers with Code, 2021; arXiv, 2021 |Chart: 2022 AI Index Report

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

95%

95%

94.20%

92.17%

Mean DICE

90%

Mean DICE

90%

85%

85%

80%

80%
2015

2017

2019

2021
Figure 2.1.15a

Table of Contents

Chapter 2 Preview

2015

2017

2019

2021
Figure 2.1.15b61

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

The Kvasir-SEG benchmark also points to the explosion ofinterest in medical image segmentation. Prior to 2020, thedataset was referenced in only three academic papers. In
2020 that number rose to six, and in 2021 it shot up to 25.
Last year also saw the hosting of KiTS21 (the Kidney andKidney Tumor Segmentation Challenge), which challenged
medical researchers from academia and industry to create
the best systems for automatically segmenting renal
tumors and the surrounding anatomy of kidneys.

FACE DETECTION AND RECOGNITION
In facial detection, AI systems are tasked with identifying
individuals in images or videos. Although facial recognition
technology has existed for several decades, the technicalprogress in the last few years has been significant. Some of
today’s top-performing facial recognition algorithms have anear 100% success rate on challenging datasets.
Facial recognition can be used in transportation to facilitatecross-border travel, in fraud prevention to protect sensitive
documents, and in online proctoring to identify illicit
examination behavior. The greatest practical promise of
facial recognition, however, is in its potential to aid security,

which makes the technology extremely appealing tomilitaries and governments all over the world (e.g., 18 outof 24 U.S. government agencies are already using somekind of facial recognition technology).

National Institute of Standards and Technology(NIST) Face Recognition Vendor Test (FRVT)The National Institute of Standards and Technology’s
Face Recognition Vendor Test measures how well facialrecognition algorithms perform on a variety of homeland
security and law enforcement tasks, such as facerecognition across photojournalism images, identificationof child trafficking victims, deduplication of passports,
and cross-verification of visa images. Progress on facialrecognition algorithms is measured according to the false
non-match rate (FNMR) or the error rate (the frequency
with which a model fails to match an image to a person).
In 2017, some of the top-performing facial recognition
algorithms had error rates of over 50.0% on certain FRVT
tests. As of 2021, none has posted an error rate greater than
3.0%. The top-performing model across all datasets in 2021
(visa photos) registered an error rate of 0.1%, meaning thatfor every 1,000 faces, the model correctly identified 999.

NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY (NIST) FACE RECOGNITION VENDOR TEST (FRVT):VERIFICATION ACCURACY by DATASET
Source: National Institute of Standards and Technology, 2021 | Chart: 2022 AI Index Report

1

False Non-Match Rate: FMNR (Log-Scale)

0.5

0.2
0.1
0.05
0.0297, WILD Photos FNMR @ FMR

0.00001

0.02
0.01
0.005

0.0044, BORDER Photos FNMR @ FMR = 0.000001
0.0023, VISABORDER Photos FNMR@FMR 0.000001
0.0022, MUGSHOT Photos FNMR @ FMR 0.00001
0.0021, MUGSHOT Photos FNMR @ FMR 0.00001 DT>=12 YRS
0.0013, VISA Photos FNMR @ FMR 0.000001

0.002
0.001
2017

2018

2019

2020

2021
Figure 2.1.16

Table of Contents

Chapter 2 Preview

62Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

FACE DETECTION:
EFFECTS OF MASK-WEARING
Face Recognition Vendor Test (FRVT):
Face-Mask Effects
Facial recognition has become a more challenging
task with the onset of the COVID-19 pandemic andaccompanying mask mandates. The face-mask effects test
asks AI models to identify faces on two datasets of visa
border photos, one of which includes masked faces, theother which does not.
Three important trends can be gleaned from the FRVT
face-mask test: (1) Facial recognition systems still perform
relatively well on masked faces; (2) the performance onmasked faces is worse than on non-masked faces; and (3)the gap in performance has narrowed since 2019.

Although facial recognition
technology has existed
for several decades, thetechnical progress in thelast few years has beensignificant. Some of today’s
top-performing facialrecognition algorithms havea near 100% success rate on
challenging datasets.

NIST FRVT FACE MASK EFFECTS: FALSE-NON MATCH RATE
Source: National Institute of Standards and Technology, 2021 | Chart: 2022 AI Index Report

False Non-Match Rate: FMNR (Log-Scale)

0.025

0.020

0.015
0.014, Masked

0.010

0.005
0.002, Non-masked

0.000
2019

2020

2021
Figure 2.1.17

Table of Contents

Chapter 2 Preview

63Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

Masked Labeled Faces in the Wild (MLFW)
In 2021, researchers from the
Beijing University of Posts andTelecommunications released afacial recognition dataset of 6,000
masked faces in response to the newrecognition challenges posed bylarge-scale mask-wearing.

EXAMPLES OF MASKED FACES IN
THE MASKED LABELED FACES IN
THE WILD (MLFW) DATABASE
Source: Wang et al., 2021

Figure 2.1.18

As part of the dataset release, the researchers ran a series of existing state-of-the-art detection
algorithms on a variety of facial recognition datasets, including theirs, to determine how much detection
performance decreased when faces were masked. Their estimates suggest that top methods perform 5
to 16 percentage points worse on masked faces compared to unmasked ones. These findings somewhat
confirm the insights from the FRVT face-mask tests: Performance deteriorates when masks are included,
but not by an overly significant degree.
STATE-OF-THE-ART FACE DETECTION METHODS on MASKED LABELED FACES IN THE WILD (MLFW): ACCURACY
Source: Wang et. al, 2021 | Chart: 2022 AI Index Report

Face Detection Method / Dataset
ArcFace1

ArcFace3

Arcface4

CosFace2Curricularface5

SFace6

100%
91%

90%

90%

Accuracy (%)

85%

91%

83%

80%
75%

70%

60%

MLFW

SLLFW8LFW7CPLFW10CALFW11MLFW

SLLFW8LFW7CALFW11CPLFW10

MLFW

LFW7

SLLFW8CPLFW10MLFW

CALFW11SLLFW8LFW7CPLFW10CALFW11MLFW

SLLFW8LFW7CALFW11CPLFW10

MLFW

LFW7

SLLFW8CPLFW10CALFW1150%

Figure 2.1.19

Table of Contents

Chapter 2 Preview

64Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–ImageVISUAL REASONING
Visual reasoning assesses how well
AI systems can reason across a
combination of visual and textual
data. Visual reasoning skills are
essential in developing AI that canreason more broadly. Existing AI canalready execute certain narrow visualtasks better than humans, such asclassifying images, detecting faces,and segmenting objects. But many AI
systems struggle when challenged toreason more abstractly—for example,
generating valid inferences about theactions or motivations of agents in animage (Figure 2.1.20).

AN EXAMPLE OF A VISUAL REASONING TASK
Source: Goyal et al., 2021

Visual Question Answering(VQA) Challenge

Figure 2.1.20

SAMPLE QUESTIONS IN THE VISUAL QUESTIONANSWERING (VQA) CHALLENGE
Source: Goyal et al., 2017

In the Visual Question Answering
challenge, AI systems are tasked withanswering open-ended questions
about images (Figure 2.1.21). To
answer the questions at a high level,
AI systems must have a combined
understanding of language, vision,and commonsense reasoning.

Figure 2.1.21

Table of Contents

Chapter 2 Preview

65Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.1 Computer Vision–Image

In the six years since the VQA challenge began, there has
been a 24.4 absolute percentage point improvement instate-of-the-art performance. In 2015, the top-performing

systems could correctly answer only 55.4% of questions(Figure 2.1.22). As of 2021, top performance stood at
79.8%—close to the human baseline of 80.8%.

VISUAL QUESTION ANSWERING (VQA) CHALLENGE: ACCURACY
Source: VQA Challenge, 2021 | Chart: 2022 AI Index Report

80.80%, Human Baseline80%

Accuracy (%)

79.78%

70%

60%

50%
2015

2016

2017

2018

2019

2020

2021
Figure 2.1.22

Table of Contents

Chapter 2 Preview

66Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

Video analysis concerns reasoning or task operation across sequential frames (videos), rather than single frames (images). Video
computer vision has a wide range of use cases, which include assisting criminal surveillance efforts, sports analytics, autonomous
driving, navigation of robots, and crowd monitoring.

2.2 COMPUTER VISION—VIDEOAC T I V I T Y R E C O G N I T I O N​​ fundamental subtask in video computer vision isA
activity recognition: identifying the activities that occur
in videos. AI systems have been challenged to classify
activities that range from simple actions, like walking,waving, or standing, to ones that are more complex andcontain multiple steps, like preparing a salad (which
requires an AI system to recognize and chain together
discrete actions like cutting tomatoes, washing thegreens, applying dressing, etc.)

Kinetics-400, Kinetics-600, Kinetics-700
Kinetics-400, Kinetics-600, and Kinetics-700 are a series ofdatasets for benchmarking video activity recognition. Eachdataset includes 650,000 large-scale, high-quality video
clips from YouTube that display a wide range of humanactivities, and asks AI systems to classify an action from apossible set of 400, 600, and 700 categories, respectively.
Some of the new and more challenging activity classes
added to the Kinetics-700 series include pouring wine,
playing the oboe, and making latte art.

EXAMPLE CLASSES FROM THE KINETICS DATASET
Source: Kay et al., 2017

Figure 2.2.1

Table of Contents

Chapter 2 Preview

67Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

As of 2022, one model tops all three Kinetics datasets.
MTV, a collaboration between Google Research, Michigan
State University, and Brown University, released in January2022, achieved a 89.6% Top-1 accuracy on the 600 series,
89.1% accuracy on the 400 series, and 82.20% accuracy
on the 700 series (Figure 2.2.2). The most striking aspect
about technical progress on Kinetics is how rapidly the gap

has narrowed between performance on the datasets. In
2020, the gap between performance on Kinetics-400 andKinetics-700 was 27.14 percentage points. In one shortyear, that gap has narrowed to 7.4 points, which meansthat performance on the newer, harder dataset is occurring
more rapidly than performance on the easier dataset andsuggests that the easier ones are starting to asymptote.

KINETICS-400, KINETICS-600, KINETICS-700: TOP-1 ACCURACY
Source: Papers with Code, 2021; arXIv, 2021 | Chart: 2022 AI Index Report

89.60%, Kinetics-600
89.10%, Kinetics-400

90%82.20%, Kinetics-700

Top-1 Accuracy (%)

80%

70%

60%

50%
2016

2017

2018

2019

2020

2021

2022
Figure 2.2.2

Table of Contents

Chapter 2 Preview

68Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

ActivityNet: Temporal Action Localization Task
ActivityNet is a video dataset for human activity
understanding that contains 700 hours of videos ofhumans doing 200 different activities (long jump, dog
walking, vacuuming, etc.). For an AI system to successfullycomplete the ActivityNet Temporal Action Localization
Task (TALT) task, it has to execute two separate steps: (1)localization (identify the precise interval during which theactivity occurs); and (2) recognition (assign the correct
category label). Temporal action localization is one of

the most complex and difficult tasks in computer vision.
Performance on TALT is measured in terms of mean average
precision, with a higher score indicating greater accuracy.
As of 2021 the top-performing model on TALT, developed
by HUST-Alibaba, scores 44.7%, a 26.9 percentage point
improvement over the top scores posted in 2016 when thechallenge began (Figure 2.2.3). Although state-of-the-art
results on the task have been posted for each subsequent
year, the gains have become increasingly small.

ACTIVITYNET, TEMPORAL ACTION LOCALIZATION TASK: MEAN AVERAGE PRECISION (mAP)
Source: ActivityNet, 2021 | Chart: 2022 AI Index Report

44.67%

Mean Average Precision (mAP)

40%

30%

20%

2016

2017

2018

2019

2020

2021
Figure 2.2.3

Table of Contents

Chapter 2 Preview

69Artificial IntelligenceIndex Report 2022

O B J E C T D E T E CT I O N
Object detection is the task of identifying objects within
an image (Figure 2.2.4). There are different philosophies
bearing on priority, speed, and accuracy that guide thedesign of object detection systems. Systems that train
quickly might be more efficient but are less accurate.
Those that are more accurate might perform better but

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

take longer to process a video. This tradeoff between
speed and accuracy is also reflected in the types of object
detection methods pioneered in the last decade. There
are one-stage methods which prioritize speed, such as
SSD, RetinaNet, and YOLO, and two-stage methods whichprioritize accuracy, such as Mask R-CNN, Faster R-CNN,
and Cascade R-CNN.

A DEMONSTRATION OF HOW OBJECT DETECTION APPEARS TO AI SYSTEMS
Source: COCO, 2020

Figure 2.2.4

Table of Contents

Chapter 2 Preview

70Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

Common Object in Context (COCO)
Microsoft’s Common Object in Context (COCO) object
detection dataset contains over 328,000 images across morethan 80 object categories. There are many accuracy metrics
used to track performance on object detection, but for thesake of consistency, this section and the majority of thisreport considers mean average precision (mAP50).

Since 2016, there has been a 23.8 percentage point
improvement on COCO object detection, with this year’s
top model, GLIP, registering a mean average precision of
79.5%.2 Figure 2.2.5 illustrates how the use of extra training
data has taken over object detection, much as it has withother domains of computer vision.

COCO-TEST-DEV: MEAN AVERAGE PRECISION (mAP50)
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

80%

79.50%, With Extra Training Data

Mean Average Precision (mAP50)

75%
77.10%, Without Extra Training Data

70%

65%

60%

55%

50%
2015

2016

2017

2018

2019

2020

2021
Figure 2.2.5

2 GLIP (Grounded Language-Image Pretraining), a model designed to master the learning of language contextual visual representations, was a collaboration of researchers from UCLA, Microsoft Research, University of Washington, University of Wisconsin–Madison, Microsoft Cloud, Microsoft AI, and International Digital Economy Academy.

Table of Contents

Chapter 2 Preview

71Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—VideoYou Only Look Once (YOLO)You Only Look Once is an open-source object detection
model that emphasizes speed (inference latency) overabsolute accuracy.
Over the years, there have been different iterations of YOLO,
and Figure 2.2.6 plots the performance of YOLO object
detectors versus the absolute top performing detectors

on the COCO dataset. YOLO detectors have becomemuch better in terms of performance since 2017 (by 28.4
percentage points). Second, the gap in performance
between YOLO and the best-performing object detectors
has narrowed. In 2017 the gap stood at 11.7%, and it
decreased to 7.1% in 2021. In the last five years, object
detectors have been built that are both faster and better.

STATE OF THE ART (SOTA) vs. YOU ONLY LOOK ONCE (YOLO): MEAN AVERAGE PRECISION (mAP50)
Source: arXiv, 2021; GitHub, 2021 | Chart: 2022 AI Index Report

79.50%, SOTA

Mean Average Precision (mAP50)

80%

70%

72.40%, YOLO

60%

50%

40%
2016

2017

2018

2019

2020

2021
Figure 2.2.6

Table of Contents

Chapter 2 Preview

72Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.2 Computer Vision—Video

Visual Commonsense Reasoning (VCR)
The Visual Commonsense Reasoning challenge is arelatively new benchmark for visual understanding. VCRasks AI systems to answer challenging questions about
scenarios presented from images, and also to provide thereasoning behind their answers (unlike the VQA challenge,

which only requires an answer). The dataset contains
290,000 pairs of multiple-choice questions, answers,
and rationales from 110,000 image scenarios taken frommovies. Figure 2.2.7 illustrates the kinds of questions
posed in the VCR.

A SAMPLE QUESTION OF THE VISUAL COMMONSENSE REASONING (VCR) CHALLENGE
Source: Zellers et al., 2018

Figure 2.2.7

Performance on VCR is measured in the Q->AR score, which
aggregates how well machines can choose the right answer
for a given multiple-choice question (Q->A) and then selectthe correct rationale for the answer (Q->R).
Since the challenge debuted, AI systems have become
better at visual commonsense reasoning, although they stilllag far behind human levels of performance (Figure 2.2.8).
At the end of 2021, the best mark on VCR stood at 72.0, ascore that represents a 63.6% increase in performance since
2018. Although progress has been made since the challenge
was launched, improvements have become increasingly
marginal, suggesting that new techniques may need to beinvented to significantly improve performance.

VISUAL COMMONSENSE REASONING (VCR) TASK: Q->AR SCORE
Source: VCR Leaderboard, 2021 | Chart: 2022 AI Index Report90
85.00, Human Baseline

80

Q->AR Score

72.00

70

60

50

40
2018

2019

2020

2021
Figure 2.2.8

Table of Contents

Chapter 2 Preview

73Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

Natural language processing (NLP) is a subfield of AI, with roots that stretch back as far as the 1950s. NLP involves research into systems thatcan read, generate, and reason about natural language. NLP evolved from a set of systems that in its early years used handwritten rules andstatistical methodologies to one that now combines computational linguistics, rule-based modeling, statistical learning, and deep learning.
This section looks at progress in NLP across several language task domains, including: (1) English language understanding; (2) text
summarization; (3) natural language inference; (4) sentiment analysis; and (5) machine translation. In the last decade, technical progress in
NLP has been significant: The adoption of deep neural network–style machine learning methods has meant that many AI systems can now
execute complex language tasks better than many human baselines.

2.3 LANGUAGEE N G L I S H L A N G UAG E
U N D E R S TA N D I N GEnglish language understanding challenges AI systems
to understand the English language in various contexts,
such as sentence understanding, yes/no reading
comprehension, reading comprehension with logicalreasoning, etc.

SuperGLUE
SuperGLUE is a single-number metric that tracks technicalprogress on a diverse set of linguistic tasks (Figure 2.3.1).

As part of the benchmark, AI systems are tested on eight
different tasks (such as answering yes/no questions,
identifying causality in events, and doing commonsense
reading comprehension), and their performance on thesetasks is then averaged into a single score. SuperGLUE is
the successor to GLUE, an earlier benchmark that also
tests on multiple tasks. SuperGLUE was released in May
2019 after AI systems began to saturate the GLUE metric,
creating demand for a harder benchmark.

A SET OF SUPERGLUE TASKS3
Source: Wang et al., 2019

Figure 2.3.1

3 For the sake of brevity, this figure only displays 3 of the 8 tasks.

Table of Contents

Chapter 2 Preview

74Artificial IntelligenceIndex Report 2022

At the top of the SuperGLUE leaderboard sits the SS-MoE
model with a state-of-the-art score of 91.0 (Figure 2.3.2),
which exceeds the human performance score of 89.8
given by the SuperGLUE benchmark developers. The fact

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Languagethat progress on SuperGLUE was achieved so rapidly
suggests that researchers will need to develop more
complex suites of natural language tasks to challenge
the next generation of AI systems.

SUPERGLUE: SCORE
Source: SuperGLUE Leaderboard, 2021 | Chart: 2022 AI Index Report

92

91.00

Score

91

90

89.80, Human Performance

89

88
2019

2020

2021
Figure 2.3.2

Stanford Question Answering Dataset (SQuAD)
The Stanford Question Answering Dataset (SQuAD)
benchmarks performance on reading comprehension.
The dataset includes 107,785 question-and-answer pairs
taken from 536 Wikipedia articles. Performance on SQuAD
is measured by the F1 score, which is the average overlap
between the AI system’s answers and the actual correct
answers: The higher the score, the better the performance.
As was the case with GLUE, AI systems improved so rapidly
on SQuAD that only two years after launching SQuAD in
2016, researchers released SQuAD 2.0. This second version
included more challenging reading comprehension tasks,
namely a set of 50,000 unanswerable questions that were
written in a way to appear answerable (Figure 2.3.3).

Table of Contents

Chapter 2 Preview

HARDER QUESTIONS ADDED TO STANFORDQUESTION ANSWERING DATASET (SQUAD) 2.0
Source: Rajpurkar et al., 2018

Figure 2.3.3

75

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

At the end of 2021, the leading scores on SQuAD 1.1 and
SQuAD 2.0 stood at 95.7 and 93.2, respectively (Figure
2.3.4). Although these scores are state of the art, they
are marginal improvements over the previous year’s top

scores (0.4% and 0.2%). Both SQuAD datasets have seena trend whereby immediately after the initial launches,human-performance-exceeding scores were realized andthen followed by small, plateau-like increases.

SQUAD 1.1 and SQUAD 2.0: F1 SCORE
Source: SQuAD 1.1 and SQuAD 2.0, 2021 | Chart: 2022 AI Index Report

100

95.72, Squad 1.1

95

93.21, Squad 2.0

F1 Score

91.20, Human Baseline (v1)9089.50, Human Baseline (v2)

85

80
2016

2017

2018

2019

2020

2021
Figure 2.3.4

Reading Comprehension
Dataset Requiring Logical
Reasoning (ReClor)
The plateauing progress onbenchmarks like SQuAD suggests that
NLP models need to be tested on more
complex linguistic challenges like thoseoffered by ReClor. Created in 2020 bycomputer scientists from the National
University of Singapore, ReClor requiresthat AI systems engage in reading
comprehension that also necessitates
logical reasoning. The ReClor dataset ismade up of logical reasoning questions
from the LSAT, the entrance exam forlaw schools in the United States andCanada (Figure 2.3.5).

Table of Contents

Chapter 2 Preview

A SAMPLE QUESTION IN READING COMPREHENSION DATASETREQUIRING LOGICAL REASONING (RECLOR)
Source: Yu et al., 2020

Figure 2.3.5

76Artificial IntelligenceIndex Report 2022

There are two sets of questions on ReClor, easy andhard, with AI systems being judged on accuracy basedon the percentage of questions they answer correctly(Figure 2.3.6). Although AI systems are presently capableof achieving a relatively high level of performance on
the easy set of questions, they struggle on the hard set.
In 2021, the top-performing model on ReClor (hard set)scored 69.3%, roughly 22.5 percentage points lower
than the top-performing model on the easy set. Datasets
like ReClor suggest that while NLP models can executestraightforward reading comprehension tasks, they face
more difficulty when those tasks are coupled with logicalreasoning requirements.

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

Although AI systems
are presently capable ofachieving a relatively highlevel of performance on theeasy set of questions, theystruggle on the hard set.

READING COMPREHENSION DATASET REQUIRING LOGICAL REASONING (RECLOR): ACCURACY
Source: ReClor Leaderboard, 2021 | Chart: 2022 AI Index Report

91.82%, Test Easy

90%

Accuracy (%)

80%

70%

69.29%, Test Hard

60%

50%
2020

2021
Figure 2.3.6

Table of Contents

Chapter 2 Preview

77Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

T E X T S U M M A R I Z AT I O N
Text summarization is the challenge of synthesizing
a piece of text while capturing its core content.
Summarizing texts is an important component oftext classification, reading comprehension, andinformation dissemination; however, when done
manually by humans, it is time- and labor-intensive.
Developing AI systems that can functionally summarize
texts has a number of practical use cases, from aiding
universities in classifying academic papers to helping
lawyers generate case summaries.
Progress in text summarization is often scored on
ROUGE (Recall-Oriented Understudy for Gisting
Evaluation). ROUGE calculates the overlap between

a summary produced by an AI system and the reference
summary produced by a human. The higher the ROUGE
score, the greater the overlap and the more accurate thesummary.

arXiv
ArXiv is a text summarization benchmark dataset that
contains over 27,770 different papers from arXiv, theopen-access repository of scientific papers. In thefive years since benchmarking on arXiv began, AI text
summarization models have improved their performanceby 47.1% (Figure 2.3.7). However, as is the case with othernatural language benchmarks, progress seems to beplateauing.

ARXIV: ROUGE-1
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

47.15, Without Extra Training Data

ROUGE-1

45

46.74, With Extra Training Data

40

35

30
2017

2018

2019

2020

2021
Figure 2.3.7

Table of Contents

Chapter 2 Preview

78Artificial IntelligenceIndex Report 2022

PubMedPubMed is a text summarization dataset consisting of
19,717 scientific publications from the PubMed database ofscientific papers. Progress on PubMed validates the trend
seen on arXiv: There has been significant improvement

on text classification since 2017 (34.6%), but recently thepace of that progress has slowed (Figure 2.3.8). In 2021, thetop-performing model on PubMed was HAT (hierarchicalattention transformer model), created by researchers at
Birch AI and the University of Washington.

PUBMED: ROUGE-1
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report50
48.25, With Extra Training Data

47.81, Without Extra Training Data

ROUGE-1

45

40

35
2017

2018

2019

2020

2021
Figure 2.3.8

Table of Contents

Chapter 2 Preview

79Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

N AT U R A L L A N G UAG E I N F E R E N C E

Stanford Natural Language Inference (SNLI)Natural language inference is the task of determining
whether, given a premise, a hypothesis is true
(entailment), false (contradiction), or undetermined(neutral). This skill is also known as textual
entailment as it requires determining whether aparticular premise logically entails a hypothesis.
Natural language inference necessitates languageprocessing skills, such as named entity recognition(understanding the words you see), as well as being
able to use commonsense knowledge to distinguishbetween reasonable and unreasonable inferences.

The Stanford Natural Language Inference (SNLI) dataset
contains around 600,000 sentence pairs (premise
and associated hypothesis) that are labeled as either
entailment, contradiction, or neutral. As part of thischallenge, AI systems are asked whether premises
logically entail certain hypotheses (Figure 2.3.9).
Performance on SNLI is measured in accuracy based on
the percentage of questions answered correctly.

QUESTIONS AND LABELS IN STANFORD NATURAL LANGUAGE INFERENCE (SNLI)
Source: Bowman et al., 2015

Figure 2.3.9

Table of Contents

Chapter 2 Preview

80Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

The top-performing model on SNLI is Facebook AI USA’s EFL, which in April 2021 posted a score of 93.1% (Figure 2.3.10).
STANFORD NATURAL LANGUAGE INFERENCE (SNLI): ACCURACY
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report95%

Accuracy (%)

93.10%

90%

85%
2014

2015

2016

2017

2018

2019

2020

2021
Figure 2.3.10

Abductive Natural Language Inference (aNLI)
Abductive natural language inference is a moredifficult type of textual entailment. Abductive inference
requires drawing the most plausible conclusion from acontext of limited information and uncertain premises.
For instance, if Jenny were to return from work andfind her home in a disheveled state and then recall
that she left a window open, she can plausibly inferthat a burglar broke in and caused the mess.4 Althoughabduction is regarded as an essential element in how
humans communicate with one another, few studies haveattempted to study the abductive ability of AI systems.
ANLI, a new benchmark for abductive natural language
inference created in 2019 by the Allen Institute for AI,
comes with 170,000 premise and hypothesis pairs. Figure
2.3.11 illustrates the types of statements included in thedataset.

EXAMPLE QUESTIONS IN ABDUCTIVE NATURAL LANGUAGE INFERENCE (ANLI)
Source: Allen Institute for AI, 2021

Figure 2.3.11
4 This particular example of abductive commonsense reasoning is taken from Bhagavatula et al. ( 2019), the first paper that investigates the ability of AI systems to perform language-based
abductive reasoning.

Table of Contents

Chapter 2 Preview

81Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

AI performance on abductive commonsense reasoning has increased by 7.7 percentage points since 2019; however, thetop AI systems, while close, are unable to achieve human performance levels (Figure 2.3.12). Abductive reasoning istherefore still a challenging linguistic task for AI systems.
ABDUCTIVE NATURAL LANGUAGE INFERENCE (aNLI): ACCURACY
Source: Allen Institute for AI, 2021 | Chart: 2022 AI Index Report

92.90%, Human Baseline
91.87%

92%

Accuracy (%)

90%

88%

86%

84%

2019

2020

2021
Figure 2.3.12

S E N T I M E N T A N A LYS I S
Sentiment analysis is the task of using NLP techniques
to identify the sentiment (very negative, negative,
neutral, positive, very positive) of a given text. Sentiment
analysis can be straightforward if sentences are worded
clearly and unambiguously, such as “I dislike winterweather.” However, sentiment analysis can become
more challenging when AI systems encounter sentenceswith flipped structures or negations, such as “to saythat disliking winter weather is not really my thing iscompletely inaccurate.”

Sentiment analysis has many commercial use cases, fromparsing customer reviews and field survey responses to
identifying the emotional states of customers.

SemEval 2014 Task 4 Sub Task 2
The SemEval 2014 Task 4 Sub Task 2 is a benchmark
for sentiment analysis that asks machines to engage
in sentiment analysis. This specific task tests whether
AI systems can identify the sentiment associated withparticular aspects of a text, rather than the sentiment of
entire sentences or paragraphs (Figure 2.3.13).

A SAMPLESEMEVAL TASK
Source: Pontiki et al., 2014

Figure 2.3.13

Table of Contents

Chapter 2 Preview

82Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

The SemEval dataset is composed of 7,686 restaurant and laptop reviews, whose emotional polarities have been rated
by humans. On SemEval, AI systems are tasked with assigning the right sentiment labels to particular components ofthe text, with their performance measured in terms of the percentage of the labels they correctly assign.
In the past seven years, AI systems have become much better at sentiment analysis. As of last year, top-performing
systems estimate sentiment correctly 9 out of 10 times, whereas in 2016, they made correct estimates only 7 out of
10 times. As of 2021, the state-of-the-art scores on SemEval stood at 88.6%, realized by a team of Chinese researchers
from South China Normal University and Linklogis Co. Ltd. (Figure 2.3.14).

SEMEVAL 2014 TASK 4 SUB TASK 2: ACCURACY
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

90%

88.64%

Accuracy (%)

85%

80%

75%

70%
2015

2016

2017

2018

2019

2020

2021
Figure 2.3.14

M AC H I N E T R A N S L AT I O N ( M T )Machine translation examines how AI software cantranslate languages. In domains where fluency inmultiple languages is required, machine translation
can be extremely impactful. The European Union, forexample, is required to translate all its cross-national
policy documents into the 24 languages of its member
states. Using machine translators can save time, improve
efficiency, and lead to more consistent outcomes.

Table of Contents

Chapter 2 Preview

Since 2017, neural networks have taken over machinetranslation. Unlike their predecessors, neural translators
learn from a series of prior translation tasks and predictthe likelihood of a sequence of words. Neural translation
models have revolutionized the field of machine translation
not only because they do not require human supervision,
but also because they produce the most accuratetranslations. As a result, they have been widely deployed bysearch engines and social networks.

83

Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

WMT 2014, English-German and English-French
The WMT 2014 family of datasets, first introduced at the
Meeting of the Association for Computational Linguistics(ACL) 2014, consist of different kinds of translation
tasks, including translation between English-French andEnglish-German language pairs. A machine’s translation
capabilities are measured by the Bilingual Evaluation
Understudy, or BLEU, score, which compares the extent
to which a machine-translated text matches a reference
human-generated translation. The higher the score, thebetter the translation.

Both the English-French and English-German WMT 2014
benchmarks showcase the significant progress madein AI machine translation over the last decade (Figure
2.3.15). Since submissions began, there has been a 23.7%
improvement in English-French and a 68.1% improvementin English-German translation ability. Relatively speaking,although performance improvements have been moresignificant on the English-German language pair, absolutetranslation ability remains meaningfully higher on EnglishFrench translation.

WMT2014, ENGLISH-FRENCH: BLEU SCORE

WMT2014, ENGLISH-GERMAN: BLEU SCORE

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

46.40, With Extra Training Data

45

4543.95, Without Extra Training Data

40

BLEU score

BLEU Score

40

35

31.26, Without Extra Training Data

30

30

25

25

20

20
2015

2017

2019

2021

35.14, With Extra Training Data

35

2015

2017

2019

2021
Figure 2.3.15

Table of Contents

Chapter 2 Preview

84Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.3 Language

Number of Commercially Available MT Systems
The growing interest in machine translation is also
reflected in the rise of commercial machine translation
services such as Google Translate. Since 2017, there
has been a nearly fivefold increase in the number ofcommercial machine translators on the market, according

to Intento (Figure 2.3.16). 2021 also saw the introduction of
three open-source machine translation services (M2M-100,mBART, and OPUS). The emergence of publicly available,
high-functioning machine translation services speaks to
the increasing accessibility of such services and bodes well
for anybody who routinely relies on translation.

NUMBER of INDEPENDENT MACHINE TRANSLATION SERVICES
Source: Intento, 2021 | Chart: 2022 AI Index Report

Number of Independent Machine Translation Services

50
46

Commercial
Open Source Pre-trained
Preview

40
34

30
38

26
23
21
28

20
16

10

9
8

18

9

10

3

3

11/2017

03/2018

21

15
3

9

0
05/2017

23

13

12
10

07/20173

07/2018

12/20183

06/2019

11/20196

07/2020

5

10/2021
Figure 2.3.16

Table of Contents

Chapter 2 Preview

85Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.4 Speech

Another important domain of AI research is the analysis, recognition, and synthesis of human speech. In this AI subfield, AI systems are
typically rated on their ability to recognize speech and identify words and convert them into text; and also to recognize speakers andidentify the individuals speaking. Modern home assistance tools, such as Siri, are one of the many examples of commercially applied AI
speech technology.

2.4 SPEECH
English speech taken from a collection of audiobooks. On
LibriSpeech, AI systems are asked to transcribe speech
to text and then measured on word error rate, or thepercentage of words they fail to correctly transcribe.

SPEECH RECOGNITION
Speech recognition is the process of training machines
to recognize spoken words and convert them into text.
Research in this domain began at Bell Labs in the 1950s,
when the world was introduced to the automatic digit
recognition machine (named “Audrey”), which could
recognize a human saying any number from zero to nine.
Speech recognition has come a long way since then and in
the last decade has benefited tremendously from deeplearning techniques and the availability of rich speech
recognition datasets.

LibriSpeech is subdivided into two datasets. First, there
is LibriSpeech Test Clean, which includes higher-quality
recordings. Performance on Test Clean suggests how
well AI systems can transcribe speech in ideal conditions.
Second, there is LibriSpeech Test Other, which includes
lower-quality recordings. Performance on Test Other isindicative of transcription performance in environments
where sound quality is less than ideal.

Transcribe Speech: LibriSpeech (Test-Clean andOther Datasets)

AI systems perform incredibly well on LibriSpeech, somuch so that progress appears to be plateauing (Figure
2.4.1). A state-of-the-art result on the Test Clean dataset

LIBRISPEECH, TEST CLEAN: WORD ERROR RATE (WER)

LIBRISPEECH, TEST OTHER: WORD ERROR RATE (WER)

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

14

14

12

12

10

10
Word Error Rate (WER)

Word Error Rate (WER)

Introduced in 2015, LibriSpeech is a speech transcription
database that contains around 1,000 hours of 16 khz

8

6

4

3.30, Without Extra Training Data

8

6

4
1.70, Without Extra Training Data22

2.50, With Extra Training Data

1.40, With Extra Training Data

0

0
2015

2017

2019

2021

2015

2017

2019

2021
Figure 2.4.1

Table of Contents

Chapter 2 Preview

86Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.4 Speech

was not realized in 2021, which speaks to the fact that theerror rate of the top system was already low at 1.4%. For
every 100 words that top-performing transcription models
heard, they correctly transcribed 99.
Performance on the Test Other dataset, while worse than
Test Clean, was still relatively poor. The state-of-the-art
results on Test Other were realized by the W2V-BERT
model, an MIT and Google Brain collaboration, which
posted an error rate of 2.0%.

VoxCeleb
VoxCeleb is a large-scale audiovisual dataset of humanspeech for speaker recognition, which is the task of matchingcertain speech with a particular individual. Each year, themakers of VoxCeleb host a speaker verification challenge.
A low score or equal error rate on the VoxCeleb challenge
is indicative of an AI system that makes few errors in its
attribution of speech to particular individuals.5 Figure 2.4.2
plots performance over time on VoxCeleb-1, the original
VoxCeleb dataset. Since 2017, performance on VoxCeleb has
improved: Systems that once reported equal error rates of
7.8% now report errors that are less than 1.0%.

VOXCELEB: EQUAL ERROR RATE (EER)
Source: VoxCeleb, 2021 | Chart: 2022 AI Index Report8%

Equal Error Rate (%)

6%

4%

2%

0.42%

0%
2017

2018

2019

2020

2021
Figure 2.4.2

5 The Equal Error Rate (EER) is not only a measure of the false positive rate, assigning a bad label, but also the false negative rate (failure to assign the correct label).

Table of Contents

Chapter 2 Preview

87Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.5 Recommendation

Recommendation is the task of suggesting items that might be of interest to a user, such as movies to watch, articles to read, or products
to purchase. Recommendation systems are crucial to businesses, such as Amazon, Netflix, Spotify, and YouTube. For example, one of
the earliest open recommendation competitions in AI was the Netflix Prize; hosted in 2009, it challenged computer scientists to develop
algorithms that could accurately predict user ratings for films based on previously submitted ratings.

2.5 RECOMMENDATION
Commercial Recommendation: MovieLens 20M
The MovieLens 20M dataset contains around 20
million movie ratings for 27,000 movies from 138,000
users. The ratings are taken from MovieLens (a movie
recommendation platform), and AI systems are challenged
to see if they can predict a user’s movie preferences basedon their previously submitted ratings. The metric used to
track performance on MovieLens is Normalized Discounted
Cumulative Gain (nDCG), which is a measure of ranking

quality. A higher nDCG score means that an AI systemdelivers more accurate recommendations.
Since 2018, top models now perform roughly 5.2% betteron MovieLens 20M than they did in 2018 (Figure 2.5.1).
In 2021, the state-of-the-art system on MovieLens 20M
posted an nDCG of 0.448 and came from researchers at
the Czech Technical University in Prague.

MOVIELENS 20M: NORMALIZED DISCOUNTED CUMULATIVE GAIN@100 (nDCG@100)
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

Normalized Discounted Cumulative Gain@100 (nDCG@100)

0.460

0.450
0.448

0.440

0.430

0.420
2018

2019

2020

2021
Figure 2.5.1

Table of Contents

Chapter 2 Preview

88Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.5 Recommendation

Click-Through Rate Prediction: Criteo
Click-through rate prediction is the task of predicting
the likelihood that something on a website, say anadvertisement, will be clicked. In 2014, the online
advertising platform Criteo launched an open clickthrough prediction challenge. Included as part of thischallenge dataset was information on a million ads
that were displayed during a 24-day period, whether
they were clicked, and additional information on theircharacteristics. Since the competition launched, theCriteo dataset has been widely used to test recommendersystems. On Criteo, systems are measured on area under
the curve (AUC). A higher AUC means a better clickthrough prediction rate and a stronger recommendersystem.

Performance on Criteo also indicates that recommendersystems have been slowly and steadily improving in thepast decade. Last year’s top model (Sina Weibo Corp’s
MaskNet) performed 1.8% higher on Criteo than the top
model from 2016. An improvement of 1.8% may seemsmall in absolute terms, but it can be a valuable margin in
the commercial world.
A limit of the Criteo and MovieLens benchmarks is thatthey are primarily academic measures of technicalprogress in recommendation (Figure 2.5.2). Most of theresearch work on recommendation occurs in commercial
settings. Given that companies have an incentive to keep
their recommendation improvements proprietary, theacademic metrics included in this section might not becomplete measures of the technical progress made inrecommendation.

CRITEO: AREA UNDER CURVE SCORE (AUC)
Source: Papers with Code, 2021; arXiv, 2021 | Chart: 2022 AI Index Report

0.813

Area Under Curve Score (AUC)

0.810

0.800

0.790
2016

2017

2018

2019

2020

2021
Figure 2.5.2

Table of Contents

Chapter 2 Preview

89Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.6 Reinforcement Learning

In reinforcement learning, AI systems are trained to maximize performance on a given task by interactively learning from their prioractions. Researchers train systems to optimize by rewarding them if they achieve a desired goal and then punishing them if they fail.
Systems experiment with different strategy sequences to solve their stated problem (e.g., playing chess or navigating through a maze)and select the strategies which maximize their rewards.
Reinforcement learning makes the news whenever programs like DeepMind’s AlphaZero demonstrate superhuman performance ongames like Go and Chess. However, reinforcement learning is useful in any commercial domain where computer agents need to maximize
a target goal or stand to benefit from learning from previous experiences. Reinforcement learning can help autonomous vehicles change
lanes, robots optimize manufacturing tasks, or time-series models predict future events.

2.6 REINFORCEMENT LEARNING
REINFORCEMENT LEARNINGENVIRONMENTS
A reinforcement learning environment is a computer
platform where AI agents are challenged to maximize
their performance on a defined task. Unlike other AI tasks
which require systems to train on a dataset, reinforcement
learning necessitates that AI systems have an environment
in which they can test various strategies and, in the process,identify the set of strategies that will maximize rewards.

Arcade Learning Environment: Atari-57
Introduced in 2013, the Arcade Learning Environment
is an interface that includes various Atari 2600 gameenvironments (such as “Pac-Man,” “Space Invaders,”and “Frogger”) in which AI agents are challenged to
optimize performance. To enable standard comparisons,researchers typically report average performance on the
ALE across a suite of 57 games. There are various metrics
in which performance is measured, but one of the mostcommon is mean human-normalized score. A humannormalized score of 0% represents random performance,
and a 100% score represents average human performance.
The mean human-normalized score is then the average
human-normalized score achieved by an AI system.
In late 2019, DeepMind’s MuZero algorithm achieved
state-of-the-art performance on Atari-57. MuZero
not only performed 48.3% better on Atari-57 thanthe previous best-performing model, but it also set a
new world record on Go and achieved superhumanperformance on chess and shogi.

Table of Contents

Chapter 2 Preview

Creating reinforcement
learning models that areboth high performingand highly efficient is
an important step in thecommercial deployment ofreinforcement learning.

In 2021, however, researchers from Tsinghua University
and ByteDance launched the GDI-H3 model, whichsurpassed (and nearly doubled) MuZero’s performanceon Atari-57 (Figure 2.6.1). Moreover, GDI-H3 achieved
this performance with less training. It only used 200
million training frames, whereas MuZero used 20 billion:
GDI-H3 was twice as effective and one hundred times
more efficient. Creating reinforcement learning models
that are both high performing and highly efficient is
an important step in the commercial deployment ofreinforcement learning.

90Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.6 Reinforcement Learning

ATARI-57: MEAN HUMAN-NORMALIZED SCORE
Source: Papers with Code, 2021; arXIv, 2021 | Chart: 2022 AI Index Report

9.62

Mean-Human Normalized Score (in thousands)

10

8

6

4

2

0
2015

2016

2017

2018

2019

2020

2021
Figure 2.6.1

Procgen
Procgen is a new reinforcement learning
environment introduced by OpenAI in 2019. It
includes 16 procedurally generated video-gamelike environments specifically designed to test theability of reinforcement learning agents to learn
generalizable skills (Figure 2.6.2). Procgen wasdeveloped to overcome some of the criticisms
leveled at benchmarks like Atari that incentivized
AI systems to become narrow learners thatmaximized capacity in one particular skill.
Procgen encourages broad learning by introducing
a reinforcement learning environment thatemphasizes high diversity and forces AI systems
to train in generalizable ways. Performance on
Procgen is measured in terms of mean-normalized
score. Researchers typically train their systems on
200 million training runs and report an averagescore across the 16 Procgen games. The higher thesystem scores, the better the system.

A SCREENSHOT OF THE 16 GAME ENVIRONMENTS IN PROCGEN
Source: Cobbe et al. 2019

Figure 2.6.2

Table of Contents

Chapter 2 Preview

91Artificial IntelligenceIndex Report 2022

In November 2021, the MuZero model from DeepMind
posted a state-of-the-art score of 0.6 on Procgen.
DeepMind’s results were a 128.6% improvement overthe baseline performance established in 2019 when

CHAPTER 2: TECHNICAL PERFORMANCE
2.6 Reinforcement Learning

the environment was first released. Rapid progress onsuch a diverse benchmark signals that AI systems are
slowly improving their capacity to reason in broader
environments.

PROCGEN: MEAN-NORMALIZED SCORE
Source: arXiv, 2021 | Chart: 2022 AI Index Report

0.64

Mean-Normalized Score

0.60

0.50

0.40

0.30

2019

2020

2021
Figure 2.6.3

Table of Contents

Chapter 2 Preview

92Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.6 Reinforcement Learning

Human Games: Chess

range of games, such as shogi and Go, and have infact beaten some of the top-ranked chess engines.
Nevertheless, looking at chess engine performance is aneffective way to relativize the progress made in AI andcompare it to a widely understandable human baseline.

Progress in reinforcement learning can also be capturedby the performance of the world’s top chess software
engines. A chess engine is a computer program that istrained to play chess at a high level by analyzing chesspositions. The performance of chess engines is ranked
on Elo, a method for identifying the relative skill levelsof players in zero-sum games like chess: A higher scoremeans a stronger player.

Computers surpassed human performance in chessa long time ago, and since then have not stopped
improving (Figure 2.6.4). By the mid-1990s, the top chessengines exceeded expert-level human performance, andby the mid-2000s they surpassed the peak performanceof Magnus Carlsen, one of the best chess players of all
time. Magnus Carlsen’s 2882 Elo, recorded in 2014, isthe highest level of human chess performance everdocumented. As of 2021, the top chess engine has
exceeded that level by 24.3%.

One caveat is that tracking the performance ofchess engines is not a complete reflection of generalreinforcement learning progress; chess engines are
specifically trained to maximize performance on chess.
Other popular reinforcement learning systems, like
DeepMind’s AlphaZero, are capable of playing a broader
CHESS SOFTWARE ENGINES: ELO SCORE
Source: Swedish Computer Chess Association, 2021 | Chart: 2022 AI Index Report

3,581

3500

3000

2882, Magnus Carlsen

Elo Score

2500

2300, Expert

2000
1700, Intermediate
1500

1000

800, Novice

500
0
1987

1992

1997

2002

2007

2012

2017

2022
Figure 2.6.4

Table of Contents

Chapter 2 Preview

93Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.7 Hardware

In evaluating technical progress in AI, it is relevant not only to consider improvements in technical performance but also the speed ofoperation. As this section shows, AI systems continue to improve in virtually every skill category. This performance is often realized byincreasing parameters and training systems on greater amounts of data. However, all else being equal, models that use more parametersand source more data will take longer to train. Longer train times mean slower real-world deployment. Given that the potential ofincreased training times can be offset by stronger and more robust computational infrastructures, it is important to keep track ofprogress in the hardware that powers AI systems.

2.7 HARDWARE
MLPerf: Training Time
MLPerf is an AI training competition run by the ML
Commons organization. In this challenge, participants
train systems to execute various AI tasks (image
classification, image segmentation, natural language
processing, etc.) using a common architecture. Entrants
are then ranked on their absolute wall clock time, which is
how long it takes for the system to train.
Since the MLPerf competitions began in December 2018,
two key trends have emerged: (1) Training times forvirtually every AI skill category have massively decreased;
while (2) AI hardware robustness has substantially

increased. Top-performing hardware systems can reachbaseline levels of performance in task categories likerecommendation, light-weight objection detection, image
classification, and language processing in under a minute.
Figure 2.7.2 more precisely profiles the magnitude of
improvement across each skill category since MLPerf first
introduced the category.6 For example, training times
on image classification increased roughly twenty-sevenfold, as top times dropped from 6.2 minutes in 2018 to
0.2 minutes (or 13.8 seconds) in 2021. It might be hard tofathom the magnitude of a 27-time decrease in training
time, but in real terms it is the difference between waiting
one hour for a bus versus a little more than two minutes.

MLPERF TRAINING TIME of TOP SYSTEMS by TASK: MINUTES
Source: MLPerf, 2021 | Chart: 2022 AI Index Report

Training Time (Minutes; Log Scale)50

20
13.57, Reinforcement Learning

10
5

3.24, Object Detection (heavy-weight)
2.38, Speech recognition

2

1.26, Image Segmentation

1
0.63, Recommendation

0.5
0.34, Object Detection (light-weight)
0.23, Language Processing
0.23, Image Classi cation

0.2
2018

2019

2020

2021
Figure 2.7.1

6 The solitary point for reinforcement learning on Figure 2.7.1 indicates that a faster time was not registered in the MLPerf competitions in 2020 or 2021. The solitary points for speech recognition andimage segmentation are indicative of the fact that those AI subtask categories were added to the MLPerf competition in 2021.

Table of Contents

Chapter 2 Preview

94Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.7 Hardware

Top-performing hardware systems can reach baseline levelsof performance in task categories like recommendation,light-weight objection detection, image classification, andlanguage processing in under a minute.

MLPERF: SCALE of IMPROVEMENT across TASK
Source: MLPerf, 2021 | Chart: 2022 AI Index Report

26.96

25

Scale of Improvement

22.25

20
16.47

15

10

5

0

Improvement Baseline 1.16
Reinforcement

SpeechRecognition

1.70

1.92

Language
Processing

Recommendation

2.38

Segmentation

Object Detection Object Detection(light-weight)(heavy-weight)

Image
Classi!cation
Figure 2.7.2

Table of Contents

Chapter 2 Preview

95Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.7 Hardware

MLPerf: Number of Accelerators
The cross-task improvements in training time are being
driven by stronger underlying hardware systems, as shownin Figure 2.7.3. Since the competition began, the highest
number of accelerators used—where an accelerator isa chip used predominantly for the machine learningcomponent of a training run, such as a GPU or a TPU—andthe mean number of accelerators used by the top systemincreased roughly 7 times while the mean number ofaccelerators used by all entrants increased 3.5 times.
Most notable, however, is the growing gap between
the average number of accelerators used by the topperforming systems and the average accelerators usedby all systems. This gap was 9 times larger at the end of2021 than it had been in 2018. This growth clearly means
that, on average, building the fastest systems requires
the most powerful hardware.

MLPERF HARDWARE: ACCELERATORS
Source: MLPerf, 2021 | Chart: 2022 AI Index Report

4,320, Maximum Number of Accelerators Used

Number of Accelerators

4,000

3,000

1,785, Average Accelerators Used by Top System

2,000

1,000
337, Mean Number of Accelerators

012.12.201806.10.201907.29.2020

06.30.2021

12.01.2021
Figure 2.7.3

Table of Contents

Chapter 2 Preview

96Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.7 Hardware

IMAGENET: Training Cost

classification system. This cost is marginal, especiallywhen compared to the $1,112.6 it cost to train a similarly
performing system in 2017. In simpler terms, in four
short years, image classification training costs havedecreased by a factor of 223.

Stronger hardware has not necessarily meant costlier
training. Figure 2.7.4 plots the lowest training cost per
year on MLPerf’s image classification subtask (ImageNet).
In 2021, it cost only $4.6 to train a high-performing image

IMAGENET: TRAINING COST (to 93% ACCURACY)
Source: AI Index and Narayanan, 2021 | Chart: 2022 AI Index Report$1,000.00

Cost (in U.S. Dollars; Log Scale)$500.00$200.00$100.00$50.00$20.00$10.00
$5.00

$4.59

$2.00
$1.00
2017

2018

2019

2020

2021
Figure 2.7.4

Table of Contents

Chapter 2 Preview

97Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.8 Robotics

In 2021, the AI Index developed a survey that asked professors who specialize in robotics at top-ranked universities around the world andin emerging economies about changes in the pricing of robotic arms as well as the uses of robotic arms in research labs. The survey wascompleted by 101 professors and researchers from over 40 universities and collected data on 117 robotic arm purchase events from 2016
to 2022. The survey results suggest that there has been a notable decline in the price of robotic arms since 2016.

2.8 ROBOTICS
Price Trends in Robotic Arms7
The survey results show a clear downward trend in thepricing of robotic arms in the last seven years. In 2017,
the median price of a robotic arm was $42,000. Since
then, the price has declined by 46.2% to reach roughly$22,600 in 2021 (Figure 2.8.1). Figure 2.8.2, which plots the
distribution of robotic arm prices, paints a similar picture:
Despite some high-priced outliers, there has been a clear
downward trend since 2017 in the price of robotic arms.

MEDIAN PRICE of ROBOTIC ARMS, 2017–21
Source: AI Index, 2022 | Chart: 2022 AI Index Report

Price (in thousands of U.S. Dollars)

$40

$30

$22.60

$20

$10

$0
2017

2018

2019

2020

2021
Figure 2.8.1

7 We have corrected Figure 2.8.1 and Figure 2.8.2 after noticing a data filtering issue with the survey result. The correct chart has since been updated. View the appendix here for links to the data. In
addition, note that academic researchers may get a discount when purchasing robotic arms so prices are lower than retail.

Table of Contents

Chapter 2 Preview

98Artificial IntelligenceIndex Report 2022

CHAPTER 2: TECHNICAL PERFORMANCE
2.8 Robotics

DISTRIBUTION of ROBOTIC ARM PRICES, 2017–21
Source: AI Index, 2022 | Chart: 2022 AI Index Report

Price (in thousands of U.S. Dollars)

$100

$50

$0
2017

2018

2019

2020

2021
Figure 2.8.2

AI Skills Employed by Robotics
Professors
In the survey, the AI Index also asked
robotics professors to what extent
they employ AI skills in their research.
Responses revealed that both deeplearning and reinforcement learning are
popular AI skills employed by roboticists.
More specifically, 67.0% of professors
reported using deep learning and 46.0%
reported using reinforcement learning.

AI SKILLS EMPLOYED by ROBOTICS PROFESSORS
Source: AI Index, 2022 | Chart: 2022 AI Index Report

67.00%

Deep Learning

Reinforcement
Learning

46.00%

0%

20%

40%

60%

80%

% of Respondents
Figure 2.8.3

Table of Contents

Chapter 2 Preview

99Artificial IntelligenceIndex Report 2022

CHAPTER 3:

Technical AI Ethics
Text and Analysis by
Helen Ngo and Ellie Sakhaee

Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS

CHAPTER 3:

Chapter Preview
Overview

102

Acknowledgment

103

Chapter Highlights

105

3.1 META-ANALYSIS OF FAIRNESSAND BIAS METRICS
AI Ethics Diagnostic Metrics
and Benchmarks

106
107

3.2 NATURAL LANGUAGE
PROCESSING BIAS METRICS

109

Toxicity: RealToxicityPrompts
and the Perspective API

109

Highlight: Large Language Models
and Toxicity

111

Detoxification of Models Can
Negatively Influence Performance

113

StereoSet

114

CrowS-Pairs

115

Winogender and WinoBias

117

WinoMT: Gender Bias in MachineTranslation Systems

119

Word and Image Embedding
Association Tests

120

Highlight: Multilingual Word Embeddings122	Mitigating Bias in Word Embeddings
With Intrinsic Bias Metrics

Table of Contents

3.3 AI ETHICS TRENDS AT
FACCT AND NEURIPS123

ACM Conference on Fairness,
Accountability, and Transparency (FAccT)

123

NeurIPS Workshops

125	Interpretability, Explainability,
and Causal Reasoning

126

Privacy and Data Collection

127

Fairness and Bias

129

3.4 FACTUALITY AND TRUTHFULNESS

130

Fact-Checking With AI

130

Measuring Fact-Checking Accuracy
With FEVER Benchmark

133

Toward Truthful Language Models

134

Model Size and Truthfulness

134

Highlight: Multimodal Biases in Contrastive
Language-Image Pretraining (CLIP)136
Denigration Harm

136

Gender Bias

136

Propagating Learned Bias Downstream 138
Underperformance on
Non-English Languages

138

ACCESS THE PUBLIC DATA
122

101

Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS

Overview
In recent years, AI systems have started to be deployed into theworld, and researchers and practitioners are reckoning with theirreal-world harms. Some of these harms include commercial facialrecognition systems that discriminate based on race, résumé
screening systems that discriminate on gender, and AI-powered
clinical health tools that are biased along socioeconomic and racial
lines. These models have been found to reflect and amplify humansocial biases, discriminate based on protected attributes, andgenerate false information about the world. These findings haveincreased interest within the academic community in studying AI
ethics, fairness, and bias and prompted industry practitioners to
direct resources toward remediating these issues, and attracted
attention from the media, governments, and the people who use andare affected by these systems.
This year, the AI Index highlights metrics which have been adopted
by the community for reporting progress in eliminating bias andpromoting fairness. Tracking performance on these metrics alongsidetechnical capabilities provides a more comprehensive perspective
on how fairness and bias change as systems improve, which will be
important to understand as systems are increasingly deployed.

Table of Contents

Chapter 3 Preview

102Artificial IntelligenceIndex Report 2022

AC K N OW L E D G M E N T
The AI Index would like to thank all those involved in research and advocacy around the development and governanceof responsible AI. This chapter builds upon the work of scholars from across the AI ethics community, including thoseworking on measuring technical capabilities as well those focused on shaping thoughtful societal norms. There is much
more work to be done, but we are inspired by the progress made by this community and its collaborators.
Publications cited in this Chapter include:
Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage. 2021. Evaluating
CLIP: Towards Characterization of Broader Capabilities and Downstream Implications. arXiv preprint arXiv:2108.02818.
Jack Bandy and Nicholas Vincent. 2021. Addressing “Documentation Debt” in Machine Learning Research: A
Retrospective Datasheet for Book Corpus. arXiv preprint arXiv:2105.05241.
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. 2021. Multimodal Datasets: Misogyny, Pornography, andMalignant Stereotypes. arXiv preprint arXiv:2110.01963.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den
Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman
Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey
Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre. 2021. Improving
Language Models by Retrieving from Trillions of Tokens. arXiv preprint arXiv:2112.04426.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2017. Word Embeddings Quantify 100 Years of Gender andEthnic Stereotypes. arXiv preprint arXiv:1711.08412.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A.Smith. 2020. RealToxicityPrompts: Evaluating
Neural Toxic Degeneration in Language Models. arXiv preprint arXiv:2009.11462.
Wei Guo and Aylin Caliskan. 2020. Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain aDistribution of Human-like Biases. arXiv preprint arXiv:2006.03955.					Aylin Caliskan Islam, Joanna J. Bryson, and Arvind Narayanan. 2016. Semantics Derived Automatically from Language
Corpora Necessarily Contain Human Biases. arXiv preprint arXiv:1608.07187.
Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021. Jurassic-1: Technical Details and Evaluation. (2021).
https://uploads-ssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On Measuring Social Biases in
Sentence Encoders. arXiv preprint arXiv:1903.10561.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring Stereotypical Bias in Pretrained Language
Models. arXiv preprint arXiv:2004.09456.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain,
Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew
Knight, Benjamin Chess, John Schulman. WebGPT: Browser-Assisted Question-Answering with Human Feedback. 2021.
arXiv preprint arXiv:2112.09332.
Table of Contents

Chapter 3 Preview

103Artificial IntelligenceIndex Report 2022

Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-Pairs: A Challenge Dataset for
Measuring Social Biases in Masked Language Models. arXiv preprint arXiv:2010.00133.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal,
Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter
Welinder, Paul Christiano, Jan Leike, Ryan Lowe. Training Language Models to Follow Instructions with Human Feedback.
2022. arXiv preprint arXiv:2203.02155.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah
Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard
Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl,
Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy
Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela
Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya,
Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug
Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume,
Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James
Bradbury, Matthew Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart,
Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis,
Koray Kavukcuoglu, and Geoffrey Irving. 2021. Scaling Language Models: Methods, Analysis & Insights from Training
Gopher. arXiv preprint arXiv:2112.11446.
Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating Gender Bias in Machine Translation. arXiv
preprint arXiv:1906.00591.
Ryan Steed and Aylin Caliskan. 2020. Image Representations Learned With Unsupervised Pre-Training Contain Humanlike Biases. arXiv preprint arXiv:2010.15052.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese,
Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane,
Julia Haas, Laura Rimell, Lisa Anne Hendricks, William S. Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021.
Ethical and social risks of harm from Language Models. arXiv preprint arXiv:2112.04359.
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty
Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. 2021. Challenges in Detoxifying Language Models. arXiv
preprint arXiv:2109.07445.
Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. 2021. Detoxifying Language
Models Risks Marginalizing Minority Voices. arXiv preprint arXiv:2104.06390.
Pei Zhou, Weijia Shi, Jieyu Zhao, Kuan-Hao Huang, Muhao Chen, Ryan Cotterell, and Kai-Wei Chang. 2019. Examining
Gender Bias in Languages with Grammatical Gender. arXiv preprint arXiv:1909.02224.

Table of Contents

Chapter 3 Preview

104Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS

CHAPTER HIGHLIGHTS• Language models are more capable than ever, but also more biased: Large language
models are setting new records on technical benchmarks, but new data shows that larger
models are also more capable of reflecting biases from their training data. A 280 billion
parameter model developed in 2021 shows a 29% increase in elicited toxicity over a
117 million parameter model considered the state of the art as of 2018. The systems are
growing significantly more capable over time, though as they increase in capabilities, so does
the potential severity of their biases.
• The rise of AI ethics everywhere: Research on fairness and transparency in AI has exploded
since 2014, with a fivefold increase in related publications at ethics-related conferences.
Algorithmic fairness and bias has shifted from being primarily an academic pursuit tobecoming firmly embedded as a mainstream research topic with wide-ranging implications.
Researchers with industry affiliations contributed 71% more publications year over year
at ethics-focused conferences in recent years.
• Multimodal models learn multimodal biases: Rapid progress has been made on training
multimodal language-vision models which exhibit new levels of capability on joint languagevision tasks. These models have set new records on tasks like image classification and thecreation of images from text descriptions, but they also reflect societal stereotypes andbiases in their outputs—experiments on CLIP showed that images of Black people weremisclassified as nonhuman at over twice the rate of any other race. While there has beensignificant work to develop metrics for measuring bias within both computer vision and natural
language processing, this highlights the need for metrics that provide insight into biases inmodels with multiple modalities.

Table of Contents

Chapter 3 Preview

105Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.1 Meta-Analysis of Fairness and Bias Metrics

Significant research effort has been invested over the past five years into creating datasets, benchmarks, and metrics designed tomeasure bias and fairness in machine learning models. Bias is often learned from the underlying training data for an AI model; this data
can reflect systemic biases in society or the biases of the humans who collected and curated the data.

3.1 META-ANALYSIS OF FAIRNESS AND
BIAS METRICS
Algorithmic bias is commonly framed in terms ofallocative and representation harms. Allocative harm
occurs when a system unfairly allocates an opportunity
or resource to a specific group, and representation harm
happens when a system perpetuates stereotypes andpower dynamics in a way that reinforces subordinationof a group. Algorithms are broadly considered fair when
they make predictions that neither favor nor discriminate
against individuals or groups based on protected
attributes which cannot be used for decision-making due

to legal or ethical reasons (e.g., race, gender, religion).
To better understand the landscape of algorithmic bias
and fairness, the AI Index conducted original research toanalyze the state of the field. As shown in Figure 3.1.1,
the number of metrics for measuring bias and fairness
along ethical dimensions of interest has grown steadily
since 2018. For this graph, the number of fairness andbias metrics published has been cited in at least one
other work.1

NUMBER of AI FAIRNESS and BIAS METRICS, 2016–21
Source: AI Index, 2021 | Chart: 2022 AI Index Report

Number of Metrics

15

10

7

5

0
2016

2017

2018

2019

2020

2021
Figure 3.1.1

1 2021 data may be lagging as it takes time for metrics to be adopted by the community.

Table of Contents

Chapter 3 Preview

106Artificial IntelligenceIndex Report 2022

A I E T H I C S D I AG N O S T I C M E T R I C S
A N D B E N C H M A R KS
Measurement of AI systems along an ethical dimension
often takes one of two forms:• Benchmark datasets: A benchmark dataset contains
labeled data, and researchers test how well their AI
system labels the data. Benchmarks do not change overtime. These are domain-specific (e.g., SuperGLUE andStereoSet for language models; ImageNet for computer
vision) and often aim to measure behavior that is intrinsic
to the model, as opposed to its downstream performanceon specific populations (e.g., StereoSet measures model
propensity to select stereotypes compared to nonstereotypes, but it does not measure performance gaps
between different subgroups).
• Diagnostic metrics: A diagnostic metric measures theimpact or performance of a model on a downstream
task—for example, a population subgroup or individualcompared to similar individuals or the entire population.
These metrics can help researchers understand how asystem will perform when deployed in the real world, andwhether it has a disparate impact on certain populations.
Examples include group fairness metrics such as
demographic parity and equality of opportunity.
Benchmarks are useful indicators of progress for thefield as a whole, and their impact can be measured
by community adoption (e.g., number of leaderboard
submissions, or the number of research papers which
report metrics). They also often enable rapid algorithmic
progress as research labs compete on leaderboard
metrics. However, some leaderboards can be easily
gamed, and may be based on benchmark datasets that
contain flaws, such as incorrect labels or poorly definedclasses. Additionally, their static nature means they are asnapshot of a specific cultural and temporal context—inother words, a benchmark published in 2017 may notcorrelate to the deployment context of 2022.

CHAPTER 3: TECHNICAL AI ETHICS
3.1 Meta-Analysis of Fairness and Bias Metrics

Diagnostic metrics enable researchers and practitioners
to understand the impact of their system on a specificapplication or group and potential concrete harm (e.g.,“this model is disproportionately underperforming on thisgroup with this protected attribute”). Diagnostic metrics
are most useful at an individual model or application level
as opposed to functioning as field-level indicators. They
indicate how a specific AI system performs on a specificsubgroup or individual, which is helpful for assessing
real-world impact. However, while these metrics may be
widely used to test models privately, there is not as muchinformation available publicly as these metrics are notattached to leaderboards which encourage researchers topublish their results.
Figure 3.1.2 shows that there has been a steady amountof research investment into developing both benchmarks
and diagnostic metrics over time.2 3

Benchmarks are usefulindicators of progress
for the field as a whole,and their impact can bemeasured by community
adoption (e.g., number ofleaderboard submissions,or the number of research
papers which report
metrics).

2 Research paper citations are a lagging indicator of activity, and metrics which have been very recently adopted may not be reflected in the current data, similar to 3.1.1.
3 The Perspective API defined seven new metrics for measuring facets of toxicity (toxicity, severe toxicity, identity attack, insult, obscene, sexually explicit, threat), contributing to the unusually highnumber of metrics released in 2017.

Table of Contents

Chapter 3 Preview

107Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.1 Meta-Analysis of Fairness and Bias Metrics

NUMBER of AI FAIRNESS and BIAS METRICS (DIAGNOSTIC METRICS vs. BENCHMARKS), 2016–21
Source: AI Index, 2021 | Chart: 2022 AI Index Report

15

Number of Metrics

Benchmarks
Diagnostic Metrics

13

10
9

9

9

5
4
3
2

3

2

2

1
0

0
0

1

2
2016

3

40

1

2
2017

3

40

1

2
2018

3

40

1

2
2019

3

40

1

2
2020

3

40

1

2
2021

3

4

Figure 3.1.2

The rest of this chapter examines the performance ofrecent AI systems on these metrics and benchmarks indepth within domains such as natural language andcomputer vision. The majority of these metrics measure

Table of Contents

Chapter 3 Preview

intrinsic bias within systems, and it has been shown that
intrinsic bias metrics may not fully capture the effects of
extrinsic bias within downstream applications.

108Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

Current state-of-the-art natural language processing (NLP) relies on large language models or machine learning systems that process
millions of lines of text and learn to predict words in a sentence. These models can generate coherent text; classify people, places, andevents; and be used as components of larger systems, like search engines. Collecting training data for these models often requires scraping
the internet to create web-scale text datasets. These models learn human biases from their pretraining data and reflect them in theirdownstream outputs, potentially causing harm. Several benchmarks and metrics have been developed to identify bias in natural language
processing along axes of gender, race, occupation, disability, religion, age, physical appearance, sexual orientation, and ethnicity.

3.2 NATURAL LANGUAGE PROCESSING
BIAS METRICS
Bias metrics can be split into two major categories: intrinsic
metrics, which measure bias in internal embedding spacesof models, and extrinsic metrics, which measure bias in
the downstream tasks and outputs of the model. Examples
of extrinsic metrics include group fairness metrics (parity
across protected groups) and individual fairness metrics(parity across similar individuals), which measure whethera system has a disproportionately negative impact on a
subgroup or individual, or gives preferential treatment to
one group at the expense of another.

TOX I C I T Y: REA LTOXICIT Y PROM PTS
AN D T H E PERS PECTIVE A PI
Measuring toxicity in language models requires labels for
toxic and nontoxic content. Toxicity is defined as a rude,

disrespectful or unreasonable comment that is likely tomake someone leave a conversation. The Perspective
API is a tool developed by Jigsaw, a Google company. It
was originally designed to help platforms identify toxicity
in online conversations. Developers input text into the
Perspective API, which returns probabilities that the text
should be labeled as falling into one of the following
categories: toxicity, severe toxicity, identity attack, insult,obscene, sexually explicit, and threat.
Since the Perspective API was released in 2017, the NLP
research community has rapidly adopted it for measuring
toxicity in natural language. As seen in Figure 3.2.1, thenumber of papers using the Perspective API doubled
between 2020 and 2021, from 8 to 19.

NUMBER of RESEARCH PAPERS USING PERSPECTIVE API
Source: AI Index, 2021 | Chart: 2022 AI Index Report

20
18

Number of Research Papers

15

10

5

0
2018

2019

2020

2021
Figure 3.2.1

Table of Contents

Chapter 3 Preview

109Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

RealToxicityPrompts consists of English natural language
prompts used to measure how often a language model
completes a prompt with toxic text. Toxicity of a language
model is measured with two metrics:

Figure 3.2.2 shows that toxicity in language models
depends heavily on the underlying training data. Models
trained on internet text with toxic content filtered out are
significantly less toxic compared to models trained on
various corpora of unfiltered internet text. A model trainedon BookCorpus (a dataset containing books from e-bookwebsites) produces toxic text surprisingly often. Thismay be due to its composition—BookCorpus contains asignificant number of romance novels containing explicit
content, which may contribute to higher levels of toxicity.

• M aximum toxicity: the average maximum toxicity scoreacross some number of completions• P robability of toxicity: how often a completion is
expected to be toxic

TOXICITY in LANGUAGE MODELS by TRAINING DATASET
Source: Gehman et al., 2021; Rae et al., 2021; Welbl et al., 2021 | Chart: 2022 AI Index Report

Expected maximum toxicity

Toxicity probability

0.90

0.88

0.85

Toxicity Score

0.80

0.78
0.73

0.82
0.75

0.71

0.60

0.40
0.25

0.20
0.08

0.00
BookCorpus

openwebtext (C4, toxicity!ltered)openwebtext (CTRL,
un!ltered)

openwebtext (GPT-3,
un!ltered)Wikipedia

Training Data
Figure 3.2.2

Table of Contents

Chapter 3 Preview

110Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

Large Language Models and Toxicity
Recent developments around mitigating toxicity
in language models have lowered both expected
maximum toxicity and the probability of toxicity.
However, detoxification methods consistently leadto adverse side effects and somewhat less capablemodels. (For example, filtering training datatypically comes at the cost of model performance.)
In December 2021, DeepMind released a paper
describing its 280 billion parameter language
model, Gopher. Figure 3.2.3a and Figure 3.2.3bfrom the Gopher paper show that larger models
are more likely to produce toxic outputs when

prompted with inputs of varying levels of toxicity,
but that they are also more capable of detecting
toxicity with regard to their own outputs as well as
in other contexts, as measured by increased AUC(area under the receiver operating characteristic
curve) with model size. The AUC metric plots thetrue positive rate against the false positive rate
to characterize how well a model distinguishes
between classes (higher is better). Larger models
are dramatically better at identifying toxic
comments within the CivilComments dataset, asshown in Figure 3.2.3b.

GOPHER: PROBABILITY of TOXIC CONTINUATIONS BASED on PROMPT TOXICITY by MODEL SIZE
Source: Rae et al., 2021 | Chart: 2022 AI Index Report

0.4

High

Toxicity Probability

0.3
Med

0.2

Low

0.1

Very Low
Unprompted

0.0
117

417

1,400

7,100

280,000

Model Size (Number of Parameters in millions)Figure 3.2.3a

Table of Contents

Chapter 3 Preview

111

Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

Large Language Models and Toxicity (cont’d)
GOPHER: FEW-SHOT TOXICITY CLASSIFICATION on the CIVILCOMMENTS DATASET
Source: Rae et al., 2021 | Chart: 2022 AI Index Report

10-shot

Classi!cation Area Under the Curve (AUC)

0.7

0.6

0.5

0.4

0.3

0.2

0.1
0.0
44

117

417

1,400

7,100

280,000

Model Size (Number of Parameters in millions)Figure 3.2.3b

Table of Contents

Chapter 3 Preview

112Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

D E TOX I F I CAT I O N O F M O D E L S
CA N N E G AT I V E LY I N F LU E N C E
PERFORMANCE
Detoxification methods aim to mitigate toxicity bychanging the underlying training data as in domainadaptive pretraining (DAPT), or by steering the model
during generation as in Plug and Play Language Models(PPLM) or Generative Discriminator Guided Sequence
Generation (GeDi).

A study on detoxifying language models shows thatmodels detoxified with these strategies all perform worse
on both white-aligned and African American English on
perplexity, a metric that measures how well a model has
learned a specific distribution (lower is better) (Figure
3.2.4). These models also perform disproportionately
worse on African American English and text containing
mentions of minority identities compared to whitealigned text, a result that is likely due to human biases
causing annotators to be more apt to mislabel African
American English as toxic.

PERPLEXITY: LANGUAGE MODELING PERFORMANCE by MINORITY GROUPS on ENGLISH
POST-DETOXIFICATION
Source: Xu et al., 2021 | Chart: 2022 AI Index Report

Baseline (GPT-2)
Detoxi!cation method - DAPT
Detoxi!cation method - GeDi
Detoxi!cation method - PPLM

500

Perplexity

400

300

200

100

0
African American–aligned English(nontoxic)

Minority identity mentions(nontoxic)

White-aligned English (nontoxic)

White-aligned English (toxic)

Text
Figure 3.2.4

Table of Contents

Chapter 3 Preview

113Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

STEREOSET
StereoSet is a benchmark measuring stereotype bias along
the axes of gender, race, religion, and profession, alongwith raw-language modeling ability. One of the associatedmetrics is stereotype score, which measures whether amodel prefers stereotypes and anti-stereotypes equally. A
stereotype is an over-generalized belief widely held abouta group and an anti-stereotype is a generalization about agroup which contradicts widely accepted stereotypes.

Figure 3.2.5 shows that StereoSet performance follows
the same trend seen with toxicity: Larger models reflect
stereotypes more often unless interventions are taken
to reduce learned stereotypes during training. The
prevalence of toxic content online has been estimatedto be 0.1–3%, which aligns with research showing thatlarger language models are more capable of memorizingrare text.

STEREOSET: STEREOTYPE SCORE by MODEL SIZE
Source: Nadeem et al., 2020; Lieber et al., 2021; StereoSet Leaderboard, 2021; | Chart: 2022 AI Index Report

64

62.00

Stereotype Score

62

60.70
60.10

60
59.30
58.30

58.20

58

56

110

340

345

1,558

175,000

178,000

Number of Parameters (in millions)Figure 3.2.5

StereoSet has several major flaws in its underlying dataset:
Some examples fail to express a harmful stereotype,conflate stereotypes about countries with stereotypes
about race and ethnicity, and confuse stereotypes
between associated but distinct groups. Additionally,

Table of Contents

Chapter 3 Preview

these stereotypes were sourced from crowdworkers
located in the United States, and the resulting values andstereotypes within the dataset may not be universally
representative.

114

Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

C R OW S - PA I R S
CrowS-Pairs (Crowdsourced Stereotype Pairs) is anotherbenchmark measuring stereotype bias. While StereoSet
compares attributes about a single group, CrowS-Pairs
contrasts relationships between historically disadvantaged
and advantaged groups (e.g., Mexicans versus white
people).
The creators of CrowS-Pairs measured stereotype bias
using three popular language models: BERT, RoBERTa,
and ALBERT (Figure 3.2.6). On standard language
modeling benchmarks, ALBERT outperforms RoBERTa,
which outperforms BERT.4 However, ALBERT is the mostbiased of the three models according to CrowS-Pairs.
This mirrors the trend observed with StereoSet and
RealToxicityPrompts: More capable models are also morecapable of learning and amplifying stereotypes.

Like earlier examples, BERT, RoBERTa, and ALBERT
appear to inherit biases from their training data. They
were all trained on a combination of BookCorpus, English
Wikipedia, and text scraped from the internet. Analysis
of BookCorpus reveals that its books about religion areheavily skewed toward Christianity and Islam comparedto other major world religions,5 though it is unclear theextent to which these books contain historical content
versus content written from a specific religious viewpoint.6
We can examine how language models may inherit biases
about certain religions by looking at their underlying
datasets. Figure 3.2.7 shows the number of books
pertaining to different religions in two popular datasets,
BookCorpus and Smashwords21. Both datasets havefar more mentions of Christianity and Islam than otherreligions.

CROWS-PAIRS: LANGUAGE MODEL PERFORMANCE across BIAS ATTRIBUTES
Source: Nangia et al., 2020 | Chart: 2022 AI Index Report

BERT

RoBERTaALBERT

80

Stereotype Score

60

40

20

0
Age

Disability

GenderIdentity

Nationality

PhysicalAppearance

Race/Color

Religion

SexualOrientation

Socioeconomic
StatusBias Attribute
Figure 3.2.6

4 Per results on the SQuAD, GLUE, and RACE benchmarks.
5 Such as Sikhism, Judaism, Hinduism, Buddhism, Atheism.
6 Hate speech classifiers fine-tuned on top of BERT in particular have been shown to frequently misclassify texts containing mentions of “Muslim” as toxic, and researchers find that GPT-3 contains
significant bias along religious axes for mentions of both “Jewish” and “Muslim.”

Table of Contents

Chapter 3 Preview

115Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

BOOKCORPUS and SMASHWORDS21: SHARE of BOOKS about RELIGION in PRETRAINING DATASETS
Source: Bandy and Vincent, 2021 | Chart: 2022 AI Index Report

Books (as % of All Religious Books in Corpus)

50%

BookCorpus
smashwords21

40%

30%

20%

10%

0%
Atheism

Buddhism

Christianity

Hinduism

Islam

Judaism

Sikhism

Religion
Figure 3.2.7

Table of Contents

Chapter 3 Preview

116Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

WINOGENDER AND WINOBIAS
Winogender measures gender bias related to occupations.
Systems are measured on their ability to fill in the correct
gender in a sentence containing an occupation (e.g.,“The teenager confided in the therapist because he / sheseemed trustworthy”). Examples were created by sourcing
data from the U.S. Bureau of Labor Statistics to identify
occupations skewed toward one gender (e.g., the cashieroccupation is made up of 73% women, but drivers are only
6% women).
Performance on Winogender is measured by the accuracy
gap between the stereotypical and anti-stereotypicalcases, along with the gender parity score (the percentageof examples for which the predictions are the same).
The authors use crowdsourced annotations to estimate
human performance to be 99.7% accuracy.
Winogender results from the SuperGLUE leaderboard
show that larger models are more capable of correctly
resolving gender in the zero-shot and few-shot setting(i.e., without fine-tuning on the Winogender task) and lesslikely to magnify occupational gender disparities (Figure
3.2.8). However, a good score on Winogender does notindicate that a model is unbiased with regard to gender,
only that bias was not captured by this benchmark.

MODEL PERFORMANCE on the WINOGENDER TASK from the SUPERGLUE BENCHMARK
Source: SuperGLUE Leaderboard, 2021 | Chart: 2022 AI Index Report

0.72

0.71
0.64

0.59

0.60

0.60

0.55

0.53

Coreference Accuracy

0.59
0.50

0.40

0.20

0.00
31

100

223

340
10,00013,000
Number of Parameters (in millions)

64,000175,000

280,000
Figure 3.2.8

Table of Contents

Chapter 3 Preview

117Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

WinoBias is a similar benchmark measuring gender bias
related to occupations that was released concurrently
with Winogender by a different research group. As
shown in Figure 3.2.9, WinoBias is cited more often than

Winogender, but the adoption of Winogender within the
SuperGLUE leaderboard for measuring natural language
understanding has led to more model evaluations being
reported on Winogender.

WINOBIAS and WINOGENDER: NUMBER of CITATIONS, 2018–21
Source: AI Index, 2021; Semantic Scholar, 2021 | Chart: 2022 AI Index Report

128, WinoBias

Number of Citations

100

95, Winogender

50

0
2018

2019

2020

2021
Figure 3.2.9

Table of Contents

Chapter 3 Preview

118

Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

W I N O M T: G E N D E R B I A S I N
M AC H I N E T R A N S L AT I O N SYS T E M S
Commercial machine translation systems have been
documented to reflect and amplify societal biases
from their underlying datasets. As these systems areused broadly in global industries such as e-commerce,stereotypes and mistakes in translation can be costly.
WinoMT is a benchmark measuring gender bias inmachine translation that is created by combining theWinogender and WinoBias datasets. Models are evaluatedby comparing the sentences translated from English to
another language and extracting the translated gender tocompare with the original gender. Systems are scored on
the percentage of translations with correct gender (genderaccuracy), the difference in F1 score between masculine
and feminine examples, and the difference in F1 score
between examples with stereotypical gender roles andanti-stereotypical gender roles.
As seen in Figure 3.2.10, Google Translate has beenshown to perform better across all tested languages(Arabic, English, French, German, Hebrew, Italian,
Russian, Ukrainian) when translating examples containing
occupations that conform to societal biases about genderroles. Additionally, these systems translate sentenceswith the correct gender only up to 60% of the time. Other
major commercial machine translation systems (Microsoft
Translator, Amazon Translate, SYSTRAN) have been shown
to behave similarly.

WINOMT: GENDER BIAS in GOOGLE TRANSLATE across LANGUAGES
Source: Stanovsky et al., 2019 | Chart: 2022 AI Index Report

Gender Accuracy

F1 Delta Masculine Feminine

F1 Delta Stereo Anti-stereo

ArabicArabicArabicEnglishEnglishEnglishFrenchFrenchFrenchGermanGermanGermanHebrew

Hebrew

Hebrew

ItalianItalianItalianRussianRussianRussian

Ukrainian

Ukrainian

Ukrainian

0

10

20

30

40

50

60

0

10

20

30

40

50

60

0

10

20

30

40

50

60

Figure 3.2.10

Table of Contents

Chapter 3 Preview

119Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

WO R D A N D I M AG E E M B E D D I N G
A S S O C I AT I O N T E S T S
Word embedding is a technique in NLP that allows words
with similar meanings to have similar representations.
Static word embeddings are fixed representations which
do not change with context. For example, polysemouswords will have the same representation (embedding)regardless of the sentence in which they appear. Examples
of static word embeddings include GloVe, PPMI, FastText,
CBoW, and Dict2vec. In contrast, contextualized wordembeddings are dynamic representations of words thatchange based on the word’s accompanying context. For
example, “bank” would have different representations in“riverbank” and “bank teller.”
The Word Embedding Association Test (WEAT) quantifies
bias in English static word embeddings by measuring
the association (“effect size”) between concepts (e.g.,
European-American and African American names) andattributes (e.g., pleasantness and unpleasantness).
Word embeddings trained on large public corpora(e.g., Wikipedia, Google News) consistently replicate
stereotypical biases when evaluated on WEAT (e.g.,associating male terms with “career” and female
terms with “family”). CEAT (Contextualized Embedding
Association Test) extends WEAT to contextualized wordembeddings.
The Image Embedding Association Test (iEAT) modifies
WEAT to measure associations between social concepts
and image attributes. Using iEAT, researchers showed that
pretrained generative vision models (iGPT and simCLRv2)
exhibit humanlike biases with regard to gender, race, age,and disability.
Word embeddings can be aggregated into sentence
embeddings with models known as sentence encoders.
The Sentence Encoder Association Test (SEAT) extends
WEAT to measure bias in sentence encoders related
to gendered names, regional names, and stereotypes.
Newer transformer-based language models which use
contextualized word embeddings appear to be less biased
than their predecessors, but most models still show
significant bias with regard to gender and occupations,as well as African American names versus EuropeanAmerican names, as shown in Figure 3.2.11.

SENTENCE EMBEDDING ASSOCIATION TEST (SEAT): MEASURING STEREOTYPICAL ASSOCIATIONS with EFFECT
SIZE
Source: May et al., 2019 | Chart: 2022 AI Index Report

Angry Black Woman Stereotype
European American/African American Names
Male/Female Names, CareerE"ect Size

1.5

1.0

0.5

0.0

10/2014 (CBoW)

9/2017 (InferSent)

2/2018 (ELMo)

3/2018 (GenSen)

3/2018 (Universal
Sentence Encoder)

6/2018 (GPT)

10/2018 (BERT)
Figure 3.2.11

Table of Contents

Chapter 3 Preview

120Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

Word embeddings also reflect cultural shifts: A temporal
analysis of word embeddings over 100 years of U.S. Census
text data shows that changes in embeddings closely track
demographic and occupational shifts over time. Figure
3.2.12 shows that shifts in embeddings trained on the
Google Books and Corpus of Historical American English(COHA) corpora reflect significant historical events like thewomen’s movement in the 1960s and Asian immigration
to the United States. In this analysis, embedding bias ismeasured with the relative norm difference: the average
Euclidean distance between words associated withrepresentative groups (e.g., men, women, Asians) andwords associated with occupations. The blue line shows
gender bias over time, where negative values indicatethat embeddings more closely associate occupations withmen. The red line shows the bias of embeddings relatingrace to occupations, specifically in the case of Asian
Americans and whites.

GENDER and RACIAL BIAS in WORD EMBEDDINGS TRAINED on 100 YEARS of TEXT DATA
Source: Garg et al., 2018 | Chart: 2022 AI Index Report-0.02
Racial Bias

Average Bias Score-0.03

Gender Bias

-0.04

-0.05

-0.06

1910

1920

1930

1940

1950

1960

1970

1980

1990
Figure 3.2.12

Table of Contents

Chapter 3 Preview

121Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.2 Natural Language Processing Bias Metrics

Multilingual Word Embeddings
Large language models are often monolingual
since they require a significant amount of text datato train. While English text can be easily sourced
by scraping the internet, the challenge is greaterwith low-resource languages like Fula. XWEAT is amultilingual and cross-lingual extension of WEAT
that is designed for comparative bias analyses
between languages. Results on XWEAT show thatbias in cross-lingual embeddings can roughly be
predicted from the biases in the corresponding
monolingual embedding, indicating that biases canbe transferred between languages.

Another study on gender bias extends WEAT
to quantify biases in bilingual embeddings inlanguages with grammatical gender, such as
Spanish or French. Figure 3.2.13 shows thatmasculine words in Spanish are closer to theEnglish words for historically male-dominatedoccupations (e.g., architect) as well as theneutral position, as indicated by the vertical line.
Similarly, feminine occupation words are closer toEnglish words for historically female-dominatedoccupations (e.g., nurse).

GENDER BIAS in SPANISH WORD EMBEDDINGS: EMBEDDING SIMILARITY DISTANCE
Source: Zhou et al., 2019 | Chart: 2022 AI Index Reportarchitectarquitectaarquitecto
lawyerabogadoabogadaprofesora

professor

doctoraprofesor

doctordoctorenfermeraenfermero

nurse

-0.3

-0.2

-0.1

0.0

0.1

Female <— Gender Direction —> Male
Figure 3.2.13

Mitigating Bias in Word Embeddings With
Intrinsic Bias Metrics
It is often assumed that reducing intrinsic bias by debiasing embeddings will reduce downstream biases
in applications (extrinsic bias). However, it has been

Table of Contents

Chapter 3 Preview

demonstrated that there is no reliable correlation
between intrinsic bias metrics and downstream
application biases. Further investigation is needed toestablish meaningful relationships between intrinsic andextrinsic metrics.

122Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.3 AI Ethics Trends at FAccT and NeurIPS

To grasp how the field of AI ethics has evolved over time, this section studies trends from the ACM Conference on Fairness, Accountability,
and Transparency (FAccT), which publishes work on algorithmic fairness and bias, and from NeurIPS workshops. The section identifies
emergent trends in workshop publication topics and shares insights on authorship trends by affiliation and geographic region.

3.3 AI ETHICS TRENDS AT FACCTAND NEURIPS
AC M CO NFERENCE O N FA IRNESS,
ACCO UNTA B ILIT Y, A ND
T RANS PA R ENCY (FACCT)ACM FAccT is an interdisciplinary conference publishingresearch in algorithmic fairness, accountability, andtransparency.7 While several AI conferences offer
workshops dedicated to similar topics, FAccT was oneof the first major conferences created to bring together
researchers, practitioners, and policymakers interested insociotechnical analysis of algorithms.

Figure 3.3.1 shows that industry labs are making up alarger share of publications at FAccT year over year. They
often produce work in collaboration with academia but
are increasingly producing standalone work as well. In
2021, 53 authors listed an industry affiliation, up from
31 authors in 2020 and only 5 authors at the inauguralconference in 2018. This aligns with recent findings thatpoint to a trend of deep learning researchers transitioning
from academia to industry labs.

NUMBER of ACCEPTED FACCT CONFERENCE SUBMISSIONS by AFFILIATION, 2018–21
Source: FAccT, 2021; AI Index, 2021 | Chart: 2022 AI Index Report

Number of Papers

300

Education
Industry
Government
Nonpro!t
Other

302

244

200

227

166
200

100

139

71
53

63
21

0
2018

2019

31
17

2020

2021
Figure 3.3.1

7 Work accepted by FAccT includes technical frameworks for measuring fairness, investigations into the harms of AI in specific industries (e.g., discrimination in online advertising, biases inrecommender systems), proposals for best practices, and better data collection strategies. Several works published at FAccT have become canonical works in AI ethics; examples include Model Cards for
Model Reporting (2019) and On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? (2021). Notably, FAccT publishes a significant amount of work critical of contemporary methods andsystems in AI.

Table of Contents

Chapter 3 Preview

123Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.3 AI Ethics Trends at FAccT and NeurIPS

While there has been increased interest in fairness,accountability, and transparency research from all typesof organizations, the majority of papers published at FAccT
are written by researchers based in the United States,

followed by researchers based in Europe and Central Asia
(Figure 3.3.2). From 2020 to 2021, the proportion of papers
from institutions based in North America increased from
70.2% to 75.4%.

NUMBER of ACCEPTED FACCT CONFERENCE SUBMISSIONS by REGION, 2018–21
Source: FAccT, 2021; AI Index, 2021 | Chart: 2022 AI Index Report

Number of Publications (% of World Total)

75.40%, North America

60%

40%

20%
17.34%, Europe and Central Asia
4.03%, East Asia and Paci c
1.61%, Latin America and the Caribbean
1.21%, Middle East and North Africa
0.40%, Sub-Saharan Africa
0.00%, South Asia

0%
2018

Table of Contents

2019

Chapter 3 Preview

2020

2021

Figure 3.3.2

124

Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.3 AI Ethics Trends at FAccT and NeurIPSN E U R I P S WO R KS H O P S
NeurIPS, one of the largest AI conferences, held its firstworkshop on fairness, accountability, and transparency in2014. Figure 3.3.3 shows the number of research papers

at NeurIPS ethics-related workshops in the past six years
by research topic, indicating an increased interest in
AI applied to high-risk, high-impact use cases such as
climate, finance, and healthcare.

NEURIPS WORKSHOP RESEARCH TOPICS: NUMBER of ACCEPTED PAPERS on REAL-WORLD IMPACTS, 2015–21
Source: NeurIPS, 2021; AI Index, 2021 | Chart: 2022 AI Index Report

250

ClimateDeveloping World
Finance
Healthcare
Other

243
19
29

200

191

Number of Papers

172
79

150
114
15

100

86

81

75
50

13

172
17

13

17

99
75

47

97
62

16

0
2015

2017

2018

2019

2020

2021
Figure 3.3.3

Table of Contents

Chapter 3 Preview

125Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.3 AI Ethics Trends at FAccT and NeurIPS

Interpretability, Explainability, andCausal Reasoning
Several workshops have been created at NeurIPS aroundinterpretability and explainability, including safetycritical AI affecting human decisions,8 interpretability
and causality for algorithmic fairness,9 and the necessityof explainability for high-risk use cases.10 Interpretability
and explainability work focus on designing systems thatare inherently interpretable and providing explanations
for the behavior of a black-box system, while the study of
causal inference aims to understand cause and effect by
uncovering associations between variables that dependon each other and asking what would have happened if a
different decision had been made—that is, if this had notoccurred, then that would not have happened.
Counterfactual analysis can be used to gain insight intoa black-box system by changing an input feature andobserving how the output changes. This can be applied tomeasure fairness by changing protected attributes of anindividual input (e.g., race, gender) and observing how themodel outputs a different prediction—for example, a bankcan change the “age” feature in a model to understand if
its model performs fairly on customers over 60 years old.
Counterfactual fairness formalizes the idea that a model
makes fair decisions with regard to an individual if thedecision would be the same if the individual belonged to adifferent demographic.
Since 2018, an increasing number of papers on causal
inference have been published at NeurIPS. In 2021, there
were three workshops at NeurIPS dedicated to causal
inference, including one devoted entirely to causality andalgorithmic fairness (Figure 3.3.4). Figure 3.3.5 shows that
there has been a similar increase in research papers ininterpretability and explainability work at NeurIPS overtime, especially in the NeurIPS main track.

NEURIPS RESEARCH TOPICS: NUMBER of ACCEPTED PAPERS on CAUSAL EFFECT and COUNTERFACTUALREASONING, 2015–2021
Source: NeurIPS, 2021; AI Index, 2021 | Chart: 2022 AI Index Report

Main Track
Workshop

80

78

76
72

20

Number of Papers

60
43

53

39

40

16

58

20

6
0

6

2015

9
4

9

2016

2017

29

23

2018

2019

2020

23

2021
Figure 3.3.4

8 See 2017 Transparent and Interpretable Machine Learning in Safety Critical Environments, 2019 Workshop on Human-Centric Machine Learning: Safety and Robustness in Decision-Making, 2019,“‘Do the Right Thing’: Machine Learning and Causal Inference for Improved Decision-Making.”
9 See 2020 “Algorithmic Fairness Through the Lens of Causality and Interpretability.”
10 See 2020 “Machine Learning for Health (ML4H): Advancing Healthcare for All,” 2020 Workshop on Fair AI in Finance.

Table of Contents

Chapter 3 Preview

126Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.3 AI Ethics Trends at FAccT and NeurIPS

Privacy and Data Collection
Amid growing concerns about privacy, data sovereignty,
and the commodification of personal data for profit, there
has been significant momentum in industry and academia
to build methods and frameworks to help mitigate
privacy concerns. Since 2018, several workshops havebeen devoted to privacy in machine learning, covering
topics such as privacy in machine learning within specificdomains (e.g., financial services), federated learning fordecentralized model training, and differential privacy
to ensure that training data does not leak personallyidentifiable information.11 This section shows the numberof papers submitted to NeurIPS mentioning “privacy” inthe title along with papers accepted to privacy-themed
NeurIPS workshops, and finds a significant increase in thenumber of accepted papers since 2016 (Figure 3.3.6).

NEURIPS RESEARCH TOPICS: NUMBER of ACCEPTED PAPERS on INTERPRETABILITY and EXPLAINABILITY, 2015–21
Source: NeurIPS, 2021; AI Index, 2021 | Chart: 2022 AI Index Report

41

Main Track
Workshop40

18

Number of Papers

30

23
20
17
12
10

7

19
23

5

6

6

2

6

7

2015

2016

2017

10
6

0
2018

2019

4

2020

2021
Figure 3.3.5

11 See “Privacy Preserving Machine Learning,” Workshop on Federated Learning for Data Privacy and Confidentiality, Privacy in Machine Learning (PriML).

Table of Contents

Chapter 3 Preview

127Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.3 AI Ethics Trends at FAccT and NeurIPS

NEURIPS RESEARCH TOPICS: NUMBER of ACCEPTED PAPERS on PRIVACY in AI, 2015–21
Source: NeurIPS, 2021; AI Index, 2021 | Chart: 2022 AI Index Report

Main Track
Workshop

150

150
12

128

Number of Papers

15

100
88
79

13
138
113

50

21

0

1
2015

19

2016

72

75

2018

2019

16
14

2017

2020

2021
Figure 3.3.6

Table of Contents

Chapter 3 Preview

128Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.3 AI Ethics Trends at FAccT and NeurIPS

Fairness and Bias
In 2020, NeurIPS started requiring authors to submit
broader impact statements addressing the ethical andpotential societal consequences of their work, a movethat suggests the community is signaling the importanceof AI ethics early in the research process. One measure of
the interest in fairness and bias at NeurIPS over time isthe number of papers accepted to the conference main
track that mention fairness or bias in the title, alongwith papers accepted to a fairness-related workshop.
Figure 3.3.7 shows a sharp increase from 2017 onward,
demonstrating the newfound importance of these topics
within the research community.

NEURIPS RESEARCH TOPICS: NUMBER of ACCEPTED PAPERS on FAIRNESS and BIAS in AI, 2015–21
Source: NeurIPS, 2021; AI Index, 2021 | Chart: 2022 AI Index Report

168
Main Track
Workshop

150

149
50
36

Number of Papers

125
16

100

36

109

50

2

4

2015

2016

113

118

2020

2021

78

34
10

0

114

24

2017

2018

2019

Figure 3.3.7

Table of Contents

Chapter 3 Preview

129Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.4 Factuality and Truthfulness

This section analyzes trends in using AI to verify the factual accuracy of claims, as well as research related to measuring the truthfulnessof AI systems. It is imperative that AI systems deployed in safety-critical contexts (e.g., healthcare, finance, disaster response) provide
users with knowledge that is factually accurate, but today’s state-of-the-art language models have been shown to generate false
information about the world, making them unsafe for fully automated decision making.

3.4 FACTUALITY AND TRUTHFULNESSFACT- C H EC KING W ITH A I
In recent years, social media platforms have deployed
AI systems to help manage the proliferation of online
misinformation. These systems may aid human factcheckers by identifying potential false claims for them toreview, surfacing previously fact-checked similar claims, or
surfacing evidence that supports a claim. Fully automatedfact-checking is an active area of research: In 2017, theFake News Challenge encouraged researchers to build
AI systems for stance detection, and in 2019, a Canadianventure capital firm invested $1 million in an automatedfact-checking competition for fake news.
The research community has developed several
benchmarks for evaluating automatic fact-checking
systems, where verifying the factuality of a claim is posed
as a classification or scoring problem (e.g., with twoclasses classifying whether the claim is true or false).
Figure 3.4.1 shows that most datasets binarize labels intotrue or false categories, while some datasets have many
categories for claims.

DATASETS for AUTOMATED FACT-CHECKING: GRANULARITY of LABELS
Source: AI Index, 2021 | Chart: 2022 AI Index Report

75

2
28

3
16

Number of Classes

4

15

5
6

6
3

7
13

1

17

1

18

1

40

1

Numeric

1
3

Ranking
0

10

20

30

40

50

60

70

80

Number of Datasets
Figure 3.4.1

Table of Contents

Chapter 3 Preview

130Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.4 Factuality and Truthfulness

The increased interest in automated fact-checking
is evidenced by the number of citations of relevantbenchmarks: FEVER is a fact extraction and verification
dataset made up of claims classified as supported, refuted,
or not enough information. LIAR is a dataset for fake news
detection with six fine-grained labels denoting varying

levels of factuality. Similarly, Truth of Varying Shades is amulticlass political fact-checking and fake news detection
benchmark. Figure 3.4.2 shows that these three English
benchmarks have been cited with increasing frequency inrecent years.

AUTOMATED FACT-CHECKING BENCHMARKS: NUMBER of CITATIONS, 2017–21
Source: AI Index, 2021 | Chart: 2022 AI Index Report

LIAR

200

Number of Citations

FEVER

150

Truth of Varying Shades

100

50

0
2017

2018

2019

2020

2021
Figure 3.4.2

Table of Contents

Chapter 3 Preview

131Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.4 Factuality and Truthfulness

Figure 3.4.3 shows the number of fact-checking datasets
created for English compared to all other languages
over time. As seen in Figure 3.4.4, there are only 35 non-English datasets (including 14 in Arabic, 5 in Chinese, 3
in Spanish, 3 in Hindi, and 2 in Danish) compared to 142
English-only datasets.12

NUMBER of AUTOMATED FACT-CHECKING BENCHMARKS for ENGLISH, 2010–21
Source: AI Index, 2021 | Chart: 2022 AI Index Report50

48
English
Non-English

Number of Datasets

40

30

38

27

20

19

20

25

13
23

12

15
18

10
6
0

1

0
2010

0

2011

2012

1
2013

9

12

10

2

6

2014

2015

3

2016

2017

4

5

2018

2019

2020

2021
Figure 3.4.3

NUMBER of AUTOMATED FACT-CHECKING BENCHMARKS by LANGUAGE
Source: AI Index, 2021 | Chart: 2022 AI Index Report142

English14

Arabic5Chinese3

Hindi
Spanish3

Danish2French2German2

Portugese2

Bengali 1
Bulgarian 1
Croatian 1
Italian 1
Malayalam 1
Tamil 1
0

20

40

60

80

100

120

140

Number of Benchmarks
Figure 3.4.4
12 Modern language models are trained on disproportionately larger amounts of English text, which negatively impacts performance on other languages. The Gopher family of models is trained on MassiveText(10.5 TB), which is 99% English. Similarly, only 7% of training data in GPT-3 was in languages other than English. See the Appendix for a comparison of a multilingual model (XGLM-564M) and GPT-3.

Table of Contents

Chapter 3 Preview

132Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.4 Factuality and Truthfulness

Measuring Fact-Checking Accuracy
With FEVER Benchmark
FEVER (Fact Extraction and VERification) is a benchmark
measuring the accuracy of fact-checking systems, where
the task requires systems to verify the factuality of aclaim with supporting evidence extracted from English
Wikipedia. Systems are measured on classification
accuracy and FEVER score, a custom metric whichmeasures whether the claim was correctly classified andat least one set of supporting evidence was correctly
identified. Several variations of this dataset havesince been introduced (e.g., FEVER 2.0, FEVEROUS,
FoolMeTwice).
Figure 3.4.5 shows that state-of-the-art performance has
steadily increased over time on both accuracy and FEVER
score. Some contemporary language models only reportaccuracy, as in the case of Gopher.

FACT EXTRACTION and VERIFICATION (FEVER) BENCHMARK: ACCURACY and FEVER SCORE, 2018–21
Source: AI Index, 2021 | Chart: 2022 AI Index Report

79.35, Label Accuracy

80

76.78, FEVER Score

Score

75

70

65

60
2018

2019

2020

2021
Figure 3.4.5

Table of Contents

Chapter 3 Preview

133Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.4 Factuality and Truthfulness

TOWAR D T RU THFUL L A NG UAGE
MO DE L S

“I don’t know” may be truthful but not informative). A
combination of metrics adapted from machine translation(BLEURT, ROUGE, and BLEU) are used to compare themodel answer to reference answers. In this setting, a small
model called GPT-Judge is also learned to predict humanevaluations of truthfulness and informativeness for agiven answer. Alternatively, the task can be formulated
as a multiple-choice task where models are evaluated on
their accuracy in selecting the correct answer.

While language models have become increasingly
more coherent, they have also become more fluent at
generating factually incorrect statements and fabricating
falsehoods. This increased fluency means they also havea greater capacity to perpetuate harm by, for instance,
creating convincing conspiracy theories.

Model Size and Truthfulness

In the multiple-choice version of this task, initial
experiments on GPT-Neo, GPT-2, T5 (UnifiedQA), and GPT3 showed that larger models provide more informative
answers but are not necessarily more truthful. Later
experiments on DeepMind’s Gopher model contradicted
this finding: Figure 3.4.6 from the Gopher paper
shows that accuracy improves with model size on themultiple-choice task. This contradiction may be due tothe formulation of the TruthfulQA dataset, which wascollected adversarially against GPT-3 175-B, possibly
explaining the lower performance of the GPT-3 family ofmodels.

TruthfulQA is a benchmark designed to evaluate thetruthfulness of language models on question answering,with questions drawn from categories such as health,law, finance, and politics. Questions were designed suchthat humans might be prone to answer incorrectly due
to common misconceptions: For example, when asked,“What happens if you smash a mirror?” GPT-3 responds,“You will have seven years of bad luck.”
The task is formulated as an open-ended generativetask, where systems are evaluated on the truthfulness
and informativeness of their answers (e.g., answering

TRUTHFULQA MULTIPLE-CHOICE TASK: TRUTHFUL and INFORMATIVE ANSWERS by MODEL

40%

30%

20%

10%

Gopher 280B
-10shot

Gopher 280B

Gopher 7.1B

Gopher 1.4B

GPT3 175B

GPT3 6.7B

GPT3 1.3B

GPT3 350M

GPT-2 1.5B

GPT-2 117M

T5 2.8B

T5 770M

T5 220M

T5 60M

GPT-NEO-6B

GPT-NEO-2.7B

GPT-NEO-1.3B

0%
GPT-NEO-125M

Truthful and Informative Answers (%)

Source: Rae et al., 2021 | Chart: 2022 AI Index Report

Model
Figure 3.4.6

Table of Contents

Chapter 3 Preview

134Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.4 Factuality and Truthfulness

WebGPT was designed to improve the factual accuracyof GPT-3 by introducing a mechanism to search the Web
for sources to cite when providing answers to questions.
Similar to Gopher, WebGPT also shows more truthfuland informative results with increased model size. Whileperformance improves compared to GPT-3, WebGPT
still struggles with out-of-distribution questions, and its
performance is considerably below human performance.
However, since WebGPT cites sources and appears moreauthoritative, its untruthful answers may be more harmful
as users may not investigate cited material to verify eachsource.
InstructGPT models are a variant of GPT-3 and they use
human feedback to train a model to follow instructions,created by fine-tuning GPT-3 on a dataset of humanwritten responses to a set of prompts. The fine-tuned

models using human-curated responses are called SFT(supervised fine-tuning). The baseline SFT is further
fine-tuned using reinforcement learning from human
feedback. This family is called PPO because it uses atechnique called Proximal Policy Optimization. Finally,
PPO models are further enhanced and called InstructGPT.
Figure 3.4.7 shows the truthfulness of eight language
model families on the TruthfulQA generation task. Similar
to the scaling effect observed in the Gopher family, theWebGPT and InstructGPT models yield more truthful andinformative answers as they scale. The exception to thescaling trend is the supervised fine-tuned InstructGPT
baseline, which corroborates observations from theTruthfulQA paper that the baseline GPT-3 family of models
underperforms with scale.

TRUTHFULQA GENERATION TASK: TRUTHFUL and INFORMATIVE ANSWERS by MODEL

60%

40%

20%

InstructGPT 175B

InstructGPT 6B

InstructGPT 1.3B

PPO- netuned SFT 175B

PPO- netuned SFT 6B

PPO- netuned SFT 1.3B

SFT 175B

SFT 6BSFT 1.3B

WebGPT 175B

WebGPT 13B

WebGPT 760M

GPT3 175B

GPT3 13B

GPT3 760M

T5 2.8B

T5 770M

T5 220M

T5 60M

GPT-2 1.5B

GPT-2 117M

GPT-NEO-6B

GPT-NEO-2.7B

GPT-NEO-1.3B

0%
GPT-NEO-125M

Truthful and Informative Answers (%)

Source: Rae et al., 2021; Nakano, 2021; Ouyang, 2022 | Chart: 2022 AI Index Report

Model
Figure 3.4.7

Table of Contents

Chapter 3 Preview

135Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.4 Factuality and Truthfulness

Multimodal Biases in Contrastive
Language-Image Pretraining (CLIP)
Techniques used in natural language processing
such as the transformer architecture have recently
been adapted to the vision and multimodal
domains. General-purpose models such as CLIP,
ALIGN, FLAVA, Florence, and Wu Dao 2 aretrained on joint vision-language datasets compiled
from the internet and can be used for a wide rangeof downstream vision tasks, such as classification.
CLIP (Contrastive Language-Image Pretraining) is
a model that learns visual concepts from natural
language by training on 400 million image-text
pairs scraped from the internet, and it is capableof outperforming the best ImageNet-trainedmodels on a variety of visual classification tasks.
Like other models pretrained on internet corpora,
CLIP exhibits biases along gender, race, and age.
However, while benchmarks exist for measuring
bias within computer vision and natural language,
there are no well-established metrics formeasuring multimodal bias. This section provides
insight into some ways that researchers haveprobed CLIP for bias.

Denigration Harm
Exploratory probes show that the design of
categories used in the model (i.e., ground-truthlabels) heavily influences the biases manifested
by CLIP. Probing the model by adding non-humanand crime-related classes such as “animal,”“gorilla,” “chimpanzee,” “orangutan,” “thief,”

Table of Contents

Chapter 3 Preview“criminal,” and “suspicious person” to the FairFace
dataset classes resulted in images of Black
people being misclassified as nonhuman at asignificantly higher rate than any other race (14%,compared to the next highest misclassification
rate of 7.6% for images of Indians). People ages
20 years old and younger were also more likely to
be assigned to crime-related classes compared toall other age groups.

Gender Bias
Probing CLIP with the Members of Congressdataset shows that labels such as “nanny” and“housekeeper” were associated with women,
whereas labels such as “prisoner” and “mobster”
were associated with men. Figure 3.4.8 shows
the percentage of images in the Members ofCongress dataset that are attached to a certain
label by gender, reflecting similar gender biases
found in commercial image recognition systems.
Additionally, CLIP almost exclusively associates
high-status occupation labels like “executive”
and “doctor” with men, and disproportionately
attaches labels related to physical appearance
to women. These experiments show that design
decisions such as selecting the correct similarity
thresholds can have outsized impacts on model
performance and biases.

136

Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.4 Factuality and Truthfulness

Multimodal Biases in Contrastive
Language-Image Pretraining (CLIP) (cont’d)
BIAS in CLIP: FREQUENCY of IMAGE LABELS by GENDER
Source: Agarwal et al., 2021 | Chart: 2022 AI Index Report

Women

Men

Lady
Woman
FemaleLooking
Senior Citizen
Public Speaking
Spokesperson
BlondeLaughing
BlazerBob Cut
MagentaHot
Black Hair
Pixie Cut
Bangs
Pink
Newsreader
PurpleBlouse

Man
Male
Face
PlayerBlack
Head
Facial Expression
Suit
Photo
Military O#cer
Walking
Photograph
Elder
Tie
Display
ShoulderFrown
Kid
Necktie
Yellow

Man
Woman

0%

20%

40%

60%

Frequency (%)

80%

100%

Man
Woman

0%

20%

40%

60%

80%

100%

Frequency (%)
Figure 3.4.8

Table of Contents

Chapter 3 Preview

137

Artificial IntelligenceIndex Report 2022

CHAPTER 3: TECHNICAL AI ETHICS
3.4 Factuality and Truthfulness

Multimodal Biases in Contrastive
Language-Image Pretraining (CLIP) (cont’d)
Propagating Learned Bias Downstream
CLIP has also been shown to learn historical biases
and conspiracy theories from its internet-sourced
training dataset. As one example of learned
historical bias, Figure 3.4.9 shows that CLIP assigns
higher similarity to “housewife with an orange
jumpsuit” to a picture of astronaut Eileen Collins.

This is problematic when CLIP is used for curating
datasets. Embeddings from CLIP were used to filter
the LAION-400M for high-quality image-text pairs;
however, the biases learned by CLIP were shown tobe propagated to LAION-400M, thus affecting any
future applications built with LAION-400M.

RESULTS OF THE
CLIP-EXPERIMENTSPERFORMED WITH THE
COLOR IMAGE OF THE
ASTRONAUT EILEEN
Source: Birthane et al., 2021

Figure 3.4.9

Underperformance on Non-English
Languages
CLIP can be extended to non-English languages
by replacing the original English text encoderwith a pretrained, multilingual model such as
Multilingual BERT (mBERT) and fine-tuning
further. However, its documentation cautions
against using the model for non-English
languages since CLIP was trained only on English

text, and its performance has not been evaluatedon other languages.
However, mBERT has performance gaps on lowresource languages such as Latvian or Afrikaans,14
which means that multilingual versions of CLIP
trained with mBERT will still underperform. Evenfor high-resource languages, such as French andSpanish, there are still noticeable accuracy gaps
in gender and age classification.

14 While mBERT performs well on high-resource languages like French, on 30% of languages (out of 104 total languages) with lower pretraining resources, it performs worse than using no pretrainedmodel at all.

Table of Contents

Chapter 3 Preview

138

Artificial IntelligenceIndex Report 2022

CHAPTER 4:

The Economy
and Education

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATIONCHAPTER 4:

Chapter Preview
Overview

141

4.3 CORPORATE ACTIVITY

160

Chapter Highlights

142

Industry Adoption

160

Global Adoption of AI

160

4.1 JOBS143

AI Adoption by Industry and Function

161

AI Hiring143

Type of AI Capabilities Adopted

162

AI Labor Demand

145

Consideration and Mitigation of Risks
From Adopting AI

163

Global AI Labor Demand

145

U.S. AI Labor Demand: By Skill Cluster

146

U.S. Labor Demand: By Sector

147

4.4 AI EDUCATION

165

U.S. Labor Demand: By State

147

165

AI Skill Penetration

149

CS Undergraduate Graduates inNorth America	Global Comparison

149

New CS PhDs in North America

166

Global Comparison: By Industry

149

New CS PhDs by Specialty

166

Global Comparison: By Gender150

New CS PhDs with AI/ML
and Robotics/Vision Specialties

167

4.2 INVESTMENT

151

Corporate Investment

151

Startup Activity

152

Global Trend

152

Regional Comparison by Funding Amount 154
Regional Comparison by Newly Funded
AI Companies
Focus Area Analysis

Table of Contents

New AI PhDs Employment in North America 168
Academia vs. Industry vs. Government 168
Diversity of New AI PhDs in North America 169
By Gender

169

By Race/Ethnicity

170

New International AI PhDs in North America 171

156
158

ACCESS THE PUBLIC DATA

140

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION

Overview
The growing use of artificial intelligence (AI) in everyday life, acrossindustries, and around the world generates numerous questions about
how AI is shaping the economy and education—and, conversely, how
the economy and education are adapting to AI. AI promises many
opportunities in workplace productivity, supply chain efficiency,
customized consumer experiences, and other areas. At the same time,
however, the technology gives rise to a number of concerns. How
do businesses adapt to recruiting and retaining AI talent? How is theeducation system keeping pace with the demand for AI labor and theneed to understand AI’s impact on society? All of these questions andmore are inextricable from AI today.
This chapter examines the economy and education, using datafrom Emsi Burning Glass, NetBase Quid, and LinkedIn to capture AI
trends in the global economy and data from the annual Computing
Research Association Taulbee Report to analyze trends in AI andcomputer science PhD graduates. The chapter first examines AI’s
impact on jobs, including hiring, labor demand, and skill penetration
rate, followed by an analysis of corporate investments in AI—from
global trends to startup activity in the space, and the adoption of AI
technologies among industries. The final section discusses computerscience (CS) undergraduate graduates and PhD graduates who
specialize in AI-related disciplines.

Table of Contents

Chapter 4 Preview

141

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATIONCHAPTER HIGHLIGHTS• New Zealand, Hong Kong, Ireland, Luxembourg, and Sweden are the countries or regions withthe highest growth in AI hiring from 2016 to 2021.
• In 2021, California, Texas, New York, and Virginia were states with the highest number of AI job
postings in the United States, with California having over 2.35 times the number of postingsas Texas, the second greatest. Washington, D.C., had the greatest rate of AI job postingscompared to its overall number of job postings.
• The private investment in AI in 2021 totaled around $93.5 billion—more than double thetotal private investment in 2020, while the number of newly funded AI companies continues
to drop, from 1051 companies in 2019 and 762 companies in 2020 to 746 companies in 2021.
In 2020, there were 4 funding rounds worth $500 million or more; in 2021, there were 15.
• “Data management, processing, and cloud” received the greatest amount of private AI
investment in 2021—2.6 times the investment in 2020, followed by “medical and healthcare”
and “fintech.”• In 2021, the United States led the world in both total private investment in AI and the numberof newly funded AI companies, three and two times higher, respectively, than China, the next
country on the ranking.
• Efforts to address ethical concerns associated with using AI in industry remain limited,
according to a McKinsey survey. While 29% and 41% of respondents recognize “equity andfairness” and “explainability” as risks while adopting AI, only 19% and 27% are taking steps
to mitigate those risks.
• In 2020, 1 in every 5 CS students who graduated with PhD degrees specialized in artificial
intelligence/machine learning, the most popular specialty in the past decade. From 2010 to2020, the majority of AI PhDs in the United States headed to industry while a small fraction
took government jobs.

Table of Contents

Chapter 4 Preview

142Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.1 Jobs

4.1 JOBSAI HIRING

example, an index of 1.05 in December 2021 points to ahiring rate that is 5% higher than the average month in2016. LinkedIn makes month-to-month comparisons toaccount for any potential lags in members updating theirprofiles. The index for a year is the number in Decemberof that year.

The AI hiring data draws on a dataset from LinkedIn
of skills and jobs listings on the platform. It focuses
specifically on countries or regions where LinkedIn covers
at least 40% of the labor force and where there are atleast 10 AI hires each month. China and India were also
included due to their global importance, despite notmeeting the 40% coverage threshold. Insights for thesecountries may not provide as full a picture as others, andshould be interpreted accordingly.

The relative AI hiring index captures whether hiring ofAI talent is growing faster than, equal to, or more slowly
than overall hiring in a particular country or region.
New Zealand has the highest growth in AI hiring—2.42
times greater in 2021 compared with 2016, followed byHong Kong (1.56), Ireland (1.28), Luxembourg (1.26),
and Sweden (1.24). Moreover, many countries or regions
experienced a decrease in their AI hiring growth from2020 to 2021—indicating that the pace of change in the
AI hiring rate, against the rate of overall hiring, declined
over the last year, with the exception of Germany andSweden (Figure 4.1.2).

Figure 4.1.1 shows the 15 geographic areas with thehighest relative AI hiring index for 2021. The AI hiring
rate is calculated as the percentage of LinkedIn memberswith AI skills on their profile or working in AI-related
occupations who added a new employer in the same
period the job began, divided by the total number ofLinkedIn members in the corresponding location. Thisrate is then indexed to the average month in 2016; for
RELATIVE AI HIRING INDEX by GEOGRAPHIC AREA, 2021
Source: LinkedIn, 2021 | Chart: 2022 AI Index Report

2.42

New Zealand
1.56

Hong Kong
Ireland

1.28

Luxembourg

1.26

Sweden1.24

Netherlands

1.23

United Kingdom

1.20

China

1.18

United States

1.17

Australia

1.14

Canada1.14

South Africa

1.13

Spain

1.10

Denmark

1.09

Germany

1.08

0.00

0.50

1.00

1.50
Relative AI Hiring Index

2.00

2.50
Figure 4.1.1

Table of Contents

Chapter 4 Preview

143Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.1 Jobs

RELATIVE AI HIRING INDEX by GEOGRAPHIC AREA, 2016–21
Source: LinkedIn, 2021 | Chart: 2022 AI Index Report

Australia

Canada

1.24

China

Denmark
1.08

1.00

1.18

1.07

0.00
Relative AI Hiring Index

Germany

Hong Kong

Ireland1.15

1.00

Luxembourg
1.14
1.16

1.13

0.00
Netherlands

New Zealand
1.20

South Africa

1.40

1.00

Spain
1.18

1.14

0.00
SwedenUnited Kingdom

1.24

United States
1.16

1.00

1.12

0.00
2018

2020

2022

2018

2020

2022

2018

2020

2022

2018

2020

2022
Figure 4.1.2

Table of Contents

Chapter 4 Preview

144

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.1 Jobs

AI LABOR DEMAND

Global AI Labor Demand

To analyze demand for specific AI labor skills, Emsi
Burning Glass mined millions of job postings collectedfrom over 45,000 websites since 2010 and flagged alllistings calling for AI skills.

Figure 4.1.3 shows that the percentage of AI job postings
among all job postings in 2021 was greatest in Singapore(2.33% of all job listings), followed by the United States(0.90%), Canada (0.78%), and the United Kingdom(0.74%). AI job postings increased in the United States,
Canada, Australia, and New Zealand from 2020 to2021, while they declined in Singapore and the United
Kingdom.

AI JOB POSTINGS (% of ALL JOB POSTINGS) by GEOGRAPHIC AREA, 2013–21
Source: Emsi Burning Glass, 2021 | Chart: 2022 AI Index Report

2.50%

AI Job Postings (% of All Job Postings)

2.33%, Singapore

2.00%

1.50%

1.00%

0.90%, United States
0.78%, Canada
0.74%, United Kingdom
0.58%, Australia

0.50%

0.25%, New Zealand

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 4.1.3

Table of Contents

Chapter 4 Preview

145

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.1 Jobs

The share of AI job postings
among all job postingsin 2021 was greatest formachine learning skills (0.6%of all job postings), followed
by artificial intelligence(0.33%), neural networks(0.16%), and natural
language processing (0.13%).

U.S. AI Labor Demand: By Skill Cluster
Figure 4.1.4 shows the U.S. labor demand from 2010
to 2021 by skill cluster. Each skill cluster consists of alist of AI-related skills; for example, the neural network
skill cluster includes skills like deep learning andconvolutional neural networks.1 The share of AI job
postings among all job postings in 2021 was greatest
for machine learning skills (0.6% of all job postings),
followed by artificial intelligence (0.33%), neuralnetworks (0.16%), and natural language processing(0.13%). Postings for AI jobs in machine learning andartificial intelligence have significantly increased in
the past couple of years, despite small declines inboth categories from 2019–2020. Machine learningjobs are at nearly three times the level, and artificial
intelligence jobs are at around 1.5 times the level theyeach reached, respectively, in 2018.

AI JOB POSTINGS (% of ALL JOB POSTINGS) in the UNITED STATES by SKILL CLUSTER, 2010–21
Source: Emsi Burning Glass, 2021 | Chart: 2022 AI Index Report

0.57%, Machine Learning

AI Job Postings (% of All Job Postings)

0.50%

0.40%
0.33%, Arti cial Intelligence

0.30%

0.20%
0.15%, Neural Networks
0.13%, Natural Language Processing (NLP)
0.11%, Robotics
0.10%, Visual Image Recognition
0.06%, Autonomous Driving

0.10%

0.00%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 4.1.4

1 See the Appendix for a complete list of AI skills under each skill cluster.

Table of Contents

Chapter 4 Preview

146

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.1 Jobs

U.S. Labor Demand: By Sector

followed by professional, scientific, and technicalservices (2.59% of all listings), manufacturing (2.02%),
and finance and insurance (1.81%).

Figure 4.1.5 shows that 3.30% of all job postings in theinformation sector in the United States were AI-related,

AI JOB POSTINGS (% of ALL JOB POSTINGS) in the UNITED STATES by SECTOR, 2021
Source: Emsi Burning Glass, 2021 | Chart: 2022 AI Index Report

3.30%

Information
2.59%

Professional, Scienti!c, and Technical Services
2.02%

Manufacturing
1.81%

Finance and Insurance
0.95%

Agriculture, Forestry, Fishing and Hunting

0.95%

Public Administration

0.84%

Educational Services

0.78%

Utilities

0.69%

Mining, Quarrying, and Oil and Gas Extraction
Management of Companies and Enterprises

0.67%

Wholesale Trade

0.63%
0.54%

Retail Trade

0.40%

Real Estate and Rental and Leasing

0.33%

Transportation and Warehousing

0.28%

Waste Management and Remediation Services
0.00%

1.00%

2.00%
AI Job Postings (% of All Job Postings)

3.00%
Figure 4.1.5

U.S. Labor Demand: By State
Figure 4.1.6 breaks down the U.S.
AI labor demand by state. In 2021,
the top states posting AI jobs
were California (80,238), Texas(34,021), New York (24,494), andVirginia (19,387). California, in first,
had over 2.35 times the numberof postings as Texas, the second
greatest. Proportionally, however,Washington, D.C., had the greatest
rate of AI job postings compared toits overall number of job postings(Figure 4.1.7). That was followed by
Virginia, Washington, Massachusetts,
and California.

NUMBER of AI JOB POSTINGS in the UNITED STATES by STATE, 2021
Source: Emsi Burning Glass, 2021 | Chart: 2022 AI Index Report

ME
693

AK
481

HI
866

VT
552

NH
1,211
MA
18,430WA
19,253

ID
2,309

MT
629

ND
726

MN
7,453

IL
15,775

WI
5,142

MI
9,935

NY
24,494

RI
1,090

OR
4,393

NV
2,253

WY
340

SD
693

IA
2,388

IN
3,899

OH
9,593

PA
10,806NJ
10,503

CT
4,291

CA
80,238

UT
3,631

CO
11,189

NE
1,385

MO
5,314

KY
2,107

WV
706

VA
19,387

MD
7,745

DE
1,764

AZ
9,606

NM
1,719

KS
2,058

AR
1,539

TN
5,219

NC
12,863

SC
2,259

DC
6,381

OK
1,462

LA
1,437

MS
986

AL
3,605

GA
14,283TX
34,021

FL
14,972

Figure 4.1.6

Table of Contents

Chapter 4 Preview

147

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.1 Jobs

AI JOB POSTINGS (TOTAL and % of ALL JOB POSTINGS) by U.S. STATE and DISTRICT, 2021
Source: Emsi Burning Glass, 2021 | Chart: 2022 AI Index Report

District of Columbia

AI Job Postings (% of All Job Postings)

2.00%

Virginia

1.50%

Washington
Massachusetts
Delaware1.00%

MarylandNew Mexico
Vermont

Utah

Maine

Wyoming

HawaiiArkansas

Mississippi

Kansas0.50%
AlaskaCalifornia

New York
Georgia
Illinois

Minnesota

Texas

Missouri Arizona

Alabama

Ohio
FloridaIndianaLouisiana

0.00%
200

500

1,000

2,000

5,000

10,00020,00050,000100,000

Number of AI Job Postings (Log Scale)Figure 4.1.7

Table of Contents

Chapter 4 Preview

148

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.1 Jobs

A I S K I L L P E N E T R AT I O N

AI skills is measured as the sum of the penetration ofeach AI skill across occupations in a given country or
region, divided by the global average across the same
occupations. For example, a relative penetration rate of
2 means that the average penetration of AI skills in thatcountry or region is 2 times the global average across thesame set of occupations. Figure 4.1.8 shows that India
led the world in the rate of AI skill penetration—3.09
times the global average from 2015 to 2021—followedby the United States (2.24) and Germany (1.7). After thatcame China (1.56), Israel (1.52), and Canada (1.41).2

The AI skill penetration rate shows the prevalenceof AI skills across occupations, or the intensity withwhich LinkedIn members use AI skills in their jobs. It is
calculated by computing the frequencies of LinkedIn
users’ self-added skills in a given area from 2015–2021,
then reweighting those figures by using a statisticalmodel to get the top 50 representative skills in thatoccupation.

Global Comparison
For global comparison, the relative penetration rate of

RELATIVE AI SKILL PENETRATION RATE by GEOGRAPHIC AREA, 2015–21
Source: LinkedIn, 2021 | Chart: 2022 AI Index Report

3.09

India
2.24

United States
1.70

Germany
1.56

China

1.52

Israel
1.41

Canada1.40

United Kingdom
1.28South Korea

1.22

Singapore
1.11

France
0.84

Spain

0.77

Australia
Switzerland

0.76

Italy

0.74
0.60

Sweden
0.00

0.50

1.00

1.50

2.00

2.50

3.00

Relative AI Skill Penetration Rate
Figure 4.1.8

Global Comparison: By Industry
India and the United States had the highest relative
AI skill penetration across the board—leading theother countries or regions in skill penetration rates insoftware and IT services, hardware and networking,manufacturing, education, and finance (Figure 4.1.9).
Israel and Canada are among the top seven countries
across all five industries, and Singapore holds the fourth
position on the list.

2 Those included are a sample of eligible countries or regions with at least 40% labor force coverage by LinkedIn and at least 10 AI hires in any given month. China and India were also included in thissample because of their increasing importance in the global economy, but LinkedIn coverage in these countries does not reach 40% of the workforce. Insights for these countries may not provide as full apicture as in others, and should be interpreted accordingly.

Table of Contents

Chapter 4 Preview

149Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.1 Jobs

RELATIVE AI SKILL PENETRATION RATE by INDUSTRY across GEOGRAPHIC AREA, 2015–21
Source: LinkedIn, 2021 | Chart: 2022 AI Index Report

Education

Finance
3.96

India
United States
IsraelGermany
CanadaSouth Korea
United Kingdom
France

2.20
1.35
1.27
1.23
1.18
0.91
0.80

1.00

2.00

3.00

Hardware

India
United States
Germany
CanadaSingapore
IsraelNetherlands
United Kingdom

4.00

3.08
1.89
1.48
1.31
1.28
1.15
1.12
1.00

0.00 1.00

Manufacturing

2.00

3.00

India
United States
IsraelChina
Germany
CanadaSingapore
South Korea

2.92
1.98
1.37
1.34
1.22
1.12
1.12
0.88
0.00

1.00

2.00

3.00

Software
2.39

India
United States
Germany
IsraelSwedenSingapore
CanadaChina

India
United States
China
Germany
IsraelCanadaSingapore
South Korea

2.21
1.66
1.31
1.16
1.04
0.99
0.93

0.00

1.00

3.02
2.18
1.58
1.38
1.22
1.20
1.16
1.13

0.00 1.00
2.00
3.00
Relative AI Skill Penetration Rate

2.00

Figure 4.1.9

Global Comparison: By Gender
Figure 4.1.10 shows the aggregated data from 2015 to 2021 of AI skills penetration by geographic area for female andmale talent. The data suggests that among the 15 countries listed, the AI skill penetration rates of females are higher thanthose of males in India, Canada, South Korea, Australia, Finland, and Switzerland.
RELATIVE AI SKILL PENETRATION RATE by GENDER, 2015–21
Source: LinkedIn, 2021 | Chart: 2022 AI Index Report

India
United States
CanadaSouth Korea
China
Singapore
Germany
IsraelAustralia
FinlandUnited Kingdom
Netherlands

FemaleMaleFranceSwitzerland
Brazil
0.00

1.00

2.00

3.00

4.00

Relative AI Skill Penetration Rate
Figure 4.1.10

Table of Contents

Chapter 4 Preview

150Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.2 Investment

This section on corporate AI activity draws on data from NetBase Quid, which aggregates over 6 million global public and privatecompany profiles, updated on a weekly basis, including metadata on investments, location of headquarters, and more. NetBase Quidalso applies natural language processing technology to search, analyze, and identify patterns in large, unstructured datasets, likeaggregated blogs, company and patent databases.

4.2 INVESTMENTC O R P O R AT E I N V E S T M E N T
Corporate investment in artificial intelligence, frommergers and acquisitions to public offerings, is a key
contributor to AI research and development. It also
contributes to AI’s impact on the economy. Figure 4.2.1
highlights overall global corporate investment in AI from2013–2021. In 2021, companies made the greatest AI

investment through private investment (totaling around$93.5 billion), followed by mergers and acquisitions(around $72 billion), public offerings (around $9.5
billion), and minority stake (around $1.3 billion). In
2021, investments from mergers and acquisitions grewby 3.3 times compared to 2020, led by two AI healthcarecompanies and two cybersecurity companies.3

GLOBAL CORPORATE INVESTMENT in AI by INVESTMENT ACTIVITY, 2013–21
Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report

Total Investment (in billions of U.S. Dollars)

176.47Merger/Acquisition
Minority Stake
Private Investment
Public O"ering

150

72.03

119.54
21.53

100

66.98
47.33

50

14.99
0

22.78

2013

2014

20.42

22.49

29.17

5.23

45.58

19.83

8.98

15.73

2015

2016

44.06
93.54

46.00
35.26

42.57

2018

2019

23.09

2017

7.94

9.58

2020

2021
Figure 4.2.1

3 Nuance Communications (by Microsoft, $19.8 billion), Varian Medical Systems (Siemens, $17.2 billion), and Proofpoint (Thoma Bravo, $12.4 billion) in the United States, followed by Avast in the
Czech Republic (NortonLifeLock, $8.0 billion).

Table of Contents

Chapter 4 Preview

151Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.2 Investment

S TA R T U P ACT I V I T Y

However, Figure 4.2.3 shows that the number of newly
funded AI companies continues to drop, from 762
companies in 2020 to 746 companies in 2021—thethird year of a decline that started in 2018. The largest
private investments in 2021 have been led by two data
management companies and two robotics/autonomous
driving companies.4

The following section analyzes artificial intelligence andmachine learning companies globally that have received
more than $1.5 million in investment from 2013 to 2021.

Global Trend
In 2021, global private investment in AI totaled around$93.5 billion, which is more than double the total
private investment in 2020 (Figure 4.2.2). That marks
the greatest year-over-year increase since 2014 (wheninvestment from 2013 to 2014 more than doubled).
Among companies that disclosed the amount of funding,the number of AI funding rounds that ranged from$100 million to $500 million more than doubled in 2021
compared to 2020, while funding rounds that were
between $50 million and $100 million more than doubled
as well (Table 4.2.1). In 2020, there were only four funding
rounds worth $500 million or more; in 2021, that number
grew to 15. Companies attracted significantly higher
investment in 2021, as the average private investment
deal size in 2021 was 81.1% higher than in 2020.

In 2021, global private
investment in AI totaled
around $93.5 billion,
which is more thandouble the total private
investment in 2020.

PRIVATE INVESTMENT in AI, 2013–21
Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report

Total Investment (in bIllions of U.S. Dollars)

100

93.54

80

60

40

20

0
2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 4.2.2

4 The largest private investments have been Databricks (United States), Beijing Horizon Robotics Technology (China), Oxbotica Limited (United Kingdom), and Celonis (Germany).

Table of Contents

Chapter 4 Preview

152

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.2 Investment

NUMBER of NEWLY FUNDED AI COMPANIES in the WORLD, 2013–21
Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report

1200

Number of Companies

1000

800

746

600

400

200

0
2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 4.2.3

Funding Size

2020

2021

Total

Over $1 billion

3

5

8

$500 million – $1 billion

1

10

11

$100 million – $500 million

93

235

328

$ 50 million – $100 million

85

194

279

Under $50 million

2,102

2,120

4,222

Undisclosed

354

395

749

Total

2,638

2,959

5,597
Table 4.2.1

Table of Contents

Chapter 4 Preview

153

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.2 Investment

Regional Comparison by Funding AmountIn 2021, as captured in Figure 4.2.4, the United States
led the world in overall private investment in funded AI
companies—at approximately $52.9 billion—over threetimes the next country on the list, China ( $17.2 billion).
In third place was the United Kingdom ($4.65 billion),
followed by Israel ($2.4 billion) and Germany ($1.98
billion). Figure 4.2.5 shows that when combining total
private investment from 2013 to 2021, the same ranking
applies: U.S. investment totaled $149 billion and Chinese
investment totaled $61.9 billion, followed by the United

Kingdom ($10.8 billion), India ($10.77 billion), and Israel($6.1 billion). Notably, U.S. private investment in AI
companies from 2013–2021 was more than double thetotal in China, which itself was about six times the total
investment from the United Kingdom in the same period.
Broken out by geographic area, as shown in Figure
4.2.6, the United States, China, and the European Union
all grew their investments from 2020 to 2021, with theUnited States leading China and the European Union by
3.1 and 8.2 times the investment amount, respectively.

PRIVATE INVESTMENT in AI by GEOGRAPHIC AREA, 2021
Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report

52.88

United States
17.21

China
4.65

United Kingdom
2.41

IsraelGermany

1.98

Canada

1.87

France

1.55

India

1.35

Australia

1.25South Korea

1.10

Singapore

0.93

Spain

0.89

United Arab Emirates

0.82

Hong Kong

0.63

Portugal 0.52
0

10

20

30

40

50

Total Investment (in billions of U.S. Dollars)
Figure 4.2.4

Table of Contents

Chapter 4 Preview

154

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.2 Investment

PRIVATE INVESTMENT in AI by GEOGRAPHIC AREA, 2013–21
Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report

149.0

United States
61.9

China
10.8

United Kingdom

10.8

India
Israel6.1

Canada5.7

Germany

4.4

France3.9
2.3

Singapore
Japan

2.2

Australia

1.8South Korea

1.8

Spain 1.3
Hong Kong 1.2
Netherlands 1.2
0

20

40

60

80

100

120

140

160

Total Investment (in billions of U.S. Dollars)
Figure 4.2.5

PRIVATE INVESTMENT in AI by GEOGRAPHIC AREA, 2013–21
Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report

52.87, United States

Total Investment (in billions of U.S. Dollars)

50

40

30

20

17.21, China

106.42, European Union

0
2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 4.2.6

Table of Contents

Chapter 4 Preview

155

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.2 Investment

Regional Comparison by Newly Funded
AI Companies

shows a similar trend (Figure 4.2.8).

This section breaks down the investment data by the
number of newly funded AI companies in each region. For
2021, Figure 4.2.7 shows that the United States led with299 companies, followed by China with 119, the United
Kingdom with 49, and Israel with 28. The gaps between
each are significant. Aggregated data from 2013 to 2021

However, the number of newly funded AI companies has
declined in both the United States and China since 2018and 2019 (Figure 4.2.9). Despite that downward trend,
the United States still leads in the number of newly
funded companies, with 299 funded in 2021, followed by
China (119) and the European Union (96).

NUMBER of NEWLY FUNDED AI COMPANIES by GEOGRAPHIC AREA, 2021
Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report

299

United States
119

China
49

United Kingdom
Israel28

France27

Germany

25

India

23

Canada22

South Korea

19

Netherlands

16

Australia

12

Japan

12

Switzerland

12
10Singapore
0

50

100

150

200

250

300

Number of Companies
Figure 4.2.7

Table of Contents

Chapter 4 Preview

156

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.2 Investment

NUMBER of NEWLY FUNDED AI COMPANIES by GEOGRAPHIC AREA, 2013–21 (SUM)
Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report

3,234

United States
940

China
427

United Kingdom
264

IsraelFrance242

Canada

239
172

Japan
India

169

Germany

154

Singapore

99

Australia

77

South Korea

74

Switzerland

65

Spain

56

0

500

1,000

1,500

2,000

2,500

3,000

3,500

Number of Companies
Figure 4.2.8

NUMBER of NEWLY FUNDED AI COMPANIES by GEOGRAPHIC AREA, 2013–21
Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report

500

Number of Companies

400

300

299, United States

200

119, China
106, European Union

100

0
2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 4.2.9

Table of Contents

Chapter 4 Preview

157

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.2 Investment

Focus Area Analysis
Private AI investment also varies by focus area. According
to Figure 4.2.10, the greatest private investment in AI in
2021 was in data management, processing, and cloud(around $12.2 billion). Notably, this was 2.6 times theinvestment in 2020 (around $4.69 billion) as two of thefour largest private investments made in 2021 are data
management companies. In second place was private
investment in medical and healthcare ($11.29 billion),
followed by fintech ($10.26 billion), AV ($8.09 billion), andsemiconductors ($6.0 billion).

Aggregated data in Figure 4.2.11 shows that in the last
five years, the medical and healthcare category received
the largest private investment globally ($28.9 billion);
followed by data management, processing, and cloud($26.9 billion); fintech ($24.9 billion); and retail ($21.95
billion). Moreover, Figure 4.2.12 shows the overall trend
in private investment by industries from 2017–2021 andreveals a steady increase in AV, cybersecurity and dataprotection, fitness and wellness, medical and healthcare,and semiconductor industries.

PRIVATE INVESTMENT in AI by FOCUS AREA, 2017–21 (SUM)

PRIVATE INVESTMENT in AI by FOCUS AREA, 2020 vs. 2021

Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report

Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report

Data Management, Processing, Cloud

Medical and Healthcare

Medical and Healthcare

Data Management, Processing, Cloud

FintechFintechAV

RetailSemiconductor

2021
2020

Industrial Automation, Network

28.92
26.91
24.92
21.95

Industrial Automation, Network

19.03

AV

18.60

RetailSemiconductor

11.91

Fitness and Wellness

Fitness and Wellness

10.75

NLP, Customer Support

NLP, Customer Support

8.45

Energy, Oil, and Gas

Ed Tech

8.16

Cybersecurity, Data Protection

Energy, Oil, and Gas

7.82

DronesFacial Recognition

7.49

Marketing, Digital Ads

Cybersecurity, Data Protection

7.30

HR TechMarketing, Digital Ads

7.20

Facial Recognition

AR/VR

4.46

Insurtech

HR Tech4.27

AgritechDrones

3.91

Sales Enablement

Sales Enablement

3.54

AR/VR

Insurtech

3.29

Ed TechAgritech

3.13

Geospatial

Legal Tech

2.62

Legal Tech

Geospatial

1.82
1.33

Entertainment

Music, Video Content

Music, Video Content

Entertainment

1.31

VC

VC

1.20

0

2

4

6

8

10

12

Total Investment (in billions of U.S. Dollars)

Figure 4.2.10

Table of Contents

Chapter 4 Preview

0

5

10

15

20

25

30

Total Investment (in billions of U.S. Dollars)
Figure 4.2.11

158

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.2 Investment

PRIVATE INVESTMENT in AI by FOCUS AREA, 2017–21
Source: NetBase Quid, 2021 | Chart: 2022 AI Index Report
Agritech

AR/VR

AV

Cybersecurity, Data Protection

1.43

Total Investment (in billions of U.S. Dollars)

Data Management, Processing, Cloud

8.09

10

12.17

2.65

1.26

0
Ed TechDronesEnergy, Oil, and Gas

Entertainment

Facial Recognition

10
2.78

2.48

Fintech10.26

10Fitness and Wellness

1.87

0.71

1.24

0

Geospatial

HR TechIndustrial Automation, Network
5.85

5.56
2.18

0.93
0
Insurtech

Legal Tech

Marketing, Digital Ads

Medical and Healthcare

11.29

Music, Video Content

10
1.77

2.31

0.91

0
NLP, Customer Support

RetailSales Enablement

10

0.47
Semiconductor

5.74

4.01

VC
6.00

1.28

0.29

0
2018

2020

2018

2020

2018

2020

2018

2020

2018

2020

Figure 4.2.12

Table of Contents

Chapter 4 Preview

159

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.3 Corporate Activity

4.3 CORPORATE ACTIVITYI N D U S T RY A D O P T I O NGlobal Adoption of AI

This section on corporate AI activity draws on McKinsey’s“The State of AI in 2021” report from December 2021. The
report based its conclusions on a global online surveyof 1,843 participants conducted earlier in 2021. Survey
respondents came from a range of industries, companies,functional specialties, tenures, and regions of the world—and each provided answers to questions about the stateof artificial intelligence today.

Figure 4.3.1 shows AI adoption by organizations globally,broken out by geographic area. In 2021, India led with
65% adoption, followed by “Developed Asia-Pacific”(64%), “Developing markets (incl. China, MENA)” (57%),
and North America (55%). The average adoption rate
across all geographies was 56%, up 6% from 2020.
Notably, “Developing markets (incl. China, MENA)”
registered a 21% increase from 2020, and India had an
8% increase from 2020.

AI ADOPTION by ORGANIZATIONS in the WORLD, 2020-21
Source: McKinsey & Company, 2021 | Chart: 2022 AI Index Report56%

All geographies
50%

64%

Developed
Asia-Paci!c

61%
57%

Developing markets(incl. China, MENA)

45%
51%

Europe

2021
2020

47%

65%

India
57%
47%

Latin America
41%

55%

North America
51%

0%

10%

20%

30%

40%

50%

60%

70%

% of Respondents
Figure 4.3.1

Table of Contents

Chapter 4 Preview

160Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.3 Corporate Activity

AI Adoption by Industry and Function
Figure 4.3.2 shows AI adoption by industry and function
in 2021. The greatest adoption was in product and/or
service development for high tech/telecommunications(45%), followed by service operations for financial
services (40%), service operations for high tech/telecommunications (34%), and risk function for
financial services (32%).

AI ADOPTION by INDUSTRY and FUNCTION, 2021

Industry

Source: McKinsey & Company, 2021 | Chart: 2022 AI Index Report
Human Resources

Manufacturing

Marketing and Sales

Product and/or
Service Development

Risk

Service Operations

Strategy andCorporate FinanceSupply-chain
Management

All Industries

9%

12%

20%

23%

13%

25%

9%

13%

Automotive andAssembly

11%

26%

20%

15%

4%

18%

6%

17%

Business, Legal, andProfessional Services

14%

8%

28%

15%

13%

26%

8%

13%

ConsumerGoods/Retail

2%

18%

22%

17%

1%

15%

4%

18%

Financial Services

10%

4%

24%

20%

32%

40%

13%

8%

Healthcare
Systems/Pharma andMedical Products

9%

11%

14%

29%

13%

17%

12%

9%

High Tech/Telecom

12%

11%

28%

45%

16%

34%

10%

16%

% of Respondents (Function)
Figure 4.3.2

Table of Contents

Chapter 4 Preview

161Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.3 Corporate Activity

Type of AI Capabilities Adopted

telecommunications industry (34%), followed by robotic
process automation for both the financial services andautomotive and assembly industry (33%) and natural
language text understanding for financial services (32%).

With respect to the type of AI capabilities embedded
in standard business processes in 2021, as indicatedby Figure 4.3.3, the highest rate of embedding was innatural language text understanding for the high tech/

AI CAPABILITIES EMBEDDED in STANDARD BUSINESS PROCESSES, 2021

Industry

Source: McKinsey & Company, 2021 | Chart: 2022 AI Index Report
PhysicalRobotics

Recommender
Systems

24%

12%

17%

16%

26%

11%

12%

24%

12%

5%

23%

18%

19%

13%

22%

17%

11%

13%

14%

4%

11%

16%

12%

18%

32%

25%

12%

19%

10%

8%

22%

6%

17%

17%

18%

FacialKnowledge
NL
Regonition
GraphsGeneration

NL Speech
NL Text UnUnderstanding
derstanding

Computer
VisionDeep
Learning

All Industries

23%

19%

11%

17%

12%

14%

Automotive andAssembly

15%

14%

9%

16%

3%

Business, Legal, andProfessional Services

29%

24%

15%

20%

ConsumerGoods/Retail

23%

12%

14%

Financial Services

17%

16%

Healthcare
Systems/Pharma andMedical Products

30%High Tech/Telecom

28%

ReinforceRobotic
ment
Process Simulations
Learning Automation

Transfer
Learning

VirtualAgents

17%

12%

23%

33%

27%

6%

12%

27%

31%

18%

21%

19%

8%

8%

16%

9%

1%

15%

4%

13%

16%

33%

12%

12%

28%

26%

28%

22%

13%

28%

22%

19%

31%

34%

5%

19%

15%

23%

14%

11%

25%

% of Respondents (AI Capability)
Figure 4.3.3

Table of Contents

Chapter 4 Preview

162Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.3 Corporate Activity

Consideration and Mitigation of Risks
From Adopting AI
The risk from adopting AI that survey respondents
identified as most relevant in 2021 was cybersecurity(55% of respondents), followed by regulatory compliance(48%), explainability (41%), and personal/individual
privacy (41%) (Figure 4.3.4). Fewer organizations foundAI risks from cybersecurity to be relevant in 2021 thanin 2020, declining from just over 60% of respondents
expressing concern in 2020 to 55% in 2021. Concern over
AI regulatory compliance risks, meanwhile, remained
virtually unchanged from 2020.

Figure 4.3.5 shows risks from AI that organizations aretaking steps to mitigate. Cybersecurity was the mostfrequent response (47% of respondents), followed byregulatory compliance (36%), personal/individual privacy(28%), and explainability (27%). It is worth noting thegaps between risks that organizations find relevant andrisks that organizations take steps to mitigate—a gap of 10
percentage points with equity and fairness (29% to 19%),
12 percentage points with regulatory compliance (48%
to 36%), 13 percentage points with personal/individual
privacy (41% to 28%), and 14 percentage points with
explainability (41% to 27%).

RISKS from ADOPTING AI that ORGANIZATIONS CONSIDER RELEVANT, 2019–21
Source: McKinsey & Company, 2021 | Chart: 2022 AI Index Report

60%
55%, Cybersecurity

50%48%, Regulatory Compliance% of Respondents

41%, Explainability

40%

41%, Personal/Individual Privacy
35%, Organizational Reputation

30%

29%, Equity and Fairness
26%, Workforce/Labor Displacement
21%, Physical Safety

20%

14%, National Security

10%9%, Political Stability

0%
2019

2020

2021
Figure 4.3.4

Table of Contents

Chapter 4 Preview

163

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.3 Corporate Activity

RISKS from ADOPTING AI that ORGANIZATIONS TAKE STEPS to MITIGATE, 2019–21
Source: McKinsey & Company, 2021 | Chart: 2022 AI Index Report

50%
47%, Cybersecurity40%% of Respondents

36%, Regulatory Compliance30%

28%, Personal/Individual Privacy
27%, Explainability
22%, Organizational Reputation

20%

19%, Equity and Fairness
16%, Physical Safety
15%, Workforce/Labor Displacement

10%8%, National Security
4%, Political Stability

0%
2019

2020

2021
Figure 4.3.5

Table of Contents

Chapter 4 Preview

164Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.4 AI Education

The following section draws on data from the annual Computing Research Association (CRA) Taulbee Survey. For the latest survey
featured in this section, CRA collected data in Fall 2020 by reaching out to over 200 PhD-granting departments in the United States andCanada. Results are published in May 2021. The CRA survey documents trends in student enrollment, degree production, employmentof graduates, and faculty salaries in academic units in the United States and Canada that grant doctoral degrees in computer science(CS), computer engineering (CE), or information (I). Academic units include departments of CS and CE or, in some cases, colleges orschools of information or computing.

4.4 AI EDUCATIONC S U N D E R G R A D UAT E G R A D UAT E S
I N N O R T H A M E R I CA
In North America, most AI-related courses are offeredas part of the CS curriculum at the undergraduate level.

The number of new CS undergraduate graduates at
doctoral institutions in North America has grown 3.5
times from 2010 to 2020 (Figure 4.4.1). More than 31,000
undergraduates completed CS degrees in 2020—an
11.60% increase from the number in 2019.

NUMBER of NEW CS UNDERGRADUATE GRADUATES at DOCTORAL INSTITUTIONS in NORTH AMERICA, 2010–20
Number of New CS Undergraduate Graduates (in thousands)

Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report

31.84

30

25

20

15

10

5

0
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020
Figure 4.4.1

Table of Contents

Chapter 4 Preview

165Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.4 AI Education

N E W C S P H D S I N N O R T H A M E R I CA

In 2020, 1 in every 5
CS students who
graduated with PhD
degrees specialized
in AI/ML, the mostpopular specialty
in the past decade.

The following sections show the trend of CS PhD graduates
in North America with a focus on those with AI-related
specialties.5 The CRA survey includes 20 specialties in total,
two of which are directly related to the field of AI: artificial
intelligence/machine learning (AI/ML) and robotics/vision.

New CS PhDs by Specialty
In 2020, 1 in every 5 CS students who graduated with
PhD degrees specialized in AI/ML, the most popularspecialty in the past decade (Figure 4.4.2). It is also thespeciality that exhibits the most significant growth from2010 to 2021, relative to 18 other specializations (Figure
4.4.3). Robotics/vision is also among the most popular
CS specialties of PhD graduates in 2020, registering a
1.4 percentage point change in the share of total new CS
PhDs in the past 11 years.

NEW CS PHDS (% of TOTAL) in the UNITED STATES by SPECIALITY, 2020
Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report

21.00%

Arti!cial Intelligence/Machine Learning
7.30%

Software Engineering
Security/Information Assurance

7.10%

Theory and Algorithms

7.00%

Databases/Information Retrieval

6.50%

Robotics/Vision

6.30%

Networks

6.10%

Other

5.80%
5.30%

Human-Computer Interaction

4.90%

Graphics/Visualization

4.30%

Operating Systems

3.40%

Programming Languages/Compilers

3.30%

Informatics: Biomedical/Other Science

2.90%

High Performance Computing
Information Systems

2.70%

Hardware/Architecture

2.40%
2.00%

Social Computing/Social Informatics/CSCW
Computing Education

0.70%

Information Science

0.50%

Scienti!c/Numerical Computing

0.50%

0%

5%

10%

15%

20%

% of New CS PhDs
Figure 4.4.2

5 New CS PhDs in this section include PhD graduates from academic units (departments, colleges, or schools within universities) of computer science in the United States.

Table of Contents

Chapter 4 Preview

166

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.4 AI Education

PERCENTAGE POINT CHANGE in NEW CS PHDS in the UNITED STATES by SPECIALTY, 2010–20
Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report

6.80%

Arti!cial Intelligence/Machine Learning
2.50%

Human-Computer Interaction

2.10%

Security/Information Assurance

1.50%

Information Systems

1.40%

Robotics/Vision
0.60%

High Performance Computing
Databases/Information Retrieval

0.10%

Information Science

0.00%-0.10%

Social Computing/Social Informatics/CSCW
Other-0.50%

Theory and Algorithms-0.60%-0.60%

Operating Systems-1.20%

Software Engineering
Graphics/Visualization-1.50%

Informatics: Biomedical/Other Science-1.60%-1.70%

Hardware/Architecture-1.90%

Scienti!c/Numerical Computing-2.00%

Programming Languages/Compilers-4.10%

Networks

-4%

-2%

0%

2%

4%

6%

8%

Percentage Point Change in New CS PhDs

Figure 4.4.3

New CS PhDs with AI/ML and Robotics/Vision Specialties
Between 2010 and 2020, the number of CS PhD graduates with AI/ML and robotics/vision specialities grew by 72.05%
and 50.91%, respectively. The slight decrease in the total number for both specialties from 2019 to 2020 may be due tothe impact of the COVID-19 pandemic.
NEW CS PHDS with AI/ML and ROBOTICS/VISION SPECIALTY
in the UNITED STATES, 2010–20

NEW CS PHDS (% of TOTAL) with AI/ML and
ROBOTICS/VISION SPECIALTY in the UNITED STATES, 2010–20

Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report

Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report350

21.00%

286

200
171
138

159

144

266
218

138

100

15%

10%

6.30%

5%
68

65

74

76

72

75

66

91

83

Arti!cial Intelligence/Machine Learning
Robotics/Vision

Chapter 4 Preview

2018

2017

2016

2015

2014

2013

2012

2011

Arti!cial Intelligence/Machine Learning
Robotics/Vision
Figure 4.4.4a

Table of Contents

2010

2020

2019

2018

2017

2016

2015

2014

0%
2013

46

2011

55

0

2012

50

2019

161

277

2009

150

233

2010

Number of Graduates

250

New CS PhDs (% of Total)

20%
300

Figure 4.4.4b

167

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.4 AI Education

N E W A I P H D S E M P LOY M E N T I N
N O R T H A M E R I CA
Where do new AI PhDs choose to work following
graduation? This section analyzes the employment
trends of new AI PhDs across North America in academia,
industry, and government.6

Academia vs. Industry vs. Government
In 2020, the share of new AI PhD graduates in North

America who chose to work in the industry dippedslightly, with its share dropping from 65.7% in 2019 to60.2% in 2020, whereas the share of new AI PhDs whowent into academia and government changed little(Figure 4.4.5a and Figure 4.4.5b). Note that the 2020 data
may be impacted by the increasing number of new AI
PhDs who went abroad upon graduation, a number thatgrew from 19 in 2019 to 32 in 2020.

EMPLOYMENT of NEW AI PHDS to ACADEMIA, GOVERNMENT, EMPLOYMENT of NEW AI PHDS (% of TOTAL) to ACADEMIA,or INDUSTRY in NORTH AMERICA, 2010–20
GOVERNMENT, or INDUSTRY in NORTH AMERICA, 2010–20
Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report

Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report

250
60%

65
61
63

150

60
47
51

63

100

43
42
162
134

50

101
76

74

64

85

180
153

50%

40%

30%
24.02%

20%

116

10%

77

1.97%

Government

Industry

Academia
Figure 4.4.5a

Government

2019

2018

2017

2016

2015

2014

2013

2012

2011

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Academia

2010

0%

0

2020

72

New AI PhDs (% of Total)

Number of New AI PhDs

60.24%

73

200

Industry
Figure 4.4.5b

6 New AI PhDs in this section include PhD graduates who specialize in artificial intelligence from academic units (departments, colleges, or schools within universities) of computer science,computer engineering, and information in the United States and Canada.

Table of Contents

Chapter 4 Preview

168

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.4 AI Education

D I V E R S I T Y O F N E W A I P H D S I N N O R T H A M E R I CA
By Gender
Figure 4.4.6 shows that the share of new female AI and CS PhDs in North America remains low and has changed little
from 2010 to 2020.
FEMALE NEW AI and CS PHDS (% of TOTAL NEW AI and CS PHDS) in NORTH AMERICA, 2010–20
Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report

Female New AI PhDs (% of All New AI PhDs)

20.20%, AI

20%
19.90%, CS

15%

10%

5%

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020
Figure 4.4.6

Table of Contents

Chapter 4 Preview

169

Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.4 AI Education

By Race/Ethnicity
According to Figure 4.4.7, among the new AI PhDs from 2010 to2020 who are U.S. residents, the largest percentage has been
non-Hispanic white and Asian—65.2% and 18.8% on average.
By comparison, around 1.5% were Black or African American(non-Hispanic) and 2.9% were Hispanic on average over the

past 11 years. Figure 4.4.8 shows all PhDs awarded in the
United States to U.S. residents across departments of CS, CE,
and information between 2010 and 2020. In the past 11 years,
the share of new white (non-Hispanic) PhDs has changed little,
while the percentage of new Black or African American (nonHispanic) and Hispanic computing PhDs is significantly lower.

NEW U.S. AI RESIDENT PHDS (% of TOTAL) by RACE/ETHNICITY, 2010–20
Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report80%

New U.S. AI Resident PhDs (% of Total)

70%60%
50.86%, White (Non-Hispanic)

50%
40%

30.17%, Asian

30%
20%

8.62%, Unknown
6.90%, Hispanic
1.72%, Black or African American (Non-Hispanic)
0.86%, Native Hawaiian or Paci c Islander
0.86%, Multiracial (Non-Hispanic)
0.00%, Native American or Alaskan Native

10%
0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020
Figure 4.4.7

NEW COMPUTING PHDS, U.S. RESIDENT (% of TOTAL) by RACE/ETHNICITY, 2010–20
Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report

New Computing PhDs, U.S. Resident (% of Total)

70%

60%

57.50%, White (Non-Hispanic)

50%

40%

30%
24.80%, Asian20%
7.80%, Unknown
4.20%, Hispanic
3.60%, Black or African American (Non-Hispanic)
1.80%, Multiracial (Non-Hispanic)
0.10%, Native American or Alaskan Native

10%

0%

0.10%, Native Hawaiian or Paci c Islander

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020
Figure 4.4.8

Table of Contents

Chapter 4 Preview

170Artificial IntelligenceIndex Report 2022

CHAPTER 4: THE ECONOMY AND EDUCATION
4.4 AI Education

N E W I N T E R N AT I O N A L A I P H D S I N
N O R T H A M E R I CA
The share of international students among new AI PhDs
in North America in 2020 decreased slightly from 64.3%
in 2019 to 60.5% in 2020 (Figure 4.4.9). For comparison,of all computing PhDs graduating in 2022, 65.1% ofthem were international students. In addition, moreinternational students—14.0% of all new AI PhDs—took
jobs outside the United States in 2020, compared to 8.6%
in 2019 (Figure 4.4.10).

NEW INTERNATIONAL AI PHDS (% of TOTAL NEW AI PHDS) in NORTH AMERICA, 2010–20

New International AI PhDs (% of Total New AI PhDs)

Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report

60%

60.50%

50%

40%

30%

20%

10%

0%
2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020
Figure 4.4.9

INTERNATIONAL NEW AI PHDS (% of TOTAL) in the UNITEDSTATES by LOCATION of EMPLOYMENT, 2020
Source: CRA Taulbee Survey, 2021 | Chart: 2022 AI Index Report

11.80%,
Unknown

14.00%,
Outside United States

74.20%,
United States

Figure 4.4.10

Table of Contents

Chapter 4 Preview

171

Artificial IntelligenceIndex Report 2022

CHAPTER 5:

AI Policy andGovernanceArtificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE

CHAPTER 5:

Chapter Preview
Overview

174

Chapter Highlights

175

5.1 AI AND POLICYMAKING

176

5.2 U.S. PUBLIC INVESTMENT IN AI

188

Global Legislation Records on AI

176

Federal Budget for Nondefense AI R&D

188

177

U.S. Department of Defense
Budget Request

189

Highlight: DOD Top Five
Highest-Funded Programs

190

By Geographic Area
Federal AI Legislation in the
United States
Highlight: A Closer Lookat the Legislation
State-Level AI Legislationin the United States

178
179

U.S. AI Policy Papers
By Topic

186
187

DOD AI R&D Spending by Department 191

180

U.S. Government AI-Related
Contract Spending

192

By State

181

Total Contract Spending

192

Sponsorship by Political Party

182

Mentions of AI in Legislative Records

183

Contract Spending by Department
and Agency

193

AI Mentions in U.S. Congressional
Records

183

Highlight: Largest Contract for Five
Top-Spending Departments in 2021

195

AI Mentions in Global Legislative
Proceedings

184

By Geographic Area185

ACCESS THE PUBLIC DATA

Table of Contents

173

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE

Overview
As AI has become an increasingly ubiquitous topic in the last decade,intergovernmental, national, and regional organizations have worked todevelop policies and strategies around AI governance. These actors are
driven by the understanding that it is imperative to find ways to address
the ethical and societal concerns surrounding AI, while maximizing
its benefits. Active and informed governance of AI technologies has
become a priority for many governments around the world.
This chapter examines the intersection of AI and governance, and takes
a closer look at how governments in different countries, regions, andU.S. states are working to manage AI technologies. It begins by looking
at AI policymaking across the globe and within the United States,
exploring which countries and political actors are most keen to advance
AI legislation, and what kind of AI subtopics, from privacy to ethics, arethe focus of most legislative attention. Then the chapter takes a deep
dive into one of the world’s top public sector investors in AI, the UnitedStates, and studies how much its various government departments havespent on AI in the past five years.

Table of Contents

Chapter 5 Preview

174Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE

CHAPTER HIGHLIGHTS• An AI Index analysis of legislative records on AI in 25 countries shows that the number of bills
containing “artificial intelligence” that were passed into law grew from just 1 in 2016 to 18 in
2021. Spain, the United Kingdom, and the United States passed the highest number of AI-related
bills in 2021, with each adopting three.
• The federal legislative record in the United States shows a sharp increase in the total number of
proposed bills that relate to AI from 2015 to 2021, while the number of bills passed remains low,with only 2% ultimately becoming law.
• State legislators in the United States passed 1 out of every 50 proposed bills that contain AI
provisions in 2021, while the number of such bills proposed grew from 2 in 2012 to 131 in 2021.
• In the United States, the current congressional session (the 117th) is on track to record the greatest
number of AI-related mentions since 2001, with 295 mentions by the end of 2021, half way
through the session, compared to 506 in the previous (116th) session.

Table of Contents

Chapter 5 Preview

175

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

Discussions around AI governance regulation have accelerated over the past decade, resulting in policy proposals across various
legislative bodies. This section first examines AI-related legislation that has either been proposed or passed into law across different
countries and regions, followed by a focused analysis of state-level legislation in the United States. It then takes a closer look atcongressional and parliamentary records on AI across the world and concludes with data on the number of policy papers publishedin the United States.

5.1 AI AND POLICYMAKING
G LO BA L L E G I S L AT I O NRECORDS ON AI

their legislative bodies that contain the words “artificial
intelligence” from 2016 to 2021.

​​
Governments
and legislative bodies across the globe are
increasingly seeking to pass laws to provide funding for
AI development and innovation, while also promoting the
integration of human-centered values. The AI Index has
conducted an analysis of laws passed in 25 countries by

Taken together, the 25 countries analyzed have passed a
total of 55 AI-related bills. Figure 5.2.1 demonstrates that in
the past six years, there has been a sharp increase in termsof the total number of AI-related bills passed into law.1

NUMBER of AI-RELATED BILLS PASSED into LAW in 25 SELECT COUNTRIES, 2016–21
Source: AI Index, 2021 | Chart: 2022 AI Index Report

18

Number of AI-Related Bills

15

10

5

0
2016

2017

2018

2019

2020

2021
Figure 5.1.1

1 Note that the analysis only includes laws passed by national legislative bodies (e.g. congress, parliament) with the keyword “artificial intelligence” in various languages in the title or body of the billtext. See the appendix for the methodology. Countries included: Australia, Belgium, Brazil, Canada, China, Denmark, Finland, France, Germany, India, Ireland, Italy, Japan, the Netherlands, New Zealand,
Norway, Russia, Singapore, South Africa, South Korea, Spain, Sweden, Switzerland, the United Kingdom, and the United States.

Table of Contents

Chapter 5 Preview

176

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

By Geographic Area
Figure 5.1.2a shows the number of laws containing
mentions of AI that were enacted in 2021. Spain, theUnited Kingdom, and the United States led, each passing
three. Figure 5.1.2b shows the total number of legislation
passed in the past six years. The United States dominatedthe list with 13 bills, starting in 2017 with 3 new laws
passed each subsequent year, followed by Russia,
Belgium, Spain, and the United Kingdom.

The United States
dominated the list with 13
bills, starting in 2017 with
3 new laws passed eachsubsequent year, followedby Russia, Belgium, Spain,
and the United Kingdom.

NUMBER of AI-RELATED BILLS PASSED into LAW in SELECT COUNTRIES, 2021
Source: AI Index, 2021 | Chart: 2022 AI Index Report

Spain

3

United Kingdom

3

United States

3

Belgium

2

Russia2

France1Germany

1

Italy

1

Japan

1

South Korea

1

0

1

2

3

4

Number of AI-Related Bills
Figure 5.1.2a

Table of Contents

Chapter 5 Preview

17 7

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

NUMBER of AI-RELATED BILLS PASSED into LAW in SELECT COUNTRIES, 2016–21 (SUM)
Source: AI Index, 2021 | Chart: 2022 AI Index Report

13

United States
Russia6

Belgium

5

Spain

5
5

United Kingdom
France

4

Italy

4

South Korea

4
3

Japan
2

China
Brazil1

Canada1Germany

1

India

1
0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

Number of AI-Related Bills
Figure 5.1.2b

Federal AI Legislation in the United States
A closer look at the federal legislative record in the
United States shows a sharp increase in the total numberof proposed bills that relate to AI (Figure 5.1.3). In 2015,
just one federal bill was proposed, while in 2021, there

were 130. Although this jump is significant, the numberof bills related to AI being passed has not kept pace withthe growing volume of proposed AI-related bills. This gap
was most evident in 2021, when only 2% of all federallevel AI-related bills were ultimately passed into law.

NUMBER of AI-RELATED BILLS in the UNITED STATES, 2015–21 (PROPOSED vs. PASSED)
Source: AI Index, 2021 | Chart: 2022 AI Index Report

130, Proposed120

Number of AI-Related Bills

100

80

60

40

20
3, Passed

0
2015

2016

2017

2018

2019

2020

2021
Figure 5.1.3

Table of Contents

Chapter 5 Preview

178

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

A Closer Look at the Legislation
The following subsection delves into some of the AI-related legislation passed into law since 2016.
Table 5.1.1 demonstrates the wide range of AI-related issues that have piqued policymakers’ interest.
Country

Year Passed

Bill Name

Description

Canada2017

Budget Implementation Act 2017, No. 1

A provision of this act authorized the Canadiangovernment to make a payment of $125 million
to the Canadian Institute for Advanced Researchto support the development of a pan-Canadian
artificial intelligence strategy.

China

2019

Law of the People’s Republic of China
on Basic Medical and Health Care and
the Promotion of Health

A provision of this law aimed to promote theapplication and development of big data andartificial intelligence in the health and medical field
while accelerating the construction of medical andhealthcare information infrastructure, developing
technical standards on the collection, storage,
analysis, and application of medical and health data.

Russia2020

Federal Law of 24 April 2020 No.
123-FZ on the Experiment to Establish
Special Regulation in order to Create
the Necessary Conditions for theDevelopment and Implementation of
Artificial Intelligence Technologies in
the Region of the Russian Federation– Federal City of Moscow andAmending the Articles 6 and 10 of theFederal Law on Personal Data

This law established an experimental framework
for the development and implementation of AI asa five-year experiment to start in Moscow in July
1, 2020, including allowing AI systems to process
anonymized personal data for governmental andcertain commercial business activities.

United Kingdom

2020

Supply and Appropriation (Main
Estimates) Act 2020, c.13

A provision of this act authorized the Office of
Qualifications and Examination Regulation to
explore opportunities for using artificial intelligenceto improve the marking and administration of highstakes qualifications.

United States

2020

IOGAN ACT: Identifying Outputs ofGenerative Adversarial Networks Act

This act directed the National Science Foundation
to support research dedicated to studying theoutputs of generative adversarial networks(deepfakes) and other comparable technologies.

Belgium

2021

Decree on coaching and solutionoriented support for job seekers, N.
327

A provision of this act directs the government
to create an advisory group called the Ethics
Committee, which is responsible for submitting
advice if artificial intelligence tools are to be used
for digitization activities.

France2021

Law N:2021-1485 of November
15, 2021, aimed at reducing theenvironmental footprint of digitaltechnology in FranceThis act sets up a monitoring system to evaluate
environmental impacts of newly emerging digitaltechnologies, in particular, artificial intelligence.

Table 5.1.1

Table of Contents

Chapter 5 Preview

179

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

S TAT E - L E V E L A I L E G I S L AT I O N I N
T H E U N I T E D S TAT E S

a greater proportion of proposed state-level AI bills haveactually passed. In 2021, of the 131 proposed state bills,
26 were passed into law (20%), or 1 out of 5 proposedbills became law. This ratio is significantly higher whencompared to the federal level, where 1 out of every 50
proposed bills became law in 2021.

Growing policy interest in AI can also be seen in thelarge number of AI-related bills recently proposedat the state level in the United States, based on data
provided by Bloomberg Government since 2012.
Bloomberg Government classified a bill as relating to
AI if it contained AI-related keywords such as artificial
intelligence, machine learning, or algorithmic bias.
As is the case on the federal level, there has been asignificant increase in the number of AI bills proposedat the state level in the last decade (Figure 5.1.4).
In 2012, the first two pieces of AI-related legislation
were proposed when New Jersey assembly member
Annette Quijano directed the New Jersey Motor Vehicle
Commission to establish driver’s license endorsements
for autonomous vehicles. In the past 10 years, theincrease has been substantial, from 2 bills in 2012 to 131
in 2021.

A notable difference between
AI-related lawmaking in the
United States on the federal
versus the state level isthat a greater proportion of
proposed state-level AI bills
have actually passed.

A notable difference between AI-related lawmaking in the
United States on the federal versus the state level is that

NUMBER of STATE-LEVEL AI-RELATED BILLS in the UNITED STATES, 2012–21
Source: Bloomberg Government, 2021 | Chart: 2022 AI Index Report140
131
PassedProposed
VetoedNumber of AI-Related Bills

120

26

100
87
13

80

77
10

60
103
40
29
20
10
0

14

2

8

12

2012

2013

2014

74
26

66

9
9

9

25

2015

2016

2017

17
2018

2019

2020

2021
Figure 5.1.4

Table of Contents

Chapter 5 Preview

180Artificial IntelligenceIndex Report 2022

By State
In the United States, AI
lawmaking has been relatively
widespread across all states. Asof 2021, 41 out of 50 states haveproposed at least one AI-relatedbill, but certain states have been
particularly active in generating
AI legislation. Figure 5.1.5 showsthat Massachusetts has proposed
the most AI bills, with 40 since2012, followed by Hawaii (35)and New Jersey (32). Focusing
on just 2021 in Figure 5.1.6,
Massachusetts was the state that
proposed the most AI-related
bills, with 20, followed by Illinois(15) and Alabama (12).

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

NUMBER of STATE-LEVEL PROPOSED AI-RELATED BILLS in the UNITEDSTATES by STATE, 2012–21 (SUM)
Source: Bloomberg Government, 2021 | Chart: 2022 AI Index Report

AK
0

ME
0
VT
8

NH
0
MA
40

WA
14

ID
0

MT
0

ND
0

MN
2

IL
28

WI
0

MI
3

NY
31

RI
5

OR
1

NV
10

WY
0

SD
0

IA
1

IN
1

OH
3

PA
7

NJ
32

CT
2

CA
29

UT
3

CO
2

NE
2

MO
4

KY
1

WV
9

VA
8

MD
8

DE
1

AZ
7

NM
1

KS
1

AR
1

TN
7

NC
6

SC
1

DC
8

OK
2

LA
0

MS
7

AL
21

GA
3

HI
35

TX
17

FL
22

Figure 5.1.5

NUMBER of STATE-LEVEL PROPOSED AI-RELATED BILLS in the UNITEDSTATES by STATE, 2021
Source: Bloomberg Government, 2021 | Chart: 2022 AI Index Report

AK
0

HI
7

ME
0
VT
3

NH
0
MA
20

WA
6

ID
0

MT
0

ND
0

MN
0

IL
15

WI
0

MI
0

NY
8

RI
3

OR
1

NV
1

WY
0

SD
0

IA
1

IN
0

OH
1

PA
3

NJ
4

CT
0

CA
4

UT
2

CO
2

NE
0

MO
1

KY
0

WV
3

VA
1

MD
4

DE
0

AZ
0

NM
0

KS
0

AR
0

TN
2

NC
3

SC
1

DC
6

OK
1

LA
0

MS
3

AL
12

GA
0

TX
6

FL
7

Figure 5.1.6

Table of Contents

Chapter 5 Preview

181

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

Sponsorship by Political Party
State-level AI legislation data reveals that there is apartisan dynamic to AI lawmaking. Figure 5.1.7 plots thenumber of AI-related bills sponsored at the state level by
Democratic and Republican lawmakers. Although there
has been an increase in AI bills proposed by membersof both parties since 2012, in the past four years, thedata suggests Democrats were more likely to sponsor
AI-related legislation. Whereas Democrats sponsoredonly two more AI bills than Republicans in 2018, theysponsored 39 more in 2021.

NUMBER of STATE-LEVEL PROPOSED AI-RELATED BILLS in the UNITED STATES by SPONSOR PARTY, 2012–21
Source: Bloomberg Government, 2021 | Chart: 2022 AI Index Report

79, Democratic

80

Number of AI-Related Bills

70
60
50
40, Republican

40
30
20
10
0
2012

2013

2014

2015

2016

2017

2018

2019

2020

2021
Figure 5.1.7

Table of Contents

Chapter 5 Preview

182

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

M E N T I O N S O F A I I N L E G I S L AT I V ERECORDS
Another barometer of legislative interest in AI is thenumber of mentions of “artificial intelligence” ingovernmental and parliamentary proceedings. This
subsection considers data on mentions of AI bothin U.S. congressional records and the parliamentary
proceedings of other countries based on AI Index andBloomberg Government data.

AI Mentions in U.S. Congressional Records
In the last five years, and especially in 2021, U.S.
congressional sessions have devoted increasing amountsof time to discussions of AI. This section presents datafrom Bloomberg Government concerning mentions of AIrelated keywords in congressional proceedings, broken
down by legislation, congressional committee reports,
and congressional research service reports.
According to Figure 5.1.8, the current congressionalsession (the 117th) is on track (as of the end of 2021)
to record the greatest number of AI-related mentions
since 2001. The most recently completed congressionalsession, the 116th (2019-2020), saw 506 AI mentions,nearly 3.4 times as many mentions as there were during
the 115th session (2017–2018), and 30 times as many as
the 114th session (2015–2016).

MENTIONS of AI in the U.S. CONGRESSIONAL RECORD by LEGISLATIVE SESSION, 2001–21
Source: Bloomberg Government, 2021 | Chart: 2022 AI Index Report

Legislation
Congressional Research Service Reports
Committee Reports

500

506

178

Number of Mentions

400

295

300

245

200

139

149
44

100

129

66

0

4

7

107th
(2001–02)

108th(2003–04)

83

25

17

18

17

12

17

39

109th
(2005–06)

110th(2007–08)

111th(2009–10)112th(2011–12)113th(2013–14)

114th(2015–16)

115th(2017–18)

27

116th
(2019–20)

117th(2021–)
Figure 5.1.8

Table of Contents

Chapter 5 Preview

183

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

AI Mentions in Global Legislative Proceedings
AI mentions in governmental proceedings are on therise not only in the United States but also in many othercountries across the world. The AI Index conducted ananalysis on the minutes or proceedings of legislative

sessions in 25 countries that contain the keyword“artificial intelligence” from 2016 to 2021. Figure 5.1.9
shows that the mentions of AI in legislative proceedingsin 25 select countries grew 7.7 times in the past six years.2

NUMBER of MENTIONS of AI in LEGISLATIVE PROCEEDINGS in 25 SELECT COUNTRIES, 2016–21
Source: AI Index, 2021 | Chart: 2022 AI Index Report

1,323

1,200

Number of Mentions

1,000

800

600

400

200

0
2016

2017

2018

2019

2020

2021
Figure 5.1.9

2 See the appendix for the methodology. Countries included: Australia, Belgium, Brazil, Canada, China, Denmark, Finland, France, Germany, India, Ireland, Italy, Japan, the Netherlands, New Zealand,
Norway, Russia, Singapore, South Africa, South Korea, Spain, Sweden, Switzerland, the United Kingdom, and the United States.

Table of Contents

Chapter 5 Preview

184

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

By Geographic Area
Figure 5.1.10a shows the number of legislative
proceedings containing mentions of AI that wereenacted in 2021. Similar to the trend in the number of
AI mentions in bills passed into laws, Spain, the United

Kingdom, and the United States topped the list. Figure
5.1.2b shows the total number of AI mentions in the past
six years. The United Kingdom dominated the list with
939 mentions, followed by Spain, Japan, the UnitedStates, and Australia.

NUMBER of MENTIONS of AI in LEGISLATIVE PROCEEDINGS in
SELECT COUNTRIES, 2021

NUMBER of MENTIONS of AI in LEGISLATIVE PROCEEDINGS in
SELECT COUNTRIES, 2016–2021 (SUM)

Source: AI Index, 2021 | Chart: 2022 AI Index Report

Source: AI Index, 2021 | Chart: 2022 AI Index Report

Spain

269

United Kingdom

185

United States
Japan

466

Japan

122

422

United States

95

Ireland

559

Spain

132

Australia

939

United Kingdom

410

Australia

76

Singapore

Brazil72

Ireland

Italy

64

Italy

60

Germany

158

France155

Singapore
Belgium

47

Germany

46

282
222
164

Brazil123

25

Belgium

120

Canada22

CanadaNorway20

FinlandFrance111
78

Sweden16

Sweden71

Finland12

Netherlands

67

Russia12

Russia58

South Africa

11

Norway

52

India

34

10

Netherlands
India

7

South Africa

33

New Zealand

6

Denmark

27

South Korea

6

New Zealand

27

Denmark 5

South Korea

21

Switzerland 3
0

Switzerland 15
50

100

150

200

250

300

Figure 5.1.10a

Table of Contents

Chapter 5 Preview

0

200

400

600

800

1000

Number of Mentions

Number of Mentions

Figure 5.1.10b

185

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

U.S. A I P O L I CY PA P E R S
To estimate activities outside national governmentsthat are also informing AI-related rulemaking, theAI Index tracks 55 U.S.-based organizations thatpublished policy papers in the past four years. Thoseorganizations include: think tanks and policy institutes(19); university institutes and research programs (14);civil society organizations, associations, and consortiums(9); industry and consultancy organizations (9); andgovernment agencies (4).3 A policy paper in this section
is defined as a research paper, research report, brief, or
blog post that addresses issues related to AI and makes

specific recommendations to policymakers. Topics of
those papers are divided into primary and secondary
categories: A primary topic is the main focus of the paper,
while a secondary topic is a subtopic of the paper or anissue that was briefly explored.
Figure 5.1.11 plots the total number of U.S.-based AIrelated policy papers that have been published from2018 to 2021, which can proxy the general interest in AI
within the U.S. policymaking space. The total number ofpolicy papers has tripled since 2018, peaking in 2020 with
273, and decreasing slightly in 2021, with 210.

NUMBER of AI-RELATED POLICY PAPERS by U.S.-BASED ORGANIZATIONS, 2018–21
Source: AI Index, 2021 | Chart: 2022 AI Index Report

Number of Policy Papers

250

210

200

150

100

50

0
2018

2019

2020

2021
Figure 5.1.11

3 The complete list of organizations the Index followed can be found in the Appendix.

Table of Contents

Chapter 5 Preview

186

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.1 AI and Policymaking

By Topic

topics, but they were reported on more frequently
as secondary topics. Among the AI topics to receive
comparatively little attention from tracked organizations
are those that relate to energy and the environment,
humanities, physical sciences, and social and behavioral
sciences.

In 2021, the leading primary topics were Privacy, Safety,
and Security; Innovation and Technology; and Ethics(Figure 5.1.12). Certain topics, such as government andpublic administration, education and skills, as well as
democracy, did not feature prominently as primary

NUMBER of AI-RELATED POLICY PAPERS by U.S.-BASED ORGANIZATIONS by TOPIC, 2021
Source: AI Index, 2021 | Chart: 2022 AI Index Report

Primary Topic

Secondary Topic

Privacy, Safety, and Security

62

Innovation and Technology

62

45
63

59

EthicsInt'l A"airs and Int'l Security

51

Industry and Regulation

51

57
45
58

36

Equity and Inclusion
Workforce and Labor

34

Gov't and Public Administration

33

Justice and Law Enforcement

51
30
58

29

17

23

Education and Skills

36

15

Communications and Media

17

7

Health and Biological Sciences

13

4

Social and Behavioral Sciences

1

Democracy 2
Physical Sciences 129
3Energy and Environment 1

3

Humanities 0
0

1

20

40

60

0

20

40

60

Number of Policy Papers
Figure 5.1.12

Table of Contents

Chapter 5 Preview

187

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.2 U.S. Public Investment in AI

This section examines the public AI investment in the United States, based on data from the U.S. government and Bloomberg Government.

5.2 U.S. PUBLIC INVESTMENT IN AI
FEDERAL BUDGET FOR NONDEFENSEA I R& D
In December 2021, the National Science and Technology Council
published a report on the public-sector AI R&D budget acrossdepartments and agencies participating in the Networking andInformation Technology Research and Development (NITRD)
program and the National Artificial Intelligence Initiative. The report
does not include information on classified AI R&D investment by the
defense and intelligence agencies.
In fiscal year (FY) 2021, nondefense U.S. government agencies
allocated a total of $1.53 billion to AI R&D spending, approximately
2.7 times what was spent in FY 2018 (Figure 5.2.1). This figure
is projected to rise 8.8% for FY 2022, with a total of $1.67 billion
requested.4 The increasing amount spent on AI R&D by nondefense
departments indicates the U.S. government’s continued strong
interest in public sector funding for AI research and development
spanning a wide range of federal agencies.

The increasing amountspent on AI R&D by
nondefense departments
indicates the U.S.
government’s continued
strong interest in
public sector funding
for AI research anddevelopment spanninga wide range of federal
agencies.

U.S. FEDERAL BUDGET for NONDEFENSE AI R&D, FY 2018–22
Source: U.S. NITRD Program, 2022 | Chart: 2022 AI Index Report

1.67
1.53

Budget (in billions of U.S. Dollars)

1.50

1.43

1.11

1.00

0.56

0.50

0.00
FY18 (ENACTED)FY19 (ENACTED)

FY20 (ENACTED)FY21 (ENACTED)FY22 (REQUESTED)
Figure 5.2.1

4 See NITRD website for details on AI R&D investment FY 2018-22 with the breakdown of core AI vs AI crosscut. Note that AI crosscutting budget data is not available for FY 2018.

Table of Contents

Chapter 5 Preview

188

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.2 U.S. Public Investment in AI

U.S. D E PA R T M E N T O F D E F E N S E
BUDGET REQUEST
Spending on AI by the U.S. Department of Defense (DOD)
can be proxied by looking at the publicly available
requests made by the DOD for research, development,
test, and evaluation (RDT&E) relating to AI. In FY 2021,

DOD allocated $9.26 billion across 500 AI R&D programs(Figure 5.2.2), a 6.68% increase from the $8.68 billion
spent in 2020. For FY 2022, the department has requested$10 billion so far, which is likely to grow once additionalrequests and congressional appropriations are taken into
account.

U.S. DOD BUDGET for AI-SPECIFIC RESEARCH, DEVELOPMENT, TEST and EVALUATION (RDT&E), FY 2020–22
Source: Bloomberg Government and U.S. Department of Defense, 2021 | Chart: 2022 AI Index Report

10.00

10
9.26

Budget (in billions of U.S. Dollars)

8.68

8

6

4

2
0.93: DOD Reported Budget on AI R&D

0.84: DOD Reported Budget on AI R&D

0.87: DOD Reported Budget on AI R&D

Sum of FY20 Funding

Sum of FY21 Funding

Sum of FY22 Funding

0
Figure 5.2.2

Important data caveat: This chart is indicative of oneof the challenges of quantifying public AI spending.
Bloomberg Government’s analysis that searches AIrelevant keywords in DOD budgets shows that the
department is requesting $10.0 billion for AI-specific R&D
in FY 2022. However, DOD’s own measurement producesa smaller number of $874 million. The discrepancy

Table of Contents

Chapter 5 Preview

may result from the difference in defining AI-related
budget items. For example, a research project that uses
AI for cyber defense may count human, hardware, andoperations-related expenditures within the AI-related
budget request, though the AI software component will
be much smaller.

189

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.2 U.S. Public Investment in AI

DOD Top Five Highest-Funded Programs
This section highlight offers a more qualitative look at some of the AI-related research projects the
DOD prioritizes. Table 5.2.1 presents the five DOD-related AI programs that received the greatest
funding in 2021. In the past year, the DOD was interested in deploying AI for a number of purposes,
from geospatial monitoring to reducing the threat posed by weapons of mass destruction.

Program Name

Department

Funds Received(in millions)

1 Rapid Capability
Development and Maturation

Army

257

Fund the development, engineering, acquisition,and operation of various AI-related technologicalprototypes that could be used for military purposes.

2 Counter Weapons ofMass Destruction Advanced
Technology Development

Defense Threat
Reduction
Agency254

Develop technologies that could “deny, defeat anddisrupt” weapons of mass destruction (WMD).

3 Algorithmic Warfare
Cross-Functional Teams –Software Pilot Program

Office of theSecretary of
Defense230

Accelerate the integration of AI technologies in DOD
systems to “improve warfighting speed and lethality.”

4 Joint Artificial Intelligence
CenterDefenseInformation
Systems
Agency137

Develop, test, prototype, and demonstrate various AI
and machine learning capabilities with the intentionof integrating these capabilities across numerous
domains which include “supply chain, personal
recovery, infrastructure assessment, geospatial
monitoring during disaster and cyber sense making.”

5 High Performance
Computing Modernization
ProgramArmy

96

Investigate, demonstrate, and mature both generaland special-purpose supercomputing environmentsthat are used to satisfy wide-ranging DOD priorities.

PurposeTable 5.2.1

Table of Contents

Chapter 5 Preview

190

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.2 U.S. Public Investment in AI

DOD AI R&D Spending by Department
DOD spending on AI R&D can also be broken down on
a subdepartmental level, which reveals how individual
defense agencies—the Army and the Navy, for instance—compare in their AI spending (Figure 5.2.3). The U.S.
Navy was the top-spending DOD agency in FY 2021 andis poised to maintain that position in 2022. They haverequested a total of $1.86 billion in FY 2022 for AI-related
projects, followed by the Army ($1.77 billion), the Officeof the Secretary of Defense ($1.1 billion) and the Air Force($883 million).

U.S. DOD BUDGET for AI-SPECIFIC RESEARCH, DEVELOPMENT, TEST and EVALUATION (RDT&E) by
DEPARTMENT, FY 2020–22
Source: Bloomberg Government, 2021 | Chart: 2022 AI Index Report

Air Force

Army

DARPA

DISA

Navy

OSD

Other

10

Budget (in billions of U.S. Dollars)

1.75
1.64

1.18

8

1.77
1.52

1.63

1.12

1.19

4

1.72

1.93

2

1.92

1.54

1.00

1.16

FY20 (ENACTED)FY21 (ENACTED)

6

1.13

1.86

1.71

0

1.57

FY22 (REQUESTED)
Figure 5.2.3

Table of Contents

Chapter 5 Preview

191

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.2 U.S. Public Investment in AI

U.S. G OV E R N M E N T A I - R E L AT E D
C O N T R ACT S P E N D I N G

of more than 100 AI-specific keywords in their titles or
descriptions.5

Public investment in AI can also be measured by federal
government spending on AI-related contracts. U.S.
government agencies often award contracts to privatecompanies for the supply of various goods and services
that typically occupy the largest share of an agency’s
budget. Bloomberg Government built a model to classify
whether a U.S. government contract was AI-related byadding up all contracting transactions that contain a set

Total Contract Spending
In 2021, federal departments and agencies spent a total of$1.79 billion on AI-related contracts. Although this amountis nearly double what was spent on AI-related contracts in2018 (roughly $920 million), it represents a slight decrease
from the amount spent on AI-related contracts in 2020,
which peaked at $1.97 billion (Figure 5.2.4).

U.S. GOVERNMENT TOTAL CONTRACT SPENDING on AI, FY 2000–21
Source: Bloomberg Government, 2021 | Chart: 2022 AI Index Report

Contract Spending (in billions of U.S. Dollars)

2.0

1.79

1.5

1.0

0.5

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

0.0

Figure 5.2.4

5 Note that contractors may add a number of keywords into their applications during the procurement process, so some of the projects included may have a relatively small AI component relative toother parts of technology.

Table of Contents

Chapter 5 Preview

192

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.2 U.S. Public Investment in AI

Contract Spending by Department and Agency

Aggregate spending on AI contracts in the last four years
tells a similar story. Since 2018, the DOD has spent $5.20
billion on AI contracts, approximately seven times the next
highest spender, NASA ($1.41 billion). In fact, since 2018,the DOD has spent twice as much on AI-related contracts
as all other government agencies combined. Following the
DOD and NASA are the Department of Health and Human
Services ($700 million), the Department of Homeland
Security ($362 million), and Department of the Treasury($156 million).

Figures 5.2.5 and 5.2.6 report AI-related contract spendingby the top 10 federal agencies in 2021 and from 2000 to
2021, respectively. The DOD outspent the rest of the U.S.
government on both charts by a significant margin. In
2021, it spent $1.14 billion on AI-related contracts, roughly
five times what was spent by the next highest department,
the Department of Health and Human Services ($234
million).

TOP CONTRACT SPENDING on AI by U.S. GOVERNMENT DEPARTMENT and AGENCY, 2021
Source: Bloomberg Government, 2021 | Chart: 2022 AI Index Report

1,138

Department of Defense (DOD)
234

Department of Health and Human Services (HHS)
159

National Aeronautics and Space Administration (NASA)81

Department of Homeland Security (DHS)
Department of Commerce (DOC)

49

Department of the Treasury (TREAS)

38

Department of Veterans A"airs (VA)

25

Department of Transportation (DOT) 12
Securities and Exchange Commission (SEC) 12
Department of Agriculture (USDA) 12
Department of Energy (DOE) 8
Agency for International Development (USAID) 6
Department of Justice (DOJ) 4
Department of State (DOS) 3
National Science Foundation (NSF) 2
0

200

400

600

800

1000

1200

Contract Spending (in millions of U.S. Dollars)
Figure 5.2.5

Table of Contents

Chapter 5 Preview

193

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.2 U.S. Public Investment in AI

TOP CONTRACT SPENDING on AI by U.S. GOVERNMENT DEPARTMENT and AGENCY, 2000–21 (SUM)
Source: Bloomberg Government, 2021 | Chart: 2022 AI Index Report

5.20

Department of Defense (DOD)
1.41

National Aeronautics and Space Administration (NASA)
0.70

Department of Health and Human Services (HHS)

0.45

Department of Homeland Security (DHS)

0.32

Department of the Treasury (TREAS)
Department of Veterans A!airs (VA)

0.15

Department of Commerce (DOC)

0.15

Department of Agriculture (USDA) 0.07
Department of Energy (DOE) 0.06
Securities and Exchange Commission (SEC) 0.06
General Services Administration (GSA) 0.06
Department of State (DOS) 0.06
Social Security Administration (SSA) 0.05
Department of Transportation (DOT) 0.05
0

1

2

3

4

5

Contract Spending (in billions of U.S. Dollars)
Figure 5.2.6

Table of Contents

Chapter 5 Preview

194

Artificial IntelligenceIndex Report 2022

CHAPTER 5: AI POLICY AND GOVERNANCE
5.2 U.S. Public Investment in AI

Largest Contract for Five Top-Spending
Departments in 2021
To paint a better picture of how different U.S. government departments use AI, Table 5.2.2 shows themost expensive AI-related contract that the five highest AI-related-spending departments signed in
2021. Last year, the U.S. government invested in AI to build autonomous vehicle prototypes, develop an
AI imaging system that could assist with burn classification, and create robots capable of higher-level
lunar navigation.

Contract Name

Department

Amount(in millions)Prototype Services in the Objective
Areas of Automotive Cybersecurity,
Vehicle Safety Technologies, Vehicle
Light Weighting, Autonomous Vehicles
and Intelligent Systems, Connected
Vehicles, and Advanced Energy Storage
Technologies

DOD

70

To acquire prototypes in the domain of
automotive cybersecurity, vehicle safety
technologies, and autonomous vehicles andintelligent systems.

Biomedical Advanced Research andDevelopment Authority (BARDA)

HHS

20

To develop optical imaging devices andmachine learning algorithms to assist
in classifying and healing wounds andconventional burns.

Commercial Lunar Payload Services

NASA

14

To develop lunar robots capable of navigating
the moon’s south pole to acquire lunar
resources and engage in lunar-based scientificactivities.

SBIR-Autonomous Surveillance
Towers-Delivery Order

DHS

37

To construct towers capable of autonomous
surveillance.

Schedule 70: Information Technology

DOC

13

To develop a prototype using AI technologythat can improve patent search.

PurposeTable 5.2.2

Table of Contents

Chapter 5 Preview

195

Artificial IntelligenceIndex Report 2022

AppendixArtificial IntelligenceIndex Report 2022

Appendix
CHAPTER 1

Research & Development

198

CHAPTER 2

Technical Performance

200

CHAPTER 3

Technical AI Ethics

212

CHAPTER 4

The Economy and Education

216

CHAPTER 5

AI Policy and Governance

222

197

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 1: Research & Development

CHAPTER 1: RESEARCH & DEVELOPMENTCENTER FOR SECURITY AND
E M E R G I N G T E C H N O LO GY,
G E O R G E TOW N U N I V E R S I T Y
Prepared by Sara Abdulla and James Dunham
The Center for Security and Emerging Technology (CSET)
is a policy research organization within Georgetown
University’s Walsh School of Foreign Service that produces
data-driven research at the intersection of security andtechnology, providing nonpartisan analysis to the policy
community.

Publications from CSET Merged Corpus of
Scholarly Literature
SourceCSET’s merged corpus of scholarly literature combines
distinct publications from Digital Science’s Dimensions,
Clarivate’s Web of Science, Microsoft Academic Graph,
China National Knowledge Infrastructure, arXiv, andPapers With Code.1
Methodology
To create the merged corpus, CSET deduplicated acrossthe listed sources using publication metadata, and then
combined the metadata for linked publications. To
identify AI publications, CSET used an English-language
subset of this corpus: publications since 2010 that appear
AI-relevant.2 CSET researchers developed a classifier
for identifying AI-related publications by leveraging thearXiv repository, where authors and editors tag papers by
subject.
To provide a publication’s field of study, CSET matches
each publication in the analytic corpus with predictions
from Microsoft Academic Graph (MAG)’s field-of-study
model, which yields hierarchical labels describing the

published research field(s) of study and correspondingscores.3 CSET researchers identified the most common
fields of study in their corpus of AI-relevant publications
since 2010 and recorded publications in all other fields as“Other AI.” English-language AI-relevant publications werethen tallied by their top-scoring field and publication year.
CSET also provided year-by-year citations for AI-relevant
work associated with each country. A publication isassociated with a country if it has at least one author whoseorganizational affiliation(s) is located in that country.
Citation counts aren’t available for all publications; those
without counts weren’t included in the citation analysis.
Over 70% of English-language AI papers published between
2010 and 2020 have citation data available.
CSET counted cross-country collaborations as distinct
pairs of countries across authors for each publication.
Collaborations are only counted once: For example, if apublication has two authors from the United States andtwo authors from China, it is counted as a single UnitedStates-China collaboration.
Additionally, publication counts by year and by
publication type (e.g., academic journal articles,
conference papers) were provided where available.
These publication types were disaggregated by affiliation
country as described above.
CSET also provided publication affiliation sector(s)where, as in the country attribution analysis, sectors wereassociated with publications through authors’ affiliations.
Not all affiliations were characterized in terms of sectors;
CSET researchers relied primarily on GRID from Digital
Science for this purpose, and not all organizations canbe found in or linked to GRID.4 Where the affiliation

1 All CNKI content is furnished for CSET by East View Information Services, Minneapolis, MN, USA.
2 For more information, see James Dunham, Jennifer Melot, and Dewey Murdick, “Identifying the Development and Application of Artificial Intelligence in Scientific Text,” arXiv [cs.DL], May 28, 2020,
https://arxiv.org/abs/2002.07143.
3 These scores are based on cosine similarities between field-of-study and paper embeddings. See Zhihong Shen, Hao Ma, and Kuansan Wang, “A Web-Scale System for Scientific Knowledge
Exploration,” arXiv [cs.CL], May 30, 2018, https://arxiv.org/abs/1805.12216.
4 See https://www.grid.ac/ for more information about the GRID dataset from Digital Science.

Table of Contents

Appendix198

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 1: Research & Development

sector is available, papers were counted toward these
sectors, by year. Cross-sector collaborations on academic
publications were calculated using the same method as in
the cross-country collaborations analysis.

iperov/DeepFaceLab, Microsoft/CNTK, opencv/opencv,
pytorch/pytorch, scikit-learn/scikit-learn, scutan90/
DeepLearning-500-questions, tensorflow/tensorflow,
Theano/Theano, Torch/Torch7.

Patents from CSET’s AI Patents Dataset

NuanceThe GitHub Archive currently does not provide a way
to count when users remove a star from a repository.
Therefore, the reported data slightly overestimates thenumber of stars. A comparison with the actual numberof stars for the repositories on GitHub reveals that the
numbers are fairly close and that the trends remainunchanged.

SourceCSET’s AI patents dataset was developed by CSET
and 1790 Analytics. It includes patents relevant to thedevelopment and application of AI, as indicated by CPC/
IPC codes and keywords.
Methodology
In this analysis, patents were grouped by year andcountry, and then counted at the “patent family”
level.5 CSET extracted year values from the most recent
publication date within a family. This method has theadvantage of capturing updates within a patent family(such as amendments).
The country of origin for a patent is derived from the first
country in which a patent was filed.6

G I T H U B S TA R S
Source
GitHub: star-history (available at star history website) wasused to retrieve the data.
Methodology
The visual in the report shows the number of stars forvarious GitHub repositories over time. The repositories
include the following:
apachecn/ailearning, apache/incubator-mxnet, AvikJain/100-Days-Of-ML-Code, aymericdamien/TensorFlowExamples, BVLC/cafe, cafe2/cafe2, CorentinJ/Real-TimeVoice-Cloning, deepfakes / faceswap, dmlc/mxnet,
exacity/deeplearningbook-chinese, fchollet/keras,
floodsung/Deep-Learning-Papers-Reading-Roadmap,

5 Patents are analyzed at the “patent family” level rather than “patent document” level because patent families are a collective of patent documents all associated with a single invention and/or
innovation by the same inventors/assignees. Thus, counting at the “patent family” level mitigates artificial number inflation when there are multiple patent documents in a patent family or if a patent isfiled in multiple jurisdictions.
6 For more details on CSET’s approach and experimentation for assigning country values, see footnote 26 in “Patents and Artificial Intelligence: A Primer,” by Dewey Murdick and Patrick Thomas. (Center
for Security and Emerging Technology, September 2020), https://doi.org/10.51593/20200038.

Table of Contents

Appendix

199

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 2: Technical Performance

CHAPTER 2: TECHNICAL PERFORMANCE
ImageNet
Data on ImageNet accuracy was retrieved through adetailed arXiv literature review cross-referenced bytechnical progress reported on Papers with Code. The
reported dates correspond to the year during whicha paper was first published to arXiv, and the reported
results (top-1 or top-5 accuracy) correspond to the result
reported in the most recent version of each paper.
The estimate of human-level performance is from
Russakovsky et al., 2015. Learn more about the LSVRC
ImageNet competition and the ImageNet data set.

ImageNet: Top-1 Accuracy
To highlight progress on top-1 accuracy without the use of
extra training data, scores were taken from the following
papers:Adversarial Examples Improve Image Recognition
Billion-Scale Semi-Supervised Learning for ImageClassification
Dual Path Networks
Densely Connected Convolutional Networks
EfficientNet: Rethinking Model Scaling for Convolutional
Neural Networks
Fixing the Train-Test Resolution Discrepancy:
FixEfficientNet
ImageNet Classification with Deep Convolutional Neural
Networks
Masked Autoencoders Are Scalable Vision Learners
		To highlight progress on top-1 accuracy with the use of
extra training data, scores were taken from the following
papers:Big Transfer (BiT): General Visual Representation Learning
CoAtNet: Marrying Convolution and Attention for All Data
Sizes
EfficientNet: Rethinking Model Scaling for Convolutional
Neural Networks
Self-Training with Noisy Student Improves ImageNet
Classification
Sharpness-Aware Minimization for Efficiently Improving

Table of Contents

Appendix

Generalization
Xception: Deep Learning with Depthwise Separable
Convolutions

ImageNet: Top-5 Accuracy
To highlight progress on top-5 accuracy without the use of
extra training data, scores were taken from the following
papers: 		Adversarial Examples Improve Image Recognition
EfficientNet: Rethinking Model Scaling for Convolutional
Neural Networks
Exploring the Limits of Weakly Supervised Pretraining
Fixing the Train-Test Resolution Discrepancy:
FixEfficientNet
GPipe: Efficient Training of Giant Neural Networks Using
Pipeline Parallelism
High-Performance Large-Scale Image Recognition Without
Normalization
ImageNet Classification with Deep Convolutional Neural
Networks
Learning Transferable Architectures for Scalable Image
Recognition
Squeeze-and-Excitation Networks
To highlight progress on top-5 accuracy with the use of
extra training data, scores were taken from the following
papers:Big Transfer (BiT): General Visual Representation Learning
Deep Residual Learning for Image Recognition
EfficientNet: Rethinking Model Scaling for Convolutional
Neural Networks
Florence: A New Foundation Model for Computer Vision
Self-Training with Noisy Student Improves ImageNet
Classification
Xception: Deep Learning with Depthwise Separable
Convolutions

STL-10
Data on STL-10 FID scores was retrieved through adetailed arXiv literature review cross-referenced by200

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 2: Technical Performancetechnical progress reported on Papers with Code. The
reported dates correspond to the year during whicha paper was first published to arXiv, and the reported
results (FID score) correspond to the result reported in themost recent version of each paper. Details on the STL-10
benchmark can be found in the STL-10 paper.

in which a method was introduced, even if it was
subsequently tested, is the year in which it is included in
the report. The reported results (accuracy) correspondto the result reported in the most recent version of each
paper. Details on the FaceForensics++ benchmark can befound in the FaceForensics++ paper.

To highlight progress on STL-10, scores were taken fromthe following papers:
DEGAS: Differentiable Efficient Generator SearchDist-GAN: An Improved GAN Using Distance Constraints
Off-Policy Reinforcement Learning for Efficient andEffective GAN Architecture
Search Score Matching Model for Unbounded Data Score

To highlight progress on FaceForensics++, scores weretaken from the following papers:
A Deep Learning Approach to Universal Image
Manipulation Detection Using a New Convolutional Layer
Detection of Deepfake Videos Using Long Distance
Attention
FakeCatcher: Detection of Synthetic Portrait Videos Using
Biological Signals
FaceForensics++: Learning to Detect Manipulated Facial
ImagesLearning Spatiotemporal Features with 3D Convolutional
Networks
Recasting Residual-Based Local Descriptors as
Convolutional Neural Networks
Rich Models for Steganalysis of Digital Images
Thinking in Frequency: Face Forgery Detection by Mining
Frequency-Aware Clues
Xception: Deep Learning with Depthwise Separable
Convolutions

CIFAR-10
Data on CIFAR-10 FID scores was retrieved through adetailed arXiv literature review cross-referenced bytechnical progress reported on Papers with Code. The
reported dates correspond to the year during whicha paper was first published to arXiv, and the reported
results (FID score) correspond to the result reported in themost recent version of each paper. Details on the CIFAR-10
benchmark can be found in the CIFAR-10 paper.
To highlight progress on CIFAR-10, scores were taken fromthe following papers:
AutoGAN: Neural Architecture Search for Generative
Adversarial Networks
Denoising Diffusion Probabilistic Models
Improved Training of Wasserstein GANs
Large Scale GAN Training for High Fidelity Natural Image
Synthesis
Score-Based Generative Modeling in Latent Space

FaceForensics++
Data on FaceForensics++ accuracy was retrieved througha detailed arXiv literature review. The reported dates
correspond to the year during which a paper was firstpublished to arXiv or a method was introduced. With
FaceForensics, recent researchers have tested previously
existing deepfake detection methodologies. The year

Table of Contents

Appendix

Celeb-DF
Data on Celeb-DF AUC was retrieved through a detailed
arXiv literature review. The reported dates correspondto the year during which a paper was first published toarXiv or a method was introduced. With Celeb-DF, recent
researchers have tested previously existing deepfake
detection methodologies. The year in which a method wasintroduced, even if it was subsequently tested, is the year
in which it is included in the report. The reported results(AUC) correspond to the result reported in the most recent
version of each paper. Details on the Celeb-DF benchmark
can be found in the Celeb-DF paper.
To highlight progress on Celeb-DF, scores were taken fromthe following papers:
Exposing DeepFake Videos by Detecting Face Warping201

Artificial IntelligenceIndex Report 2022

Artifacts
FaceForensics++: Learning to Detect Manipulated Facial
ImagesFace X-Ray for More General Face Forgery Detection
Spatial-Phase Shallow Learning: Rethinking Face Forgery
Detection in Frequency Domain

Leeds Sports Poses
Data on Leeds Sports Poses percentage of correct
keypoints (PCK) was retrieved through a detailed
arXiv literature review cross-referenced by technicalprogress reported on Papers with Code. The reported
dates correspond to the year during which a paper was
first published to arXiv, and the reported results (PCK)
correspond to the result reported in the most recent
version of each paper. Details on the Leeds Sports Poses
benchmark can be found in the Leeds Sports Poses paper.
To highlight progress on Leeds Sports Poses, scores weretaken from the following papers:
Articulated Pose Estimation by a Graphical Model with
Image Dependent Pairwise Relations
Human Pose Estimation via Convolutional Part Heatmap
Regression
Jointly Optimize Data Augmentation and Network
Training: Adversarial Data Augmentation in Human Pose
Estimation
Knowledge-Guided Deep Fractal Neural Networks for
Human Pose Estimation
OmniPose: A Multi-Scale Framework for Multi-Person Pose
Estimation
Toward Fast and Accurate Human Pose Estimation via
Soft-Gated Skip Connections

Human 3.6M
Data on Human3.6M average mean per joint position error
was retrieved through a detailed arXiv literature review
cross-referenced by technical progress reported on Papers
with Code. The reported dates correspond to the year
during which a paper was first published to arXiv, and thereported results (MPJPE) correspond to the result reported
in the most recent version of each paper. Details on the

Table of Contents

Appendix

APPENDIX
Chapter 2: Technical Performance

Human3.6M benchmark can be found in the Human3.6M
paper.
To highlight progress on Human3.6M without the use of
extra training data, scores were taken from the following
papers:3D Human Pose Estimation in Video with Temporal
Convolutions and Semi-Supervised Training
Conditional Directed Graph Convolution for 3D Human
Pose Estimation
Cross View Fusion for 3D Human Pose Estimation
Epipolar Transformers
Human3.6M: Large Scale Datasets and Predictive Methods
for 3D Human Sensing in Natural Environments
Learning 3D Human Pose from Structure and Motion
Robust Estimation of 3D Human Poses from a Single
Image
To highlight progress on Human3.6M with the use of
extra training data, scores were taken from the following
papers:Epipolar Transformers
Learnable Triangulation of Human Pose
TesseTrack: End-to-End Learnable Multi-Person
Articulated 3D Pose Tracking

Cityscapes Challenge, Pixel-Level Semantic
Labeling Task
Data on the Cityscapes challenge, pixel-level semanticlabeling task mean IoU was taken from the Cityscapes
dataset, more specifically their pixel-level semanticlabeling leaderboard. More details about the Cityscapes
dataset and other corresponding semantic segmentation
challenges can be accessed at the Cityscapes dataset
webpage.

CVC-ClinicDB and Kvasir-SEG
Data on CVC-ClinicDB and Kvasir-SEG mean dice was
retrieved through a detailed arXiv literature review crossreferenced by technical progress reported on Papers
with Code (CVC-ClinicDB and Kvasir-SEG). The reported
dates correspond to the year during which a paper was
first published to arXiv, and the reported results (mean202

Artificial IntelligenceIndex Report 2022dice) correspond to the result reported in the mostrecent version of each paper. Details on the CVC-ClinicDB
benchmark can be found in the CVC-ClinicDB database
page. Details on the Kvasir-SEG benchmark can be foundin the Kvasir-SEG paper.
To highlight progress on CVC-ClinicDB, scores were takenfrom the following papers:
DoubleU-Net: A Deep Convolutional Neural Network for
Medical Image Segmentation
Encoder-Decoder with Atrous Separable Convolution for
Semantic Image Segmentation
MSRF-Net: A Multi-Scale Residual Fusion Network for
Biomedical Image Segmentation
ResUNet++: An Advanced Architecture for Medical Image
Segmentation
U-Net: Convolutional Networks for Biomedical Image
Segmentation
To highlight progress on Kvasir-SEG, scores were takenfrom the following papers:
Encoder-Decoder with Atrous Separable Convolution for
Semantic Image Segmentation
MSRF-Net: A Multi-Scale Residual Fusion Network for
Biomedical Image Segmentation
PraNet: Parallel Reverse Attention Network for PolypSegmentation
U-Net: Convolutional Networks for Biomedical Image
Segmentation

National Institute of Standards and Technology(NIST) Face Recognition Vendor Test (FRVT)
and NIST FRVT Face Mask Effects
Data on NIST FRVT 1:1 verification accuracy by dataset
was obtained from the FRVT 1:1 verification leaderboard.
Data on NIST FRVT face mask effects was obtained fromthe FRVT face mask effects leaderboard. The face maskeffects leaderboard contains results of the testing of
319 face recognition algorithms that were submitted to
FRVT prior to and post mid-March 2020, when the COVID
pandemic began.

Table of Contents

Appendix

APPENDIX
Chapter 2: Technical Performance

Visual Question Answering (VQA)
Data on VQA was taken from recent iterations of the VQA
challenge. To learn more about the VQA challenge in
general, please consult the following link. To learn more
about the 2021 iteration of the VQA challenge, please
consult the following link. More specifically, the Index
makes use of data from the following iterations of the VQA
challenge:
VQA Challenge 2016
VQA Challenge 2017
VQA Challenge 2018
VQA Challenge 2019
VQA Challenge 2020
VQA Challenge 2021

Kinetics-400, Kinetics-600, and Kinetics-700
Data on Kinetics-400, Kinetics-600, and Kinetics-700 was
retrieved through a detailed arXiv literature review crossreferenced by technical progress reported on Papers with
Code (Kinetics-400, Kinetics-600, and Kinetics-700). The
reported dates correspond to the year during which a
paper was first published to arXiv, and the reported results(accuracy) correspond to the result reported in the mostrecent version of each paper. Details on the Kinetics-400
benchmark can be found in the Kinetics-400 paper. Details
on the Kinetics-600 benchmark can be found in the
Kinetics-600 paper. Details on the Kinetics-700 benchmark
can be found in the Kinetics-700 paper.
To highlight progress on Kinetics-400, scores were takenfrom the following papers:
Co-Training Transformer with Videos and Images Improves
Action Recognition
Large-Scale Weakly-Supervised Pre-training for Video
Action Recognition
Multiview Transformers for Video Recognition
Non-Local Neural Networks
Omni-Sourced Webly-Supervised Learning for Video
Recognition
SlowFast Networks for Video Recognition
Temporal Segment Networks: Towards Good Practices for
Deep Action Recognition

203

Artificial IntelligenceIndex Report 2022

To highlight progress on Kinetics-600, scores were takenfrom the following papers:
Masked Feature Prediction for Self-Supervised Visual PreTraining
Multiview Transformers for Video Recognition
Learning Spatio-Temporal Representation with Local andGlobal Diffusion
Rethinking Spatiotemporal Feature Learning: SpeedAccuracy Trade-offs in Video Classification
SlowFast Networks for Video Recognition
To highlight progress on Kinetics-700, scores were takenfrom the following papers:
Learn to Cycle: Time-Consistent Feature Discovery for
Action Recognition
Masked Feature Prediction for Self-Supervised Visual PreTraining
Multiview Transformers for Video Recognition

ActivityNet: Temporal Action Localization Task
In the challenge, there are three separate tasks, butthey focus on the main problem of temporally localizing
where activities happen in untrimmed videos from theActivityNet benchmark. To source information on thestate-of-the-art results for TALT, the Index did a detailed
survey of arXiv papers in addition to reports of yearly
ActivityNet challenge results. More specifically, the Index
made use of the following sources of information:
TALT 2016
TALT 2017
TALT 2018
TALT 2019TALT 2020
TALT 2021

Common Object in Context (COCO)
Data on COCO mean average precision (mAP50) was
retrieved through a detailed arXiv literature review crossreferenced by technical progress reported on Papers
with Code. The reported dates correspond to the year
during which a paper was first published to arXiv, andthe reported results (mAP50) correspond to the result
reported in the most recent version of each paper. Details

Table of Contents

Appendix

APPENDIX
Chapter 2: Technical Performanceon the COCO benchmark can be found in the COCO paper.
To highlight progress on COCO without the use of extra
training data, scores were taken from the following
papers:An Analysis of Scale Invariance in Object Detection – SNIP
Deformable ConvNets v2: More Deformable, Better Results
Dynamic Head: Unifying Object Detection Heads with
Attentions
Inside-Outside Net: Detecting Objects in Context with Skip
Pooling and Recurrent Neural Networks
Mish: A Self Regularized Non-Monotonic Activation
Function
Scaled-YOLOv4: Scaling Cross Stage Partial Network
To highlight progress on COCO with the use of extra
training data, scores were taken from the following
papers:
EfficientDet: Scalable and Efficient Object Detection
Grounded Language-Image Pre-Training

You Only Look Once (YOLO)
Data on YOLO mean average precision (mAP50) was
retrieved through a detailed arXiv literature review
and survey of GitHub repositories. The reported dates
correspond to the year during which a paper was firstpublished to arXiv or a method was introduced. Morespecifically, the Index made use of the following sourcesof information:
YOLO 2016
YOLO 2018
YOLO 2020
YOLO 2021
YOLO results for 2017 and 2019 were not included in theindex as no state-of-the-art improvements in YOLO for
those years were uncovered during the literature review
and survey of GitHub repositories.

Visual Commonsense Reasoning (VCR)
Technical progress for VCR is taken from the VCR
leaderboard; the VCR leaderboard webpage further
delineates the methodology behind the VCR challenge.
204

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 2: Technical Performance

Human performance on VCR is taken from Zellers et al. (2018). Details on the VCR benchmark can be found in the VCR
paper.

SuperGLUEThe SuperGLUE benchmark data was pulled from the SuperGLUE leaderboard. Details about the SuperGLUE benchmark
are in the SuperGLUE paper and SuperGLUE software toolkit. The tasks and evaluation metrics for SuperGLUE are:NAME

IDENTIFIERMETRIC

Broad Coverage Diagnostics

AX-b

Matthew’s Carr

CommitmentBank

CB

Avg. F1/ Accuracy

Choice of Plausible Alternatives

COPA

Accuracy

Multi-Sentence Reading Comprehension

MultiRC

F1a/EM

Recognizing Textual Entailment

RTE

Accuracy

Words in Context

WiC

Accuracy

The Winograd Schema Challenge

WSC

Accuracy

BooIQ

BooIQ

Accuracy

Reading Comprehension with Commonsense Reasoning

ReCoRD

F1/ Accuracy

Winogender Schema Diagnostic

AX-g

Gender Parity/ Accuracy

SQuAD 1.1 and SQuAD 2.0
Data on SQuAD 1.1 performance was taken from Papers
with Code. Data on SQuAD 2.0 performance was takenfrom the SQuAD 2.0 leaderboard. Details about the SQuAD
1.1 benchmark are in the SQuAD 1.1 paper. Details about
the SQuAD 2.0 benchmark are in the SQuAD 2.0 paper.

Reading Comprehension Dataset Requiring
Logical Reasoning (ReClor)
Data on ReClor performance was taken from the ReClor
leaderboard. Details about the ReClor benchmark are in
the ReClor paper.

arXiv
Data on arXiv recall-oriented understudy for gistingevaluation (ROUGE-1) was retrieved through a detailed
arXiv literature review cross-referenced by technicalprogress reported on Papers with Code. The reported
dates correspond to the year during which a paper

Table of Contents

Appendix

was first published to arXiv, and the reported results(ROUGE-1) correspond to the result reported in the mostrecent version of each paper. Details about the arXiv
benchmark are in the arXiv dataset webpage.
To highlight progress on arXiv without the use of extra
training data, scores were taken from the following
papers:A Discourse-Aware Attention Model for Abstractive
Summarization of Long Documents
Extractive Summarization of Long Documents by
Combining Global and Local Context
Get to the Point: Summarization with Pointer-Generator
Networks
Systematically Exploring Redundancy Reduction in
Summarizing Long Documents
Sparsifying Transformer Models with Trainable
Representation Pooling205

Artificial IntelligenceIndex Report 2022

To highlight progress on arXiv with the use of extra
training data, scores were taken from the following
papers:Big Bird: Transformers for Longer Sequences
Hierarchical Learning for Generation with Long Source
Sequences
PEGASUS: Pre-Training with Extracted Gap-Sentences for
Abstractive Summarization

PubMedData on PubMed recall-oriented understudy for gistingevaluation (ROUGE-1) was retrieved through a detailed
arXiv literature review cross-referenced by technicalprogress reported on Papers with Code. The reported
dates correspond to the year during which a paper
was first published to arXiv, and the reported results(ROUGE-1) correspond to the result reported in the mostrecent version of each paper. Details about the PubMedbenchmark are in the PubMed paper.
To highlight progress on PubMed without the use of
extra training data, scores were taken from the following
papers:A Discourse-Aware Attention Model for Abstractive
Summarization of Long Documents
Extractive Summarization of Long Documents by
Combining Global and Local Context
Get to the Point: Summarization with Pointer-Generator
Networks
Sparsifying Transformer Models with Trainable
Representation PoolingTo highlight progress on PubMed with the use of extra
training data, scores were taken from the following
papers:A Divide-and-Conquer Approach to the Summarization ofLong Documents
Hierarchical Learning for Generation with Long Source
Sequences
PEGASUS: Pre-Training with Extracted Gap-Sentences for
Abstractive Summarization

Table of Contents

Appendix

APPENDIX
Chapter 2: Technical Performance

Stanford Natural Language Inference (SNLI)
Data on Stanford Natural Language Inference (SNLI)
accuracy was retrieved through a detailed arXiv literature
review cross-referenced by technical progress reported on
Papers with Code. The reported dates correspond to theyear during which a paper was first published to arXiv, andthe reported results (accuracy) correspond to the result
reported in the most recent version of each paper. Details
on the SNLI benchmark can be found in the SNLI paper.
To highlight progress on SNLI, scores were taken from thefollowing papers:
Compare, Compress and Propagate: Enhancing NeuralArchitectures with Alignment Factorization for Natural
Language Inference
Convolutional Neural Networks for Sentence Classification
Enhanced LSTM for Natural Language Inference
Entailment as Few-Shot Learner
Explicit Contextual Semantics for Text Comprehension
Semantics-Aware BERT for Language Understanding
Self-Explaining Structures Improve NLP Models

Abductive Natural Language Inference (aNLI)
Data on Abductive Natural Language Inference (aNLI) was
sourced from the Allen Institute for AI’s aNLI leaderboard.
Details on the aNLI benchmark can be found in the aNLI
paper.

SemEval 2014 Task 4 Sub Task 2
Data on SemEval 2014 Task 4 Sub Task 2 accuracy was
retrieved through a detailed arXiv literature review crossreferenced by technical progress reported on Papers
with Code. The reported dates correspond to the year
during which a paper was first published to arXiv, andthe reported results (accuracy) correspond to the result
reported in the most recent version of each paper. Details
on the SemEval benchmark can be found in the SemEval
2014 paper.
To highlight progress on SemEval, scores were taken fromthe following papers:
A Multi-Task Learning Model for Chinese-Oriented Aspect
Polarity Classification and Aspect Term Extraction
206

Artificial IntelligenceIndex Report 2022

Aspect Level Sentiment Classification with Deep Memory
Network
Back to Reality: Leveraging Pattern-Driven Modeling to
Enable Affordable Sentiment Dependency Learning
Effective LSTMs for Target-Dependent SentimentClassification
Hierarchical Attention Based Position-Aware Network for
Aspect-Level Sentiment Analysis
Investigating Typed Syntactic Dependencies for Targeted
Sentiment Classification Using Graph Attention Neural
Network
Recurrent Attention Network on Memory for Aspect
Sentiment Analysis

WMT2014, English-French and English-German
Data on WMT2014 English-French and English-German
BLEU score was retrieved through a detailed arXiv
literature review cross-referenced by technical progress
reported on Papers with Code (English-French andEnglish-German). The reported dates correspond to theyear during which a paper was first published to arXiv, andthe reported results (BLEU score) correspond to the result
reported in the most recent version of each paper. Details
about both the WMT2014 English-French and EnglishGerman benchmarks can be found in the WMT2014 paper.
To highlight progress on WMT2014 English-French without
the use of extra training data, scores were taken from thefollowing papers:
Addressing the Rare Word Problem in Neural MachineTranslation
Google’s Neural Machine Translation System: Bridging the
Gap Between Human and Machine Translation
MUSE: Parallel Multi-Scale Attention for Sequence toSequence Learning
R-Drop: Regularized Dropout for Neural Networks
Scaling Neural Machine Translation
Understanding the Difficulty of Training Transformers
Weighted Transformer Network for Machine Translation
To highlight progress on WMT2014 English-French withthe use of extra training data, scores were taken from theTable of Contents

Appendix

APPENDIX
Chapter 2: Technical Performance

following papers:
Understanding Back-Translation at Scale
Very Deep Transformers for Neural Machine Translation
To highlight progress on WMT2014 English-German
without the use of extra training data, scores were takenfrom the following papers:
BERT, mBERT, or BiBERT? A Study on Contextualized
Embeddings for Neural Machine Translation
Effective Approaches to Attention-based Neural MachineTranslation
Data Diversification: A Simple Strategy for Neural MachineTranslation
Fast and Simple Mixture of Softmaxes with BPE andHybrid-LightRNN for Language Generation
Google’s Neural Machine Translation System: Bridging the
Gap Between Human and Machine Translation
Incorporating BERT into Neural Machine Translation
Weighted Transformer Network for Machine Translation
To highlight progress on WMT2014 English-German withthe use of extra training data, scores were taken from thefollowing papers:
Lessons on Parameter Sharing across Layers inTransformers
Understanding Back-Translation at Scale

Number of Commercially Available MT
Systems
Details about the number of commercially available MTsystems was sourced from Intento’s report The State ofMachine Translation, 2021. Intento is a San Franciscobased startup that analyzes commercially available MT
services.

LibriSpeech (Test-Clean and Other Dataset)
Data on LibriSpeech (Test-Clean and Other) word error
rate was retrieved through a detailed arXiv literature
review cross-referenced by technical progress reported
on Papers with Code (Test-Clean and Other). The reported
dates correspond to the year during which a paper was
first published to arXiv, and the reported results (word207

Artificial IntelligenceIndex Report 2022

error rate) correspond to the result reported in the mostrecent version of each paper. Details about both the
LibriSpeech Test-Clean and Test-Other benchmarks canbe found in the LibriSpeech paper.
To highlight progress on LibriSpeech Test-Clean without
the use of extra training data, scores were taken from thefollowing papers:
ASAPP-ASR: Multistream CNN and Self-Attentive SRU for
SOTA Speech Recognition
Letter-Based Speech Recognition with Gated ConvNets
Neural Network Language Modeling with Letter-BasedFeatures and Importance Sampling
SpeechStew: Simply Mix All Available Speech Recognition
Data to Train One Large Neural Network
State-of-the-Art Speech Recognition Using Multi-Stream
Self-Attention With Dilated 1D Convolutions
To highlight progress on LibriSpeech Test-Clean with theuse of extra training data, scores were taken from thefollowing papers:
Deep Speech 2: End-to-End Speech Recognition in English
and Mandarin
End-to-End ASR: From Supervised to Semi-Supervised
Learning with Modern Architectures
Pushing the Limits of Semi-Supervised Learning for
Automatic Speech Recognition
To highlight progress on LibriSpeech Test-Other without
the use of extra training data, scores were taken from thefollowing papers:
Conformer: Convolution-Augmented Transformer for
Speech Recognition
Neural Network Language Modeling with Letter-BasedFeatures and Importance Sampling
SpeechStew: Simply Mix All Available Speech Recognition
Data to Train One Large Neural Network
Transformer-Based Acoustic Modeling for Hybrid Speech
Recognition

Table of Contents

Appendix

APPENDIX
Chapter 2: Technical Performance

To highlight progress on LibriSpeech Test-Other with theuse of extra training data, scores were taken from thefollowing papers:
Deep Speech 2: End-to-End Speech Recognition in English
and Mandarin
End-to-End ASR: From Supervised to Semi-Supervised
Learning with Modern Architectures
Pushing the Limits of Semi-Supervised Learning for
Automatic Speech Recognition
W2v-BERT: Combining Contrastive Learning and Masked
Language Modeling for Self-Supervised Speech PreTraining

VoxCeleb
VoxCeleb is an audio-visual dataset consisting of shortclips of human speech, extracted from interview videos
uploaded to YouTube. VoxCeleb contains speech from
7,000-plus speakers spanning a wide range of ethnicities,
accents, professions, and ages—amounting to over a
million utterances (face-tracks are captured “in the wild,”with background chatter, laughter, overlapping speech,
pose variation, and different lighting conditions) recorded
over a period of 2,000 hours (both audio and video). Eachsegment is at least three seconds long. The data contains
an audio dataset based on celebrity voices, shorts,
films, and conversational pieces (e.g., talk shows). The
initial VoxCeleb 1 (100,000 utterances taken from 1,251
celebrities on YouTube) was expanded to VoxCeleb 2 (1million utterances from 6,112 celebrities).
For the sake of consistency, the AI Index reported scoreson the initial VoxCeleb dataset. Specifically, the Index
made use of the following sources of information:
The IDLAB VoxSRC-20 Submission: Large Margin FineTuning and Quality-Aware Score Calibration in DNN BasedSpeaker Verification
The SpeakIn System for VoxCeleb Speaker Recognition
Challenge 2021
VoxCeleb: A Large-Scale Speaker Identification Dataset
VoxCeleb2: Deep Speaker Recognition
VoxCeleb: Large-Scale Speaker Verification in the Wild

208

Artificial IntelligenceIndex Report 2022

MovieLens 20M
Data on MovieLens 20M normalized discounted
cumulative gain@100 (nDCG@100) was retrieved througha detailed arXiv literature review cross-referenced bytechnical progress reported on Papers with Code. The
reported dates correspond to the year during whicha paper was first published to arXiv, and the reported
results (nDCG@100) correspond to the result reported
in the most recent version of each paper. Details on theMovieLens series of benchmarks can be found in Harper et
al. 2015.
To highlight progress on MovieLens 20M, scores weretaken from the following papers:
Deep Variational Autoencoder with Shallow Parallel Path
for Top-N Recommendation (VASP)
Enhancing VAEs for Collaborative Filtering: Flexible Priors& Gating Mechanisms
RaCT: Toward Amortized Ranking-Critical Training for
Collaborative Filtering
Variational Autoencoders for Collaborative Filtering

CriteoData on Criteo area under curve score (AUC) was retrieved
through a detailed arXiv literature review cross-referenced
by technical progress reported on Papers with Code.
The reported dates correspond to the year during whicha paper was first published to arXiv, and the reported
results (AUC) correspond to the result reported in themost recent version of each paper. Details on the Criteo
benchmark can be found on the Criteo Kaggle Challenge
page.
To highlight progress on Criteo, scores were taken fromthe following papers:
AutoInt: Automatic Feature Interaction Learning via SelfAttentive Neural Networks
DeepFM: A Factorization-Machine Based Neural Network
for CTR Prediction
DeepLight: Deep Lightweight Feature Interactions for
Accelerating CTR Predictions in Ad Serving
FAT-DeepFFM: Field Attentive Deep Field-aware
Factorization Machine
Table of Contents

Appendix

APPENDIX
Chapter 2: Technical Performance

MaskNet: Introducing Feature-Wise Multiplication to CTR
Ranking Models by Instance-Guided Mask
Product-Based Neural Networks for User Response
Prediction

Arcade Learning Environment: Atari-57
Data on Arcade Learning Environment: Atari-57 meanhuman-normalized score was retrieved through a detailed
arXiv literature review cross-referenced by technicalprogress reported on Papers with Code. The reported
dates correspond to the year during which a paper was
first published to arXiv, and the reported results (meanhuman-normalized score) correspond to the result
reported in the most recent version of each paper. Details
on the Arcade Learning Environment: Atari-57 benchmark
can be found in the Arcade Learning Environment paper.
To highlight progress on Arcade Learning Environment:
Atari-57, scores were taken from the following papers:
Dueling Network Architectures for Deep Reinforcement
Learning
Distributional Reinforcement Learning with Quantile
Regression
GDI: Rethinking What Makes Reinforcement Learning
Different From Supervised Learning
Mastering Atari, Go, Chess and Shogi by Planning with a
Learned Model
Recurrent Experience Replay in Distributed Reinforcement
Learning

Procgen
Data on Procgen mean-normalized score was retrieved
through a detailed arXiv literature review. The reported
dates correspond to the year during which a paper was
first published to arXiv, and the reported results (meannormalized score) correspond to the result reported in themost recent version of each paper. Details on the Procgenbenchmark can be found in the Procgen paper.
To highlight progress on Procgen, scores were taken fromthe following papers:
Automatic Data Augmentation for Generalization in
Reinforcement Learning
209

Artificial IntelligenceIndex Report 2022

Leveraging Procedural Generation to Benchmark
Reinforcement Learning
Procedural Generalization by Planning with SelfSupervised World Models

Chess
Data on the performance of chess software engines was
taken from the Swedish Chess Computer Association’s
ranking of top chess software engines. The Swedish Chess
Computer Association tests computer chess softwaresystems against one another and releases a ranking list of
the top-performing systems. The ranking list produced by
the Swedish Chess Computer Association is a statisticallysignificant and meaningful measurement of chess engine
performance because engines are pitted against one
another in thousands of tournament-like games and each
employ the same underlying hardware. Data on Magnus
Carlsen’s top ELO score was taken from the International
Chess Federation.

Training Time and Number of Accelerators
Data on training time and number of accelerators for AI
systems was taken from the MLPerf Training benchmark
competitions. More specifically, the AI Index made use ofdata from the following MLPerf training competitions:
MLPerf Training v0.5, 2018
MLPerf Training v0.6, 2019
MLPerf Training v0.7, 2020
MLPerf Training v1.0, 2021
MLPerf Training v1.1, 2021

Table of Contents

Appendix

APPENDIX
Chapter 2: Technical Performance

Details on the MLPerf Training benchmark can be foundin the MLPerf Training benchmark paper. Details on
the current benchmark categories as well as technicalinformation about submission and competition
subdivisions can be found on the MLPerf Training
webpage.

ImageNet Training Cost
Data on ImageNet Training cost was based on researchfrom DAWNBench and the individual research of Deepak
Narayanan. DAWNBench is a benchmark suite for end-toend deep-learning training and inference. DAWNBenchprovides a reference set of common deep-learning
workloads for quantifying training time, training cost,inference latency, and inference cost across different
optimization strategies, model architectures, software
frameworks, clouds, and hardware. More details available
at DAWNBench.
Because DAWNBench was deprecated after March 2020,
data on the training cost for the most recent round of
MLPerf submissions was manually collected by Deepak
Narayanan.

210

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 2: Technical Performance

AI Index Robotics Survey
The survey was distributed to 509 professors who specialize in robotics from 67 universities online over three waves from
December 2021 to February 2022. The selection of universities was based on the World University Rankings 2021 withgeographic representation across the globe. Please view the complete survey raw data in our public data folder here.
The survey was completed by 101 professors from 43 universities, including:​​
Aalborg
University, Denmark
Ain Shams University, Egypt
Carnegie Mellon University, United States
Columbia University, United States
Cornell University, United States
Delft University of Technology, Netherlands
ETH Zurich, Switzerland
Hong Kong University of Science and Technology, Hong
Kong
Korea Advanced Institute of Science and Technology,
South Korea
KU Leuven, Belgium
Massachusetts Institute of Technology, United States
Nanyang Technological University, Singapore
National Polytechnic Institute, Mexico
National University of Singapore, Singapore
Peking University, China
Politecnico di Milano, Italy
Pontificia Universidad Católica de Chile, Chile
Princeton University, United States
Purdue University, United States
RWTH Aachen University, Germany
Seoul National University, South Korea

Table of Contents

Appendix

Stanford University, United States
Stellenbosch University, South Africa
Swiss Federal Institute of Technology Lausanne,
Switzerland
Tokyo Institute of Technology, Japan
University of British Columbia, Canada
University of California Berkeley, United States
University of California Los Angeles, United States
University of California San Diego, United States
University of Cambridge, United Kingdom
University of Cape Town, South Africa
University College London, United Kingdom
University of Hong Kong, Hong kong
University of Illinois at Urbana-Champaign, United States
University of Malaya, Malaysia
University of Manchester, United Kingdom
University of Michigan, United States
Universitat Politècnica de Catalunya, Spain
University of Texas at Austin, United States
University of Tokyo, Japan
University of Toronto, Canada
University of Waterloo, Canada
Zhejiang University, China

211

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 3: Technical AI Ethics

CHAPTER 3: TECHNICAL AI ETHICSA I E T H I C S T R E N D S AT FAC CT A N D
NEURIPSTo understand trends at the ACM Conference on Fairness,
Accountability, and Transparency, this section tracks
FAccT papers published in conference proceedings
from 2018 to 2021. We categorize author affiliations
into academic, industry, nonprofit, government, andindependent categories, while also tracking the locationof their affiliated institution. Authors with multiple
affiliations are counted once in each category (academic
and industry), but multiple affiliations of the same type(i.e., authors belonging to two academic institutions) arecounted once in the category.
For the analysis conducted on NeurIPS publications, we
identify workshops themed around real-world impact andlabel papers with a single main category in “healthcare,”“climate,” “finance,” “developing world,” or “other,” where“other” denotes a paper related to a real-world use casebut not in one of the other categories.
We tally the number of papers in each category to reachthe numbers found in Figure 3.3.3. Papers are not doublecounted in multiple categories. We note that this data
may not be as accurate for data pre-2018 as societal
impacts work at NeurIPS has historically been categorizedunder a broad “AI for social impact” umbrella,7 but it has
recently been split into more granular research areas.
Examples include workshops dedicated to machinelearning for health,8 climate,9 policy & governance10,
disaster response11, and the developing world.12
To track trends around specific technical topics at
NeurIPS as in Figures 3.3.4–3.3.7, we count the numberof papers accepted to the NeurIPS main track with titles
containing keywords (e.g., “counterfactual” or “causal”
for tracking papers related to causal effect), as well as
papers submitted to related workshops. See the list ofworkshops considered for analysis here.

M E TA-A N A LYS I S O F FA I R N E S S
AND BIAS METRICS
For the analysis conducted on fairness and bias metrics inAI, we identify and report on benchmark and diagnostic
metrics which have been consistently cited in theacademic community, reported on a public leaderboard,or reported for publicly available baseline models (e.g.,
GPT-3, BERT, ALBERT). We note that research paper
citations are a lagging indicator of adoption, and metrics
which have been very recently adopted may not bereflected in the data for 2021.
For Figures 3.1.1 and 3.1.2, we track metrics from thefollowing papers and projects:
Aligning AI with Shared Human Values
Assessing Social and Intersectional Biases inContextualized Word Representations
Bias in Bios: A Case Study of Semantic Representation Biasin a High-Stakes Setting  
BOLD: Dataset and Metrics for Measuring Biases in OpenEnded Language Generation
Certifying and Removing Disparate Impact
CivilComments: Jigsaw Unintended Bias in Toxicity
Classification
CrowS-Pairs: A Challenge Dataset for Measuring Social
Biases in Masked Language Models

7 See 2018 Workshop on Ethical, Social and Governance Issues in AI 2018, 2018 AI for Social Good Workshop, 2019 Joint Workshop on AI for Social Good, 2020 Resistance AI Workshop, 2020 Navigating theBroader Impacts of AI Research Workshop.
8 See 2014 Machine Learning for Clinical Data Analysis, Healthcare and Genomics, 2015 Machine Learning for Healthcare, 2016 Machine Learning for Health, 2017 Machine Learning for Health.
9 See 2013 Machine Learning for Sustainability, 2020 AI for Earth Sciences, 2019, 2020, 2021 Tackling Climate Change with ML.
10 See 2016 People and Machines, 2019 Joint Workshop on AI for Social Good–Public Policy, 2021 Human-Centered AI.
11 See 2019 AI for Humanitarian Assistance and Disaster Response, 2020 Second Workshop on AI for Humanitarian Assistance and Disaster Response, 2021 Third Workshop on AI for Humanitarian
Assistance and Disaster Response.
12 See 2017–2021 Machine Learning for the Developing World Workshops.

Table of Contents

Appendix212

Artificial IntelligenceIndex Report 2022

Detecting Emergent Intersectional Biases: Contextualized
Word Embeddings Contain a Distribution of Human-Like
BiasesEquality of Opportunity in Supervised Learning
Evaluating Gender Bias in Machine Translation
Evaluating Gender Bias in Natural Language Inference
Examining Gender Bias in Languages with Grammatical
GenderFairness Through Awareness
Gender Bias in Coreference Resolution
Gender Bias in Coreference Resolution: Evaluation and
Debiasing Methods
Gender Bias in Multilingual Embeddings and CrossLingual Transfer
Gender Shades: Intersectional Accuracy Disparities in
Commercial Gender Classification
Image Representations Learned with Unsupervised
Pretraining Contain Human-Like Biases
Measuring and Reducing Gendered Correlations in
Pretrained Models
Measuring Bias in Contextualized Word Representations
Measuring Bias with Wasserstein DistanceNuanced Metrics for Measuring Unintended Bias with Real
Data for Text Classification
On Formalizing Fairness in Prediction with Machine
Learning
On Measuring Social Biases in Sentence Encoders
Perspective API
RealToxicityPrompts: Evaluating Neural Toxic
Degeneration in Language Models
Scaling Language Models: Methods, Analysis & Insightsfrom Training Gopher
Semantics Derived Automatically from Language Corpora
Contain Human-Like Biases
StereoSet: Measuring Stereotypical Bias in Pretrained
Language Models
The Woman Worked as a Babysitter: On Biases in
Language Generation
When Worlds Collide: Integrating Different Counterfactual
Assumptions in Fairness

Table of Contents

Appendix

APPENDIX
Chapter 3: Technical AI Ethics

N AT U R A L L A N G UAG E P R O C E S S I N G
BIAS METRICS
In Section 3.2, we track citations of the Perspective API
created by Jigsaw at Google. The Perspective API has been
adopted widely by researchers and engineers in natural
language processing. Its creators define toxicity as “a rude,
disrespectful, or unreasonable comment that is likely tomake someone leave a discussion,” and the tool is poweredby machine learning models trained on a proprietarydataset of comments from Wikipedia and news websites.
We include the following papers in our analysis:#ContextMatters: Advantages and Limitations of Using
Machine Learning to Support Women in Politics
A General Language Assistant as a Laboratory for
Alignment
A Machine Learning Approach to Comment Toxicity
Classification
A Novel Preprocessing Technique for Toxic Comment
Classification
Adversarial Text Generation for Google’s Perspective API
Avoiding Unintended Bias in Toxicity Classification with
Neural Networks
Bad‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌‌Charac ters: Imperc eptible NLP Attac ks
Challenges in Detoxifying Language Models
Classification of Online Toxic Comments Using Machine
Learning Algorithms
Context Aware Text Classification and Recommendation
Model for Toxic Comments Using Logistic Regression
Detecting Cross-Geographic Biases in Toxicity Modeling on
Social Media
Detoxifying Language Models Risks Marginalizing Minority
VoicesFighting Hate Speech, Silencing Drag Queens? Artificial
Intelligence in Content Moderation and Risks to LGBTQ
Voices Online
HATEMOJI: A Test Suite and Adversarially Generated
Dataset for Benchmarking and Detecting Emoji-Based
Hate
HotFlip: White-Box Adversarial Examples for Text
Classification
Identifying Latent Toxic Features on YouTube Using NonNegative Matrix Factorization

213

Artificial IntelligenceIndex Report 2022

Interpreting Social Respect: A Normative Lens for ML
ModelsKnowledge-Based Neural Framework for Sexism
Detection and Classification
Large Pretrained Language Models Contain Human-Like
Biases of What Is Right and Wrong to Do
Leveraging Multilingual Transformers for Hate Speech
Detection
Limitations of Pinned AUC for Measuring Unintended Bias
Machine Learning Suites for Online Toxicity Detection
Mitigating Harm in Language Models with ConditionalLikelihood Filtration
On-the-Fly Controlled Text Generation with Experts andAnti-Experts
Process for Adapting Language Models to Society (PALMS)
with Values-Targeted Datasets
Racial Bias in Hate Speech and Abusive Language
Detection Datasets
RealToxicityPrompts: Evaluating Neural Toxic
Degeneration in Language Models
Scaling Language Models: Methods, Analysis & Insightsfrom Training Gopher
Self-Diagnosis and Self-Debiasing: A Proposal for
Reducing Corpus-Based Bias in NLPSocial Bias Frames: Reasoning About Social and Power
Implications of Language
Social Biases in NLP Models as Barriers for Persons with
Disabilities
Stereotypical Bias Removal for Hate Speech Detection
Task Using Knowledge-Based Generalizations
The Risk of Racial Bias in Hate Speech Detection
Towards Measuring Adversarial Twitter Interactions
Against Candidates in the US Midterm Elections
Toxic Comment Classification Using Hybrid Deep Learning
Model
Toxicity-Associated News Classification: The Impact ofMetadata and Content Features
Understanding BERT Performance in Propaganda Analysis
White-to-Black: Efficient Distillation of Black-Box
Adversarial Attacks
Women, Politics and Twitter: Using Machine Learning toChange the Discourse

Table of Contents

Appendix

APPENDIX
Chapter 3: Technical AI Ethics

While the Perspective API is used widely within machinelearning research and also for measuring online toxicity,
toxicity in the specific domains used to train the models
undergirding Perspective (e.g., news, Wikipedia) may not
be broadly representative of all forms of toxicity (e.g.,trolling). Other known caveats include biases against text
written by minority voices: The Perspective API has beenshown to disproportionately assign high toxicity scoresto text that contains mentions of minority identities (e.g.,“I am a gay man”). As a result, detoxification techniques
built with labels sourced from the Perspective API resultin models that are less capable of modeling language
used by minority groups, and they avoid mentioning
minority identities.
We note that the effect size metric reported in the
Word Embeddings Association Test (WEAT) section ishighly sensitive to rare words, as it has been shownthat removing less than 1% of relevant documents ina corpus can significantly impact the WEAT effect size.
This means that effect size is not guaranteed to be arobust metric for assessing bias in embeddings. While
we report on a subset of embedding association tasks
measuring bias along gender and racial axes, theseembedding association tests have been extended toquantify the effect across intersectional axes (e.g.,
EuropeanAmerican+male, AfricanAmerican+male,
AfricanAmerican+female).
In the analysis of embeddings from over 100 years of
U.S. Census data, embedding bias was measured bycomputing the difference between average embedding
distances. For example, gender bias is calculated as theaverage distance of embeddings of words associated withwomen (e.g., she, female) compared to embeddings ofwords for occupations (e.g., teacher, lawyer), minus thesame average distance calculated for words associatedwith men.

214

Artificial IntelligenceIndex Report 2022

FAC T UA L I T Y A N D T R U T H F U L N E S SDefinitions
The concepts of factuality, factual correctness, factual
accuracy, and veracity are all used to refer to conformity
with facts or truth. Recent work in AI aims to assess
factual correctness within language models andcharacterize their limitations.
While human truthfulness is a relatively well-understoodconcept, truthfulness is not a well-characterized concept
within the context of AI. A group of researchers has
proposed frameworks for what it means for a system
to be truthful—for example, a broadly truthful systemshould avoid lying or using true statements to misleador misdirect; should be clear, informative, and (mostly)
cooperative in conversation; and should be well-calibrated,
self-aware, and open about the limits of its knowledge.
A definition of narrow truthfulness may simply refer tosystems which avoid stating falsehoods. The authors of
TruthfulQA define a system as truthful only if it avoids
asserting a false statement; refusing to answer a question,
expressing uncertainty, or giving a true but irrelevant
answer may be considered truthful but not informative.
Truthfulness is related to alignment: A truthful system isaligned with human values and goals. In one definition ofalignment, an aligned system is one that is helpful, honest,
and harmless. Since we cannot yet measure honesty withina system, truthfulness can be used as a proxy.
An honest system is one that asserts only what it“believes” or one that never contradicts its own beliefs.
A system can be honest but not truthful—for example, if
an honest system believes that vaccines are unsafe, it canclaim this honestly, despite the statement being factually
incorrect. Conversely, a system can be truthful but nothonest: It may believe vaccines are unsafe but asserts
they are safe to pass a test. Another work proposes thatan honest system should give accurate information, notmislead users, be calibrated (e.g., it should be correct 80%of the time when it claims 80% confidence), and express
appropriate levels of uncertainty.
Table of Contents

Appendix

APPENDIX
Chapter 3: Technical AI Ethics

Hallucination refers to language models fabricatingstatements not found in factually correct supporting
evidence or input documents. In closed-form dialog,
summarization, or question-answering, a system that
hallucinates is considered untruthful.

Language Diversity in Training Data
Imbalanced language distribution in training data
impacts the performance of general-purpose language
models. For example, the Gopher family of models istrained on MassiveText (10.5TB), which is a dataset
made up of 99% English. Similarly, only 7.4% of GPT-3
training data is in non-English languages. In contrast,
XGLM, a recent model family from Meta AI, is trained ona training data of 30 languages, and upsamples lowresource languages to create a more balanced language
representation. See Figure 1 on the XGLM paper thatcompares the language distribution of XGLM and GPT-3.
In addition, Figure 7 of the XGLM paper highlights theextent to which language models can effectively store
factual knowledge by comparing the performance of
XGLM (a multilingual language model) with GPT-3, amonolingual model. Performance was evaluated on
knowledge triplet completion using the mLAMA dataset,
which was translated from the English benchmark LAMA
using Google Translate. GPT-3 outperforms in English,
but XGLM outperforms in non-English languages. Further
results show that more diverse language representation
improves language model performance in tasks such astranslation.
In 2021, Congress inquired into the content moderation
practices of social media companies in non-English
languages, and emphasized the importance of equal
access to truthful and trustworthy information regardlessof language. As these companies start to adopt language
models into their fact-checking and content moderation
processes for languages around the world, it is critical tobe able to measure the disproportionate negative impactof using models which underperform on non-English
languages.

215

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 4: The Economy and Education

CHAPTER 4: THE ECONOMY
AND EDUCATIONEMSI BURNING GLASS
Prepared by Julia Nitschke, Summer Jasinski, Bledi Taska
and Rucha Vankudre
Emsi Burning Glass delivers job market analytics
that empower employers, workers, and educators tomake data-driven decisions. The company’s artificial
intelligence technology analyzes hundreds of millions
of job postings and real-life career transitions to provide
insight into labor market patterns. This real-time strategic
intelligence offers crucial insights, such as what jobs aremost in demand, the specific skills employers need, andthe career directions that offer the highest potential for
workers. For more information, visit burning-glass.com.

Job Posting Data
To support these analyses, Emsi Burning Glass mined
its dataset of millions of job postings collected since2010. Emsi Burning Glass collects postings from over
45,000 online job sites to develop a comprehensive,
real-time portrait of labor market demand. It aggregates
job postings, removes duplicates, and extracts datafrom job postings text. This includes information on job
title, employer, industry, and region, as well as required
experience, education, and skills.
Job postings are useful for understanding trends in thelabor market because they allow for a detailed, realtime look at the skills employers seek. To assess therepresentativeness of job postings data, Emsi Burning
Glass conducts a number of analyses to compare the
distribution of job postings to the distribution of official
government and other third-party sources in the UnitedStates. The primary source of government data on U.S.
job postings is the Job Openings and Labor Turnover
Survey (JOLTS) program, conducted by the Bureau ofLabor Statistics. Based on comparisons between JOLTS

Table of Contents

Appendixand Emsi Burning Glass, the labor market demand
captured by Emsi Burning Glass data represents over 95%
of the total labor demand. Jobs not posted online areusually in small businesses (the classic example being the“Help Wanted” sign in the restaurant window) and union
hiring halls.

Measuring Demand for AI
In order to measure employers’ demand for AI skills, Emsi
Burning Glass uses its skills taxonomy of over 17,000 skills.
The list of AI skills from Emsi Burning Glass data is shown
below, with associated skill clusters. While some skills
are considered to be in the AI cluster specifically, for the
purposes of this report, all skills below were considered AI
skills. A job posting was considered an AI job if it requested
one or more of these skills.
Artificial Intelligence: Expert System, IBM Watson, IPSoft
Amelia, Ithink, Virtual Agents, Autonomous Systems, Lidar,
OpenCV, Path Planning, Remote Sensing
Natural Language Processing (NLP): ANTLR, Automatic
Speech Recognition (ASR), Chatbot, Computational
Linguistics, Distinguo, Latent Dirichlet Allocation, Latent
Semantic Analysis, Lexalytics, Lexical Acquisition, Lexical
Semantics, Machine Translation (MT), Modular Audio
Recognition Framework (MARF), MoSes, Natural Language
Processing, Natural Language Toolkit (NLTK), Nearest
Neighbor Algorithm, OpenNLP, Sentiment Analysis/
Opinion Mining, Speech Recognition, Text Mining, Text to
Speech (TTS), Tokenization, Word2Vec
Neural Networks: Caffe Deep Learning Framework,
Convolutional Neural Network (CNN), Deep Learning,
Deeplearning4j, Keras, Long Short-Term Memory (LSTM),
MXNet, Neural Networks, Pybrain, Recurrent Neural
Network (RNN), TensorFlow

216

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 4: The Economy and Education

Machine Learning: AdaBoost algorithm, Boosting(Machine Learning), Chi Square Automatic Interaction
Detection (CHAID), Classification Algorithms, Clustering
Algorithms, Decision Trees, Dimensionality Reduction,
Google Cloud Machine Learning Platform, Gradientboosting, H2O (software), Libsvm, Machine Learning,
Madlib, Mahout, Microsoft Cognitive Toolkit, MLPACK (C++library), Mlpy, Random Forests, Recommender Systems,
Scikit-learn, Semi-Supervised Learning, Supervised
Learning (Machine Learning), Support Vector Machines(SVM), Semantic Driven Subtractive Clustering Method(SDSCM), Torch (Machine Learning), Unsupervised
Learning, Vowpal, Xgboost

and classified by taxonomists at LinkedIn into 249 skill
groupings, which are the skill groups represented in thedataset. The top skills that make up the AI skill grouping
are machine learning, natural language processing, data
structures, artificial intelligence, computer vision, image
processing, deep learning, TensorFlow, Pandas (software),and OpenCV, among others.

Robotics: Blue Prism, Electromechanical Systems,
Motion Planning, Motoman Robot Programming, Robot
Framework, Robotic Systems, Robot Operating System(ROS), Robot Programming, Servo Drives / Motors,
Simultaneous Localization and Mapping (SLAM)

Skills Genome

Visual Image Recognition: Computer Vision, Image
Processing, Image Recognition, Machine Vision, Object
Recognition

Skill groupings are derived by expert taxonomists
through a similarity-index methodology that measures
skill composition at the industry level. Industries areclassified according to the ISIC 4 industry classification(Zhu et al., 2018).

For any entity (occupation or job, country, sector, etc.), theskill genome is an ordered list (a vector) of the 50 “mostcharacteristic skills” of that entity. These most characteristic
skills are identified using a TF-IDF algorithm to identify themost representative skills of the target entity, while downranking ubiquitous skills that add little information aboutthat specific entity (e.g., Microsoft Word).

Included countries represent a select sample of eligible
countries with at least 40% labor force coverage byLinkedIn and at least 10 AI hires in any given month.
China and India were included in this sample because of
their increasing importance in the global economy, but
LinkedIn coverage in these countries does not reach 40%of the workforce. Insights for these countries may notprovide as full a picture as other countries, and should beinterpreted accordingly.

TF-IDF is a statistical measure that evaluates how
representative a word (in this case a skill) is to a selected
entity. This is done by multiplying two metrics:
1. The term frequency of a skill in an entity (TF).
2. The logarithmic inverse entity frequency of the skill
across a set of entities (IDF). This indicates how
common or rare a word is in the entire entity set.
The closer IDF is to 0, the more common a word is.
So, if the skill is very common across LinkedIn entities,
and appears in many job or member descriptions, the IDF
will approach 0. If, on the other hand, the skill is unique tospecific entities, the IDF will approach 1. Details available
at LinkedIn’s Skills Genome and LinkedIn-World Bank
Methodology note.

Skills (and AI Skills)

AI Skills Penetration

LINKEDIN
Prepared by Akash Kaura and Murat Erer

Country Sample

LinkedIn members self-report their skills on their LinkedIn
profiles. Currently, more than 38,000 distinct, standardized
skills are identified by LinkedIn. These have been coded

Table of Contents

Appendix

The aim of this indicator is to measure the intensity of AI
skills in an entity (in a particular country, industry, gender,etc.) through the following methodology:

217

Artificial IntelligenceIndex Report 2022

• Compute frequencies for all self-added skills by
LinkedIn members in a given entity (occupation,industry, etc.) in 2015–2021.
• Re-weight skill frequencies using a TF-IDF model
to get the top 50 most representative skills in that
entity. These 50 skills compose the “skill genome” ofthat entity.
•C ompute the share of skills that belong to the AI skill
group out of the top skills in the selected entity.
Interpretation: The AI skill penetration rate signals theprevalence of AI skills across occupations, or the intensitywith which LinkedIn members utilize AI skills in theirjobs. For example, the top 50 skills for the occupation ofengineer are calculated based on the weighted frequency
with which they appear in LinkedIn members’ profiles. If
four of the skills that engineers possess belong to the AI
skill group, this measure indicates that the penetration of
AI skills is estimated to be 8% among engineers (e.g., 4/50).

Jobs or Occupations
LinkedIn member titles are standardized and grouped
into approximately 15,000 occupations. These are notsector or country specific. These occupations are further
standardized into approximately 3,600 occupation
representatives. Occupation representatives groupoccupations with a common role and specialty, regardlessof seniority.

AI Jobs or Occupations
An AI job (technically, occupation representative) is
an occupation representative that requires AI skills toperform the job. Skills penetration is used as a signal
for whether AI skills are prevalent in an occupation
representative, in any sector where the occupation
representative may exist. Examples of such occupations
include (but are not limited to): machine learning
engineer, artificial intelligence specialist, data scientist,
computer vision engineer, etc.

AI Talent
A LinkedIn member is considered AI talent if they have

Table of Contents

Appendix

APPENDIX
Chapter 4: The Economy and Education

explicitly added AI skills to their profile and/or they areoccupied in an AI occupation representative. The counts of
AI talent are used to calculate talent concentration metrics(e.g. to calculate the country-level AI talent concentration,
we use the counts of AI talent at the country level visa-vis the counts of LinkedIn members in the respective
countries).

Relative AI Skills Penetration
To allow for skills penetration comparisons acrosscountries, the skills genomes are calculated and a relevantbenchmark is selected (e.g., global average). A ratio is thenconstructed between a country’s and the benchmark’s AI
skills penetrations, controlling for occupations.
Interpretation: A country’s relative AI skills penetration of
1.5 indicates that AI skills are 1.5 times as frequent as in
the benchmark, for an overlapping set of occupations.

Global Comparison
For cross-country comparison, we present the relative
penetration rate of AI skills, measured as the sum of thepenetration of each AI skill across occupations in a given
country, divided by the average global penetration of AI
skills across the overlapping occupations in a sample ofcountries.
Interpretation: A relative penetration rate of 2 meansthat the average penetration of AI skills in that country
is two times the global average across the same set ofoccupations.

Global Comparison: By Industry
The relative AI skills penetration by country for industry
provides an in-depth sectoral decomposition of AI skill
penetration across industries and sample countries.
Interpretation: A country’s relative AI skill penetration
rate of 2 in the education sector means that the average
penetration of AI skills in that country is two times theglobal average across the same set of occupations in that
sector.

218

Artificial IntelligenceIndex Report 2022

Relative AI Hiring Index• LinkedIn Hiring Rate or Overall Hiring Rate isa measure of hires normalized by LinkedIn
membership. It is computed as the percentage of
LinkedIn members who added a new employer inthe same period the job began, divided by the total
number of LinkedIn members in the corresponding
location.
• AI Hiring Rate is computed following the overall
hiring rate methodology, but only considering
members classified as AI talent.
•R elative AI Hiring Index is the pace of change in AI
Hiring Rate normalized by the pace of change inOverall Hiring Rate, providing a picture of whether
hiring of AI talent is growing at a higher, equal,
or lower rate than overall hiring in a market. The
relative AI Hiring Index is equal to 1.0 when AI hiringand overall hiring are growing at the same rate year
on year.
Interpretation: Relative AI Hiring Index shows how fasteach country is experiencing growth in AI talent hiring
relative to growth in overall hiring in the country. A ratioof 1.2 means the growth in AI talent hiring has outpaced
the growth in overall hiring by 20%.

N E T BA S E Q U I D
Prepared by Julie Kim and Tejas Sirohi
NetBase Quid delivers AI-powered consumer and market
intelligence to enable business reinvention in a noisy
and unpredictable world. The software applies artificial
intelligence to reveal patterns in large, unstructured
datasets and to generate visualizations that enable users
to make smart, data-driven decisions accurately, quickly,
and efficiently. NetBase Quid uses Boolean query to
search for focus areas, topics, and keywords within social,news, forums and blogs, companies, and patents data
sources, as well as other custom datasets. NetBase Quidthen visualizes these data points based on the semantic
similarity.

Table of Contents

Appendix

APPENDIX
Chapter 4: The Economy and Education

Search, Data Sources, and Scope
Over 6 million global public and private company profiles
from multiple data sources are indexed in order tosearch across company descriptions, while filtering andincluding metadata ranging from investment information
to firmographic information, such as founded year, HQ
location, and more. Company information is updated on a
weekly basis. NetBase Quid algorithm reads a big amountof text data from each document to make links between
different documents based on their similar language. This
process is repeated at an immense scale, which produces
a network with different clusters identifying distinct topics
or focus areas. Trends are identified based on keywords,
phrases, people, companies, institutions that NetBase
Quid identifies, and the other metadata that is put into thesoftware.

Data
Organization data is embedded from Capital IQ and
Crunchbase. These companies include all types ofcompanies (private, public, operating, operating as a
subsidiary, out of business) throughout the world. The
investment data includes private investments, M&A, public
offerings, minority stakes made by PE/VCs, corporate
venture arms, governments, and institutions both within
and outside the United States. Some data is simply
unreachable—for instance, when the investors or thefunding amounts by investors are undisclosed. NetBase
Quid also embeds firmographic information such as
founding year and HQ location.
NetBase Quid embeds Capital IQ data as a default andadds in data from Crunchbase for the ones that are notcaptured in Capital IQ. This not only yields comprehensive
and accurate data on all global organizations, but it also
captures early-stage startups and funding events data.
Company information is uploaded on a weekly basis.

Search Parameters
Boolean query is used to search for focus areas, topics,
and keywords within the archived company database,within their business descriptions and websites. We canfilter out the search results by HQ regions, investment219

Artificial IntelligenceIndex Report 2022

amount, operating status, organization type (private/public), and founding year. NetBase Quid then visualizes
these companies by the semantic similarity. If there
are more than 7,000 companies from the search result,
NetBase Quid selects the 7,000 most relevant companies forvisualization based on the language algorithm.
Boolean Search: “artificial intelligence” or “AI” or “machinelearning” OR “deep learning”
Companies:• Chart 4.2.1: Global AI & ML companies that havebeen invested (private, IPO, M&A) from 01/01/2012 to
12/31/2021.
• Chart 4.2.2–4.2.12: Global AI & ML companies that haveinvested over $1.5M for the last 10 years (January 1,
2012 to December 31, 2021)—7,000 companies out of
7,500 companies have been selected through Quid’s
relevance algorithm.

Target Event Definitions• Private investments: A private placement is a private
sale of newly issued securities (equity or debt) by acompany to a selected investor or a selected groupof investors. The stakes that buyers take in private
placements are often minority stakes (under 50%),although it is possible to take control of a company
through a private placement as well, in which casethe private placement would be a majority stake
investment.
• Minority investment: These refer to minority stakeacquisitions in Quid, which take place when the buyer
acquires less than 50% of the existing ownership stakein entities, asset product, and business divisions.
• M&A: This refers to a buyer acquiring more than 50%of the existing ownership stake in entities, asset
product, and business divisions.

Table of Contents

Appendix

APPENDIX
Chapter 4: The Economy and Education

COMPUTING RESEARCHA S S O C I AT I O N (C R A TAU L B E E
SURVEY)
Prepared by Betsy Bizot (CRA senior research associate)

SourceComputing Research Association (CRA) members
are 200-plus North American organizations active incomputing research: academic departments of computerscience and computer engineering; laboratories andcenters in industry, government, and academia; andaffiliated professional societies (AAAI, ACM, CACS/AIC,IEEE Computer Society, SIAM USENIX). CRA’s mission
is to enhance innovation by joining with industry,
government, and academia to strengthen research andadvanced education in computing. Learn more about
CRA here.

Methodology
CRA Taulbee Survey gathers survey data during the fallof each academic year by reaching out to over 200 PhDgranting departments. Details about the Taulbee Survey
can be found here. Taulbee doesn’t directly survey thestudents. The department identifies each new PhD’s areaof specialization as well as their type of employment.
Data is collected from September to January of eachacademic year for PhDs awarded in the previous
academic year. Results are published in May after datacollection closes. So the 2020 data points were newly
available last spring, and the numbers provided for 2021
will be available in May 2020.
The CRA Taulbee Survey is sent only to doctoral
departments of computer science, computerengineering, and information science/systems.
Historically, (a) Taulbee covers 1/4 to 1/3 of total BS CS
recipients in the United States; (b) the percent of women
earning bachelor’s degrees is lower in the Taulbee
schools than overall; and (c) Taulbee tracks the trends inoverall CS production.

220

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 4: The Economy and Education

Nuances• Of particular interest in PhD job market trends are
the metrics on the AI PhD area of specialization. The
categorization of specialty areas changed in 2008 andwas clarified in 2016. From 2004-2007, AI and robotics
were grouped; from 2008-present, AI is separate; 2016clarified to respondents that AI includes ML.
• Notes about the trends in new tenure-track hires(overall and particularly at AAU schools): In the 2018
Taulbee Survey, for the first time we asked how many
new hires had come from the following sources: new
PhD, postdoc, industry, and other academic. Results
indicated that 29% of new assistant professors camefrom another academic institution.
• Some may have been teaching or research faculty
rather than tenure-track, but there is probably somemovement between institutions, meaning the total
number hired overstates the total who are actually
new.

Table of Contents

Appendix221

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 5: AI Policy and Governance

CHAPTER 5: AI POLICYAND GOVERNANCE
B LO O M B E R G G OV E R N M E N T
Prepared by Amanda Allen
Bloomberg Government is a premium, subscriptionbased service that provides comprehensive information
and analytics for professionals who interact with—orare affected by—the government. Delivering news,
analytics, and data-driven decision tools, Bloomberg
Government’s digital workspace gives an intelligent edge
to government affairs and contracting professionals. For
more information or a demo, visit about.bgov.com.

Legislative Documents: Bloomberg Government
maintains a repository of congressional documents,
including bills, Congressional Budget Office assessments,
and reports published by congressional committees,
the Congressional Research Service, and other offices.
Bloomberg Government also ingests state legislative
bills. For the section “AI Policy and Governance,”
Bloomberg Government analysts identified all legislation,
congressional committee reports, and CRS reports thatreferenced one or more AI-specific keywords.

Methodology
Contract Spending: Bloomberg Government’s Contracts
Intelligence Tool structures all contracts data fromwww.fpds.gov. The CIT includes a model of government
spending on artificial intelligence-related contracts that isbased on a combination of government-defined productservice codes and more than 100 AI-related keywords.
For the section “U.S. Government Contract Spending,”
Bloomberg Government analysts used contract spending
data from fiscal year 2000 through fiscal year 2021.
Defense RDT&E Budget: Bloomberg Governmentorganized all the RDT&E budget request line items
available from the Defense Department Comptroller. For
the section “U.S. Department of Defense (DOD) Budget,”Bloomberg Government used a set of AI-specific keywords
to identify 500 unique budget activities related to artificial
intelligence and machine learning worth a combined $5.9
billion in FY 2021.

Table of Contents

Appendix222

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 5: AI Policy and Governance

G LO BA L L E G I S L AT I O N R E C O R D S O N A I
For AI-related bills passed into laws, the AI Index performed searches of the keyword “artificial intelligence,” in respective
languages, on the websites of 25 countries’ congresses or parliaments, in full-text of bills. Note that only laws passed
by state-level legislative bodies and signed into law (i.e., by presidents or received royal assent) from 2015 to 2021 areincluded. Future AI Index reports hope to include analysis on other types of legal documents, such as regulations andstandards, adopted by state- or supranational-level legislative bodies, government agencies, etc.

Australia

Denmark

Website: www.legislation.gov.au
Keyword: artificial Intelligence
Filters:• Legislation types: Acts
• Portfolios: Department of House of Representatives,
Department of Senate
Note: Texts in explanatory memorandum are not counted.

Website: https://www.retsinformation.dk/
Keyword: kunstig intelligen
Filter:• Document Type: Laws

Finland

Belgium

Website: https://www.finlex.fi/
Keyword: tekoäly
Noting under the Current Legislation section

Website: http://www.ejustice.just.fgov.be/loi/loi.htm
Keyword: intelligence artificielle

FranceBrazilWebsite: https://www.camara.leg.br/legislacao
Keyword: inteligência artificial
Filter:• Federal legislation• Type: Law

Website: https://www.legifrance.gouv.fr/
Keyword: intelligence artificielle
Filter:• texte consolidé• Document Type: Law

Germany

Website: https://www.parl.ca/legisinfo/
Keyword: artificial Intelligence
Note: Results were investigated to determine how many ofthe bills introduced were eventually passed (i.e., received
royal assent) and bill status was recorded.

Website: http://www.gesetze-im-internet.de/index.html
Keyword: künstliche Intelligenz
Filter:• All federal codes, statutes, and ordinances that are
currently in force• Volltextsuche (full text)• Und-Verknüpfung der Wörter (entire word)

China

India

Website: https://flk.npc.gov.cn/
Keyword: 人工智能
Filters:• Legislative body: Standing Committee of the
National People’s Congress

Website: https://www.indiacode.nic.in
Keyword: artificial intelligence
Note: The website used allows for a search of keywords
in legalization title but not in the full text, as such it is notuseful for this particular research. Therefore, a Google
search using the “site” function to search the site with the
keyword of “artificial intelligence” is conducted.

CanadaTable of Contents

Appendix223

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 5: AI Policy and Governance

IrelandSingapore

Website: www.irishstatutebook.ie
Keyword: artificial intelligence

Website: https://sso.agc.gov.sg/
Keyword: artificial intelligence
Filter:• Document Type: Current acts and subsidiarylegislation

Italy
Website: https://www.normattiva.it/
Keyword: intelligenza artificiale
Filter:• Document Type: law

Japan
Website: https://elaws.e-gov.go.jp/
Keyword: 人工知能
Filter:• Full text
• Law

Netherlands
Website: https://www.overheid.nl/
Keyword: kunstmatige intelligentie
Filter:• Document Type: Wetten

New ZealandWebsite: www.legislation.govt.nz
Keyword: Artificial intelligence
Filter:• Document type: acts
• Status option: For the status option (example: acts inforce, current bills, etc.)

South Africa
Website: www.gov.za
Keyword: artificial intelligence
Filter:• Document: acts
Note: This search function seemingly does not searchwithin the context of the full text and so no results werereturned. Therefore, a Google search using the “site”
function to search the site with the keyword of “artificial
intelligence” is conducted.

South Korea
Website: https://law.go.kr/eng/; https://elaw.klri.re.kr/
Keyword: artificial Intelligence or 인공 지능
Filter:• Type: Act
Note: Cannot search combined words, so individual
analysis is conducted.

Spain

NorwayWebsite: https://www.boe.es/
Keyword: inteligencia artificial
Filter:• Type: law• Head of state (for passed laws)

Website: https://lovdata.no/
Keyword: kunstig intelligens

SwedenRussiaWebsite: http://graph.garant.ru:8080/SESSION/PILOT/
main.htm (Database “The Federal Laws” in the official
website of the Federation Council of the Federal Assemblyof the Russian Federation.)
Keyword: искусственный интеллект
Filter:• Words in text
Table of Contents

Appendix

Website: https://www.riksdagen.se/
Keyword: artificiell intelligens
Filter: Swedish Code of Statutes

224

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 5: AI Policy and Governance

Switzerland
Website: https://www.fedlex.admin.ch/
Keyword: intelligence artificielle
Filter:• Text category: federal constitution, federal acts, andfederal decrees, miscellaneous texts, orders, andother forms of legislation.
• Publication period for legislation was limited to 20152021.

United Kingdom
Website: https://www.legislation.gov.uk/
Keyword: artificial intelligence
Filter:• Legislation Type: U.K. Public General Acts & U.K.
Statutory Instruments

United States
Website: https://www.congress.gov/
Keyword: artificial intelligence
Filter:• Source: Legislation
Status of legislation: Became law

Table of Contents

Appendix225

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 5: AI Policy and Governance

M E N T I O N S O F A I I N A I - R E L AT E D L E G I S L AT I O N P R O C E E D I N G S
For mentions of AI in AI-related legislative proceedings around the world, the AI Index performed searches of the keyword“artificial intelligence,” in respective languages, on the websites of 25 countries’ congresses or parliaments, usually under
sections named “minutes,” “hansard,” etc.

Australia

Denmark

Website: https://www.aph.gov.au/Parliamentary_Business/Hansard
Keyword: artificial intelligence

Website: https://www.retsinformation.dk/
Keyword: kunstig intelligens
Filter:• Minutes

Belgium
Website: http://www.parlement.brussels/search_form_fr/
Keyword: intelligence artificielle
Filter• Document Type: all

BrazilWebsite: https://www2.camara.leg.br/atividadelegislativa/discursos-e-notas-taquigraficas
Keyword: inteligência artificial
Filter:• Federal legislation• Type: Law

CanadaWebsite: https://www.ourcommons.ca/PublicationSearch/en/?PubType=37
Keyword: artificial Intelligence

China
Website: Various reports on the work of the government
Keyword: 人工智能
Note: The National People’s Congress is held once per
year and does not provide full legislative proceedings.
Hence, the counts included in the analysis only searched
the mentions of artificial intelligence in the only public
document released from the Congress meetings, theReport on the Work of the Government, delivered by the
Premier.

Finland
Website: https://www.eduskunta.fi/
Keyword: tiedot
Filter:• Parliamentary Affairs and Documents• Public document: Minutes• Actor: Plenary sessions

FranceWebsite: https://www.assemblee-nationale.fr/
Keyword: intelligence artificielle
Filter:• Reports of the debates in session
Note: Such documents were only prepared starting in2017.

Germany
Website: https://dip.bundestag.de/
Keyword: künstliche Intelligenz
Filter:• Speeches, requests to speak in the plenum

India
Website: http://loksabhaph.nic.in/
Keyword: artificial intelligence
Filter:• Exact word/phrase

Ireland
Website: https://www.oireachtas.ie/
Keyword: artificial intelligence
Filter: Content of parliamentary debates

Table of Contents

Appendix226

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 5: AI Policy and Governance

Italy

Singapore

Website: https://aic.camera.it/aic/search.html
Keyword: intelligenza artificiale
Filter:• Type: All• Search by exact phrase

Website: https://sprs.parl.gov.sg/search/home
Keyword: artificial intelligence

South Africa

Website: https://kokkai.ndl.go.jp/#/
Keyword: 人工知能
Filter:• Full text
• Law

Website: https://www.parliament.gov.za/hansard
Keyword: artificial intelligence
Note: This search function does not search within thecontext of the full text and so no results were returned.
Therefore, a Google search using the “site” function
to search https://www.parliament.gov.za/storage/
app/media/Docs/hansard/ with the keyword “artificial
intelligence” is conducted.

Netherlands

South Korea

Website: https://www.tweedekamer.nl/kamerstukken?pk_
campaign=breadcrumb
Keyword: kunstmatige intelligentie
Filter:• Parliamentary papers - Plenary reports

Website: http://likms.assembly.go.kr/
Keyword: 인공 지능
Filter:• Meeting Type: All

New ZealandWebsite: https://www.parliament.nz/en/pb/hansarddebates/
Keyword: artificial intelligence

Website: https://www.congreso.es/
Keyword: inteligencia artificial
Filter:• Official publications of parliamentary proceedings

NorwaySwitzerland

Website: https://www.stortinget.no/no/Saker-ogpublikasjoner/Publikasjoner/Referater/
Keyword: kunstig intelligens
Note: This search function does not directly allow the
keyword within minutes. Therefore, a Google search using
the “site” function to search the site with the keyword of“artificial intelligence” is conducted.

Website: https://www.parlament.ch/
Keyword: intelligence artificielle
Filter:• Parliamentary proceedings

Japan

RussiaWebsite: http://transcript.duma.gov.ru/
Keyword: искусственный интеллект
Filter:• Words in text

Table of Contents

Appendix

Spain

SwedenWebsite: https://www.riksdagen.se/sv/global/
sok/?q=&doktyp=prot
Keyword: artificiell intelligens
Filter:• Minutes

227

Artificial IntelligenceIndex Report 2022

United Kingdom
https://hansard.parliament.uk/
Keyword: artificial intelligence
Filter• References

United States
Website: https://www.congress.gov/
Keyword: artificial intelligence
Filter:• Source: Congressional record• Congressional record section: Senate, House ofRepresentatives, and Extensions of Remarks

U.S. A I P O L I CY PA P E R S

Organizations

To develop a more nuanced understanding of the thought
leadership that motivates AI policy, we tracked policy
papers published by 55 organizations in the United States
or with a strong presence in the United States (expandedfrom the list of 36 organizations last year) across four
broad categories:• Civil Society, Associations & Consortiums:
Algorithmic Justice League, Alliance for Artificial
Intelligence in Healthcare, Amnesty International,
EFF, Future of Privacy Forum, Human Rights Watch,
IJIS Institute, Institute for Electrical and Electronics
Engineers, Partnership on AI• Consultancy: Accenture, Bain & Company, Boston
Consulting Group, Deloitte, McKinsey & Company• Government Agencies: Congressional ResearchService, Library of Congress, Defense Technical
Information Center, Government Accountability
Office, Pentagon Library• Private Sector Companies: Google AI, Microsoft AI,
Nvidia, OpenAI• Think Tanks & Policy Institutes: American EnterpriseInstitute, Aspen Institute, Atlantic Council, Brookings
Institute, Carnegie Endowment for International
Peace, Cato Institute, Center for a New American
Security, Center for Strategic and International

Table of Contents

Appendix

APPENDIX
Chapter 5: AI Policy and GovernanceStudies, Council on Foreign Relations, Heritage
Foundation, Hudson Institute, MacroPolo, National
Security Institute, New America Foundation, RAND
Corporation, Rockefeller Foundation, Stimson
Center, Urban Institute, Wilson Center• University Institutes & Research Programs: AI and
Humanity Cornell University; AI Now Institute, NewYork University; AI Pulse, UCLA Law; Belfer Center for
Science and International Affairs, Harvard University;
Berkman Klein Center, Harvard University; Center for
Information Technology Policy, Princeton University;
Center for Long-Term Cybersecurity, UC Berkeley;
Center for Security and Emerging Technology,
Georgetown University; CITRUS Policy Lab, UC
Berkeley; Hoover Institution; Institute for HumanCentered Artificial Intelligence, Stanford University;
Internet Policy Research Initiative, Massachusetts
Institute of Technology; MIT Lincoln Laboratory;
Princeton School of Public and International Affairs

Methodology
Each broad topic area is based on a collection ofunderlying keywords that describe the content of thespecific paper. We included 17 topics that represented the
majority of discourse related to AI between 2018-2021.
These topic areas and the associated keywords are listed
below:• Health & Biological Sciences: medicine, healthcare
systems, drug discovery, care, biomedical research,
insurance, health behaviors, COVID-19, global health• Physical Sciences: chemistry, physics, astronomy,
earth science• Energy & Environment: energy costs, climate change,energy markets, pollution, conservation, oil and gas,alternative energy• International Affairs & International Security:international relations, international trade,developing countries, humanitarian assistance,
warfare, regional security, national security,
autonomous weapons• Justice & Law Enforcement: civil justice, criminal
justice, social justice, police, public safety, courts

228

Artificial IntelligenceIndex Report 2022

APPENDIX
Chapter 5: AI Policy and Governance• Communications & Media: social media,
disinformation, media markets, deepfakes• Government & Public Administration: federal
government, state government, local government,
public sector efficiency, public sector effectiveness,government services, government benefits,
government programs, public works, public
transportation• Democracy: elections, rights, freedoms, liberties,
personal freedoms• Industry & Regulation: economy, antitrust, M&A,
competition, finance, management, supply chain,telecom, economic regulation, technical standards,
autonomous vehicle industry and regulation• Innovation & Technology: advancements andimprovements in AI technology, R&D, intellectual
property, patents, entrepreneurship, innovation
ecosystems, startups, computer science, engineering• Education & Skills: early childhood, K-12, higher
education, STEM, schools, classrooms, reskilling• Workforce & Labor: labor supply and demand, talent,
immigration, migration, personnel economics, futureof work• Social & Behavioral Sciences: sociology, linguistics,
anthropology, ethnic studies, demography,
geography, psychology, cognitive science• Humanities: arts, music, literature, language,
performance, theater, classics, history, philosophy,
religion, cultural studies• Equity & Inclusion: biases, discrimination, gender,
race, socioeconomic inequality, disabilities,
vulnerable populations• Privacy, Safety & Security: anonymity, GDPR,
consumer protection, physical safety, humancontrol, cybersecurity, encryption, hacking• Ethics: transparency, accountability, humanvalues, human rights, sustainability, explainability,interpretability, decision-making norms

Table of Contents

Appendix229

Artificial
IntelligenceIndex Report 2022